Hello Codeforces! I am writing this blog post to bring attention to and publicly discuss inconsistencies in the ICPC World Finals selection process for North America.
For those who are unaware, every North American (NA) team competes in one regional (which is usually determined by their geographical location) and top teams are selected from each regional to represent NA in World Finals. The only official rule is that the winner of each region qualifies for finals. The remaining teams are selected based on several criteria that are not officially specified. In the past few years, NA has received 23+ world finals slots.
I am a member of Georgia Tech’s premier ICPC team, and my teammates are AlanWaP and RaresC. We compete in the southeastern USA (SER) regional in North America. Our region also contains University of Central Florida, who consistently have a top ~25 team at World Finals and who were the North American Champions and bronze medalists in the 2018 world finals. This year Georgia Tech ranked second in our regional solving 7/11 problems, and the top UCF team solved 9/11 problems. We have been in similar situations for the last four years: achieving 2nd place, losing to UCF, and not advancing to world finals.
Since UCF is the reigning NA champion, we expected to qualify for World Finals this year. In the past, there seems to have been an unofficial rule that gives the region of last year’s medalist(s) additional world finals spot(s). This rule appeared to be in place for Harvard and MIT, which are both in the Northeastern North America (NENA) regional. In 2016 world finals when both Harvard and MIT medaled, NENA received 3 world final spots for the next world finals. Note that the difference in problems solved between rank 1 and rank 3 was 4 problems. Similarly, when Berkeley was a medalist in the 2015 world finals, the PACNW region received three slots instead of their normal two for the next world finals.
After communication with NA ICPC officials, it seems that the difference in number of problems solved is primarily taken into account while determining how many slots are allotted to a region. This metric is inconsistent with the NENA example above, and it is also unreliable for several reasons:
- First, different regionals have different problem sets and some are significantly easier than others. For example, in Mid-Atlantic USA 2018, there were only 8 problems, 5 distinct teams solved all of them, and every problem was solved by at least 9 teams. SER was undoubtedly one of the hardest (if not the hardest) NA problem set this year.
- Second, it doesn’t take into account the strength of the region. For regions where the top teams are not as strong as UCF, one should expect to have closer competition.
- Last, our team's computer crashed for 30 minutes during SER 2018 around the 150 minute mark, which seriously impacted our time penalty. Printing at our site also took at least 45 minutes, making it almost impossible to “debug on paper” while a teammate codes another problem. Our team did not get a contest extension even though we made multiple requests, so we ran out of time coding the 8th solution. We want to point out that several schools in our region including UCF competed at a different site and didn’t encounter these problems. That said, both of these problems were officially recorded and the NA selection committee was aware of the differences.
The World Finals teams were announced recently, and we were surprised to not have advanced. Furthermore, the number of teams from NA was reduced from 23+ to 19 this year. The officials told us that the reason for this is the “growth of ICPC”. Why were so many slots cut from North America? They also told us that “NA champ’s region gets an extra slot convention” was considered but not used this year due to the new space constraints. However, they gave additional slots to other regions based on scoreboard closeness.
It is AlanWaP s last year of eligibility, and possibly my and RaresC s last participation as well. We’ve all invested a nontrivial number of hours into programming competitions, so this is incredibly disheartening for us. We performed very well nationally in NAIPC 2018 and in NAQ 2018. It is evident that we are a very strong NA team with the potential to perform well at world finals, so it seems odd that a region like SER is so underrepresented. Unfortunately, there is very little transparency in the selection process, and Georgia Tech regularly receives the short end of the stick. It’s terrific that our region has fierce competition, but we believe SER should routinely be awarded more WF slots because of this.
On a more positive note, there is discussion about North America moving to a super-regional system to decide teams next year, which would be fantastic and should eliminate all the problems caused by comparing incomparable regions. Coincidentally, it is likely going to be hosted by Georgia Tech. There has been an ongoing push for a more fair selection process for years, so we hope that the super-regional contest becomes the new standard for qualifying for World Finals from North America.
ACM-ICPC rules are shit, especially in some regions (looking at you, Asia). I hope it will be better with superregionals.
While I don't think you can do anything about the number of slots, your computer crash is an extremely important issue. Don't rules state that there must an extra time in that case? What did organizers think while deciding about it?
We were initially told that we would probably get a 15-minute extension, so we were under that mindset for the last two hours. Then, with about 30 minutes left, we inquired again about the official extension and were told there was none.
We didn't pursue this further because we didn't think it would have changed our rank, but it could have made us closer to UCF in the number of solved problems.
On last CERC, my team's computer crashed/froze multiple times and we lost well over 30 minutes on reboots. Our requests for time compensation were denied. It was extremely demotivating so I just ended up wasting even more time walking around the sandwich area contemplating life and whatevs... We solved 3 problems :D
In which sense Asia (especially the PSP subregion) rules are shit?
PSP Rule (since 2016-2017 season)
For those who don't know: Each site has a site score (according to the number of schools/teams attending the site; formula is published beforehand) and each team can attend at most 2 sites.
For each site, remove medal schools, remove Chinese teams and keep only top team for each school. Each remaining team got a priority:
(rank - 1) / (site score).The teams with lowest priorities (over all sites) are qualified. (Keep adding teams until there are enough schools.) This is the published list in C. J. Hwang's blog. If multiple teams from one school qualify, coach decide (as only one team from each school can attend World Finals).
Factors on Choosing Site
Upcoming Changes
PSP Finals is proposed for 2019: regionals qualifies for this semifinal, and this semifinal qualifies for World Finals.
From time to time, there is something on Codeforces about the unfair choice of teams advancing to the finals. I remember more things like the first link below, but couldn't find it.
https://codeforces.net/blog/entry/22889 (unknown X and Y)
https://codeforces.net/blog/entry/22255 (inhuman approach and implicit votes)
https://codeforces.net/blog/entry/48626?#comment-327107 (ridiculous prices and disqualifying teams that complain)
https://codeforces.net/blog/entry/49295?#comment-341499 (multiple sites with a common rank list but different starting time)
I'm glad that it changed in PSP. Also, I find your previous comment very informative and new (I've never seen those rules explained before) and I think it's stupid people downvoted it.
The number of implicit votes just show to bad the system is, and how many people are in some committee and they don't care.
I didn't say anything about the quality of problems. Your example isn't the worst. It's worse if it's hard to understand the statement, tests are wrong, or the machines don't work.
Rules shouldn't help the corruption/incompetence.
There was also a drama about the ECF/China Final rules this year. I don't have the time to translate all of these, but this might be interesting to fellow Taiwainese and other oversea ethnic Chinese who can read Chinese.
AFAIK the organizers of ECF 2018 (NWPU) gets to hold the contest for a decade, starting from 2017. Hey at least the supporting problem setters have creative problems to share and their machines don't crash!
I don't know Chinese, but seeing emotes in the chat is enough ;_;
Edit: I realized your point is that the current rule is ok. While rules are becoming better, it was shittier before, and hence many people's view Asian rules in general is shit.
Asia rules were shit, though I agree in recent years it is becoming better. Some examples:
(rank - 1) / (site score). Everything was decided by C J Hwang. Each year C J Hwang just publish some numbers and list of teams qualified. Some years ago a Vietnamese team should have qualified based on published number (but was not in the published list), so they emailed and then C J Hwang just changed his magic numbers.While I do agree with you that your team should be going to World Finals this year, I want to point out that our region also experienced lots of computer issues (for example, any attempt to save code in any IDE made the system lag for 30-60 seconds every time). We were all under the impression that your team would be going to World Finals as well so I think it's really stupid that you don't get to go. Hopefully the super regional contest fixes issues like these.
May I ask what region you’re from?
Southeast USA Region (the same as the original poster)
We had wrong test data in 2 out of 4 regionals. Beat that.
I had to sit on my friend's couch for my regional. Beat that.
We almost had no restrooms in ACPC. Beat that.
Yangon 2018: the intended solution of F is wrong. Mid-contest, the problem is changed to match the intended solution. Moreover, problem H ask you to compute MD5. Beat that.
By wrong test data, I actually meant intended solutions were wrong. In our case, standings were changed after 2 days when organizers realised they messed up. Not sure which is worse.
I have heard of three cases of wrong test data:
Wrong checkers accepting everything
In the specific cases I know, (a) we didn't submit in time and no other teams even attempted it, or (b) everyone should get WA but got AC instead and the winner is unaffected (in this contest only winner matters)
This should not appear in a perfect contest. Fortunately, it did not caused controversies.
I have heard of two cases of wrong intended solutions, too:
Ones that produce correct test data, but fails on some untested cases
In the specific case I know, in-contest my teammate asked me if a solution make sense and I told them no. Later I told my teammates to try it anyway because so many teams solved it that it must be the intended one. Turns out problem setter generated the test data with a memory-heavy but correct algorithm, testing a lighter algorithm against the data, and thought the lighter one is correct.
BTW some teams did solve it with truly correct algorithm that fit in memory limit.
This should not appear in a perfect contest either. But hey! It's definitely a less severe issue. If we don't know the intended solution, we probably just thought the tests are weak.
Well, here they actually changed the standings after 2 days, so a team expecting to go to World Finals couldn't go. So I think it's much better to fix things mid contest instead of after 2 days.
Also about Yangon.
In problem C, H, G, you have to print an extra space at the end of the "Case X:" line, or you will get WA. Beat that.
In H, my team get an incorrect clarification that you don't need an extra space at the end of the "Case X:" line and wasted nearly 40 minutes. Beat that.
E is an NP problem where the intended solution is backtracking with some optimization and pray that it will pass. This type of problem also appeared in the last year regional.
In C, the constraint was ridiculously nonsense (N ≤ 232 where N is the number of vertices in the graph), while N was relatively small in the test data.
Another suspicious thing is that, most of Myanmar teams solved F pretty quickly (and before the announcement to change the problem statement). This did not affect the top of the scoreboard though.
Before participating in this regional, I have heard rumor about the terrible problemset. But this is even worse than what I expected.
Where did you find the intended solutions?
Actually, the stuff about the intended solution was quite an exaggeration. I don't know about the intended solution, but all the team got AC on E solved the problem that way.
I heard a different solution. Greedy is wrong for that problem but some teams passed it with multiple rounds of greedy and some randomisation.
The important point here is that, I don't think a standard solution exist for this problem (unless someone can propose a correct solution with acceptable time complexity)
EDIT: Comment converted to blog post https://codeforces.net/blog/entry/64800
Haha, stupid Americans
Isn't this the reason why some Chinese started CCPC? If ICPC continues acting like this sh*t, quite simply, we should replace it.
What is this CCPC?
China-CPC
I've always thought it's stupid as shit that SER gets literally one spot when you and UCF are both in that region. Ridiculously unfair to GA Tech. This is even stupider when you take into account that Mid-Central US, Mid-Atlantic US, North-Central NA, and my region East-Central NA all receive THREE slots, and 8/12 of those teams got 100th or lower (10/12 got 90th or lower), when it's clear that GT would outperform the vast majority of those schools (NAIPC 2018 standings are pretty quick evidence).
Who knows what type of principles ICPC tries to stand for.
I
Can
Pick
Competitors
I just looked at the world finals team list and I find it ridiculous that Latin America has been awarded more slots than North America. There is clearly some autocracy within the selection system that has gone awry, and they are covering it up under the guise of "growth".
Just a note to say that when the NENA region was allocated 3 slots in 2017 following Harvard and MIT both medalling in 2016, it was announced just before the start of the NENA qualifier to the 2017 world finals that 3 slots would be allocated to the region. That is, it was decided before the competition and the difference between the number of problems solved was not a consideration. Although given that UCF medalled last year, it looks like your regional should have been allocated a second slot before the competition, without considering number of problems solved. So I don't disagree. I hope that North America will transition to the super regional format soon. Under the current system it's rare that any team other than Harvard and MIT ever get through the NENA regional, except in that one year where we were allocated 3 slots.
This is very interesting information. I heard they announced 2 slots before the start of the competition in NENA 2018 as well.
Let me provide some more context. When we reached out to officials to plead our case, Dr. Ali Orooji (UCF Coach and Director of NA ICPC; he seems to be one of the key people deciding NA teams) told us that "a lot of things are taken into consideration" while assigning slots to regions, including how close a team is to the leaders, and provided examples of other regions that had closer competition. This implied that a medal is not sufficient to guarantee an extra slot.
SER is marginally larger than NENA, and we are clearly good enough on a national level, so I can't think of any reasonable justification for the differentiation.
Out of curiosity — why is it that someone (so strongly) associated with a team is one of the main decision makers about WF advancing?
(Just saying... at our site [UCF, but seemingly not at Georgia Tech's site] it was announced that GA Tech will be going to world finals... so we are all surprised about this)
Why the change of heart?
Wow, none of us were aware of that! Thanks for letting us know.
It is a complex issue. Always has been. It will sound self-serving I admit but I’ve always been one of the people that has to tell 2nd and 3rd place teams that they don’t get to go simply because they are from the same school as the first place team. I understand the rules but those are 6 real individual students that also get skipped over. And sadly will still get skipped in the proposed super regional concept.
Sure, it is unfortunate for many students at UCF that you have multiple strong teams, but ICPC WF has always been a contest between schools, not individual contestants. MIT is in a similar situation. Teams there often take all the top spots in BOSPRE (2018, 2017, 2016) but only one gets to go to NENA and have a shot at advancing to WF. This does not affect the slot allocation for NENA: Harvard went to WF even though they lost to all MIT teams in BOSPRE for the last few years.
Ratings of Codeforces users from the US, among whom many are in undergrad.
Ratings of students at your school.
As a faculty in GT SCS, I have maintained to our administration over the past three years that this perceived `skill difference' in the region is at the amateur vs amateur level, and sooner or later we'd get a break (and thus a WF trip). Unfortunately it seems that we are very far from getting to such a point: your post, more than anything, has convinced me that it would require a reverse situation (GT doing better) for SER to get 2 slots.
At UCF, biggest problem is losing WF eligibility.
It looks like the North America superregional — the "North American Championship" — will take effect for the upcoming World Finals!
See this link for confirmation and links to the allocation of the number of spots in the superregional from each regional (under rules): nac.icpc.global. There will be 55 teams at the superregional and the top 16 teams will advance to the World Finals!
Lol Georgia Tech is the school hosting NAC.
If anyone has some info about a mirror contest please reply here :)
I agreed with Errichto that ICPC rules are shit. And I also want to demonstrate an example here that even you have a REGION FINAL in your region, things still has possibility to get weird out of your expectation...
My team has trained hard to win a medal in WF2020. We also know that we must get advanced to WF first. In Asia, we are allowed to participate in 2 regionals and East-Continent Finals(EC-FINAL) to compete for slots. My team got a 4th place in a particularly competitive regional and a 10th place in another regional(Some problems of this regional are faulty). I admitted that our performance is not good enough.
Soon in the EC-FINAL, we won 4th place. We believe that we have earned a slot since it is the SUPERREGION FINAL. According to EC-FINAL from 2015-2018, it had been considered as a SPECIAL CONTEST and given at least 4 slots.
Well, things changed in this year. All regionals(including EC-FINAL) are considered THE SAME and given 3 slots EQUALLY. Even a child of 5-year-old can tell the difference between them, but the ICPC Asia committee can't.
That's how I missed my best shot in winning a WF medal.
Another interesting thing is that one of regionals was resisted and protested by many strong universities this year. That regional has been officially announced some 'punishment' in the president's Blog. And it has still get 3 slots, which has been delegated to some teams that usually they are unlikely to win a slot(But congratulations and good luck anyway).