I have been thinking about this for more than a year. I think it's time to share my thoughts. Disclaimer: all opinions written in this post are purely personal and not tied with any affiliation/institution I am working with.
The global pandemic we are encountering has affected many aspects of our life. While we need to adapt to doing most of our activities online, competing in the International Collegiate Programming Contest (ICPC) -- or similar high-stakes team programming contest -- should not be one of them.
I have no objection to the following online contests: International Olympiad in Informatics (IOI), Google Code Jam (GCJ) final, Facebook Hacker Cup (FBHC) final, and any similar contest final which is participated by the best ~20 competitive programmers in the world. What makes them different from ICPC is the amount of trust we have put in the contestants or their local community.
In the case of IOI, IOI has been built by the strong community we have among the delegation team leaders. We have put so much trust in them by letting them read the tasks several hours before the contest, and take a vote in important decisions regarding the competition. This fact makes it easy to just let the team leaders (or someone trusted by the team leaders) physically proctor the contestants.
In the case of GCJ, FBHC, etc. finals, the competition is participated only by the best ~20 competitive programmers in the world, usually almost all of them are Codeforces LGM. The usage of any materials written before the contest (including the internet) is usually allowed. The only prohibited action is to communicate with other people. However, these competitors are well-known and respected members of the community. At their level, they do not gain a huge (if any) advantage by communicating with most other people. Also, they are risking their huge reputation by trying to communicate with other people. It's just not worth it.
It is entirely different in ICPC. Even in the World Finals, it is competed by 100+ teams. Different from GCJ finals, they are not the best teams in the world due to the process of qualifying to the World Finals with the regionals. Without intending to disrespect these teams, usually, not many people know the teams on the bottom half of the scoreboard. Given how prestigious ICPC is, I can imagine how some teams might do what they are not supposed to do for glory. Additionally, ICPC setup is unique in having one workstation/team and the importance of hardcopy printings, which makes it more complicated to do it online.
It is not better in the regionals. The regionals are still an important and prestigious contest since it determines the teams going to the World Finals. However, it is competed by even less known teams. Some of these teams even compete with lower effort and without the support of their university, which makes it harder to enforce them to a slightly complicated online setup. It is just hard (if not impossible) to ensure that they are not using any prohibited materials. In IOI, most contestants are supported by many people (at the very least, their team leader. most are supported by their government and local community as well), which makes it easier to enforce a sophisticated VM/VPN setup to monitor the contestants' activity in their workstation as well.
Generally, I just think that online proctoring (in whatever format) just does not make much sense. Regardless of how many cameras or screen recordings are enforced, once the contestants leave the camera coverage, no one knows what happened (the contestants may talk to other people, access the internet, etc.). All competitors must be physically proctored by someone we can trust.
That is why I believe that ICPC World Finals 2020 should be postponed until it is possible to host it on-site and participated by (almost) all finalists. ICPC Regionals 2020 should not have happened -- ICPC Regionals 2020 and ICPC World Finals 2021 should have been suspended. When the ICPC is resumed, all contestants who are supposed to be eligible for ICPC Regionals 2020 can participate.
Some of you might know that I am the chief judge of the last four years of the Jakarta site. I am not happy with the decision that Jakarta was hosting ICPC Regional 2020 online and will be hosting ICPC Regional 2021 (most likely online). Jakarta was the only online site in the Asia-Pacific region last year and many teams in this region can only compete on this site. Nevertheless, I still just prefer that ICPC Regional did not happen last year. There were some issues from last year that was caused by online proctoring, such as different teams were mistakenly enforced different rules by different proctors (despite the huge effort of briefing the proctors, it still happened) and teams were getting disqualified because they were not able to gather their team members.
I will not be the chief judge of the Jakarta site this year.
I completely agree with you that ICPC is different than all the other competitions and that it has much higher stakes. Cheating can have a direct pathway if it is held online. Postponing it is the best option.
Not only cheating but also it is a once-in-a-lifetime opportunity for most of the participants. Representing your country at such a grand stage is a big deal.
https://codeforces.net/blog/entry/93986
Yeah I was aware of this recent announcement, which gave me the final push I need to finally post my thoughts.
Because of the Jakarta proctoring rules, combined with Singapore's travel restrictions, my team had to break up :'( (for the upcoming ICPC 2021)
Let it online, just not to cancel. There is no information about even Finals 2020, not to mention 2021
ICPC is the feast. If ICPC is done online, it deprives of all the festive atmosphere
I agree with your sentiment that ICPC contests held online are an extreme disappointment. For the 2021 season, my team has already completed in 3 official contests and we haven't even gathered in person for any contest or practice (North America has a regional, a divisional, and a championship this season; the divisional isn't a normal thing and was mainly made to reduce the number of participants in the championship when it was planned to be in person). If not for the hope of an in person World Finals 2021, the season might as well be entirely cancelled.
Now, the 2020 World Finals (planned in Moscow, October 2021) has some good news and bad news. The good news is that ICPC recognizes how essential the in-person, traditional rules are for the fairness of the contest. The bad news is that with the current plan, many teams cannot be there in-person for various reasons, such as continuing travel restrictions, vaccine shortages in their country, and life moving on (getting settled into a job or military service for example).
You are right that it would be the perfect option would be to hold the contest at a time when we're confident almost all participants can attend. However, this is just impossible in my opinion. Just look at the North American Championship. It seemed like the perfect time to host the contest, with vaccines being extremely available in North America, and cases and hospitalizations going down. However very suddenly, the Delta variant was causing a major outbreak and they basically had no option but to make the contest online or delay it.
With such unpredictable outbreaks, how will ICPC possibly plan to have almost all teams in person? It's already been over a year and a half and COVID still isn't going away. What do we do? Should we plan to have 5 overlapping ICPC seasons only to find out half the participants don't even care anymore? When is it time to just close this book and move on?
Hello Jonathan,
Thank you for sharing your opinion, and also thank you for judging the Jakarta Site. I always enjoyed participating in the ICPC regional in Jakarta.
I emphasize with your opinions. I think that right now the world is not ready for global, international events to occur. I think the Olympics last month were a huge mistake, and on-site ICPC would be another one.
Just one small correction: Japan also held an online regional in 2020.
Ah thanks for correcting this. I brought this point since I was led to believe by several people that last year, all Asia Pacific sites (other than Jakarta) conducted onsite regional and can only be participated by teams from the host country. This left teams from non-hosting countries (like Singapore) to be able to only compete in Jakarta. This fact puts some amount of pressure to the Jakarta site to have to host it.
Looking at this seems-like-Japan-site-2020-scoreboard, seems like regardless of whether it was held onsite, Japan regional also accepted only local teams?
Between 3 options: having online ICPC / postpone / cancel, I don't understand why online round isn't the better option.
For example, I gave up a lot of things to participate in 2015-2016 season (e.g. enrolled in a univ just to participate). My teammate is graduating that year and will most likely stopped training after graduation. So either delaying / cancelling would be terrible for me. I can see at least other students will have similar issues.
As for online experience, obviously there are serious issues as you mentioned. As organizers we can learn from each other, so I always read experiences from other online regionals and hope that VN one can learn something from it. You saw issues with 2020 online Jakarta regional, why not try to improve 2021 one?
I also have some questions that I hope you can answer:
how serious was this issue? e.g. did some contestants use Google search while some can't?
did team know (some time before contest) about requirements that they need to be in one place?
I heard this point several times lately and completely understand this (and I don't have strong counter-argument against this). What I can probably say is that the judges should only use tasks that were supposed to be used in 2020 (e.g., if the judge learnt a new technique found recently, they shouldn't create a new task based on that). If they already have the tasks before the WF was suspended, then those tasks should be kept and eventually be used.
Unfortunately some of the issues are hard to be improved. For example, I don't see how to improve the fact that some teams in Jakarta site are competing with lower effort and less support, thus making it hard to enforce a more complicated proctoring setup like what the IOI is using.
To be honest, I might have exaggerated this. I received one entry in the post-competition survey that their proctor does not allow them to view task statement in their main workstation -- they were only allowed to do so in the secondary workstation. I also remember one contestant (that I know personally) reported that their proctor disallowed them to do something that is supposed to be allowed by the rules, but unfortunately I can't remember the details.
ICPC Jakarta 2020 rules allow each teams to use two workstations: main and secondary. The secondary workstation can only be used to read what they normally read with printed materials (e.g., task statements, their submissions, etc.). The main workstation can be used to do anything allowed in normal ICPC rules (incl. viewing task statements, of course).
Yes, the requirement was posted many days before the competition. Sorry, it was not a disqualification, rather it was teams (or some of their team members) dropping out of the competition once they realized they cannot gather their team members.
And there was this "issue" that one of the team member from the eventual 1st-place team got ejected literally minutes before the contest starts, because the committee just realised that this team member did not participate the practice session together with the other two team members. This rule is also posted many days before the competition.
Well regarding the issue with proctors, it can probably be improved, but still hard. Let me be very transparent here and explain in more detail regarding the proctors of Jakarta site last year (at least from what I believe):
For most of the proctors (if not all):
Before the practice session, the briefings for the proctor were done by $$$S$$$ and not by the judges (most of the judges in Jakarta site, including myself, are industrial software engineers and are not working for the host university). During the practice session, I saw how messy the proctoring was: a lot of proctors do not know what they need to do, a lot of them ask questions to $$$S$$$ (but $$$S$$$ was unsure as well, so they passed it to the judges). Between the practice and competition day, I asked $$$S$$$ to gather the proctors to re-brief the proctors myself. The briefing happened, but it turns out it was still not enough.
Let alone the briefing sessions. The set of proctors from the practice session day and contest day is not the same -- there are proctors that can proctor only on one of the day. There might be proctors who proctored the competition day but never attended any of the briefing sessions -- they just relied on recordings or text materials from the briefing. This might not be enough.
There were 62 teams participating the site last year. Due to the complicated rules, each proctor must be focused to proctor only one team, and there has to be some backup proctors as well. For a regional contest, gathering this many people and asking them to spend their days to attend briefings and proctor the practice and competition days, especially if we want the people with some ICPC background, is not an easy task.
Lastly, do not get me wrong. I appreciate and thankful for what the proctors did last year. They did a great job. The mistakes were not due to incompetence, rather due to the complicated rules. The complicated rules were necessary to host online ICPC. If all of the last year's proctor can become proctor again this year, their experience from last year should improve this year's proctoring. But I think that case will be unlikely.
So when do you think the time "it is possible to host it on-site and participated by (almost) all finalists" will come? Let's say it's 2023. Then three consecutive competition years (2020-2021,2021-2022,2022-2023) will be cancelled in your opinion. What about the contestant who wishes to advance to WF in these years?
Your solution is not at all realistic. There is no foreseeable end to Covid and 2023 might even be not late enough. When should we stop the wait?
Postponing the contest harms the community a lot and new contestants will have no chance to get into ICPC as there is not at all any contest to participate. When the contest restarts, there could be no one to understand what it is for.
Last year's Seoul Regional (in Korea) was like this: they send you a tripod (along with some hand sanitizers). You need to stream:
The first two things were easily done with some browser magic. One contestant at a time can go missing for at most 3 minutes, so that we could print some things or use the toilet. (I think I almost missed it several times.)
That's how the proctoring worked here, and I think it was pretty good. It was indeed a hassle to set up all the things, and I do think the screen sharing and/or webcam streaming could have taken a toll on slow machines or networks.
You made interesting points! I also think about this a lot. So the expected gain of cheating is:
You mean: for IOI, the first term is tiny; for GCJ, etc., the probability term is too high; for ICPC, the first term is quite high, and motivates cheating in overall.
I think IOI also has a high first term. IOI is a contest for (mostly) teenagers, and a medal could make a strong impression in college admissions. And good college admissions are some big deal, right? Some people are willing to pay incredible amounts of money.
So how, then, do we manage to get by for years, without having some big cheat scandal? My answer is trust. The host country and their staffs, the International Committees, the Team Leaders and Deputy Leaders and everyone involved, they all see weak points that can be breached and exploited to give a contestant unfair step ahead. But they don't, and they trust each other that they wouldn't.
The same goes for ICPC. You are a chief judge, but how do you make sure the problems are never leaked to the contestants? You may never know if they even did. But you trust your staffs and everyone involved in preparing the contest, and that's how you manage to hold a contest with very high stakes. How about WF? They have First Solve prizes, they have corporate sponsors that might link to job opportunities, etc. The stakes are even higher.
To wrap up, my point is this:
To be honest, for WF, I think it would be good to have someone else physically proctoring; I mean, the three of you gather at one place, and ask a professor (with experiences in ICPC if possible) in your area or country and have them monitor you. However I don't know if this is possible in other parts of the world. Currently we are good to travel around the country and gather in private groups, until 6 p.m. when your groups are limited to at most 2 people (and it only applies to private ones, so you may as well claim it's an official business and go okay with it).