Disclaimer: This is a rant about Meta Hacker Cup and may not contain any useful information.
Meta Hacker Cup is one of the biggest annual programming competitions, but it has the strangest submission format, unlike any other online judge. Why does it have to be so different? Let’s take a look at how Meta Hacker Cup 2024 Round 1 went for maomao90
The time now is 1:00 AM in Singapore. The contest starts, and maomao90 begins solving the problems.
The time now is 1:41 AM. maomao90 has solved problems A, B, and C without much trouble and starts working on problem D. After quickly coming up with a theoretical solution, maomao90 begins coding.
The time now is 2:05 AM. The code is ready and passes the sample tests. maomao90 proceeds to validate the solution.
The time now is 2:06 AM. Validation passes, and the input zip file is downloaded.
The time now is 2:07 AM. maomao90 runs the code on the final test.
Oh no! How did the code pass validation tests but fail the final test with an assertion error? Panicking, maomao90 scrambles to debug the code.
The time now is 2:11 AM. Five minutes have passed since downloading the zip file. maomao90 fails to debug his code and is no longer allowed to submit problem D. maomao90 wasted 30 minutes of his time and is left frustrated and in tears.
Problem 1: Why is the validation test so weak?
Is the validation test intentionally weak, or is it a mistake by the problem setter?
If it’s intentional, what's the goal? To make participants suffer? Brute-force algorithms often pass validation easily but take far longer than five minutes for the final test. Why is that?
Problem 2: Why are participants allowed only a single 5-minute attempt?
Almost every other online judge allows multiple submissions when your solution is incorrect. Why does Meta Hacker Cup limit participants to just one try?
One possible reason is that if someone's code takes more than 5 minutes to run, they can wait until their code finishes running before making a second attempt and AC the problem even though their solution took much longer than 5 minutes to finish running. However, there's an easy solution to this:
- Instead of one input file, create three strong input files, each worth $\frac{1}{3}$ of the total points.
- Allow participants to download each input file individually, with a 5-minute submission window for each file.
- This way, if a participant fails to submit for the first input file, they can still debug and submit for the second and third, potentially earning $$$\frac{2}{3}$$$ of the total points.
- This approach would also strengthen the final test with three times more input data.
The time now is 2:12 AM. After a brief crying session, maomao90 starts on problem E.
The time now is 3:41 AM. maomao90 validates the solution for problem E but lacks confidence after the disaster with problem D.
The time now is 3:44 AM. After a final check, maomao90 downloads the zip file and runs the code for the final test.
The time now is 3:45 AM. maomao90 submits the output for problem E. There’s nothing else to do now, as problem D can’t be submitted. maomao90 is tired and wants to sleep, but at the same time, maomao90 wants to know whether his final output is correct. Unfortunately, the final verdict will only released after the contest...
Problem 3: Why is the final verdict delayed until after the contest?
Is it to reduce server load by judging only after the contest ends? The server doesn’t even need to compile or run code~--- it only has to compare two text files. Is that really too much for the server during the contest?
If the final verdict were provided immediately, along with the solution proposed in Problem 2, the contest experience would be far more pleasant. Yet, after 14 years, there’s still no improvement in the grading system. Why is that? Even Codeforces is experimenting with pretest=system test to prevent "Fail System Test" issues.
The time now is 4:00 AM. The contest finally ends, and maomao90 can check if he solved problem E correctly. Thankfully, it was accepted and he celebrates.
The time now is 4:01 AM. Looking at the leaderboard, maomao90 sees the number of WAs on problem D.
So many red crosses! maomao90 laughs, realizing many others faced the same weak validation issues on problem D.
Problem 4: Why doesn’t Meta Hacker Cup follow other online judges and run the code for us?
The ultimate solution to all these problems is simple: adopt the standard system used by most online judges, where participants submit their code, and the platform compiles and runs it. Why hasn’t Meta Hacker Cup implemented this?
Codeforces held its first round in 2010, using the current code submission system, and Meta Hacker Cup started in 2011. Why did Meta Hacker Cup opt for this convoluted system of downloading password-encoded zip files instead of following the code submission system that Codeforces uses?
Please upvote this blog if you faced similar issues or agree with the solutions mentioned. Hopefully, Meta will consider these suggestions and improve the system in the future. :(