Disclaimer: This is a rant about Meta Hacker Cup and may not contain any useful information.
Meta Hacker Cup is one of the biggest annual programming competitions, but it has the strangest submission format, unlike any other online judge. Why does it have to be so different? Let’s take a look at how Meta Hacker Cup 2024 Round 1 went for maomao90
The time now is 1:00 AM in Singapore. The contest starts, and maomao90 begins solving the problems.
The time now is 1:41 AM. maomao90 has solved problems A, B, and C without much trouble and starts working on problem D. After quickly coming up with a theoretical solution, maomao90 begins coding.
The time now is 2:05 AM. The code is ready and passes the sample tests. maomao90 proceeds to validate the solution.
The time now is 2:06 AM. Validation passes, and the input zip file is downloaded.
The time now is 2:07 AM. maomao90 runs the code on the final test.
Oh no! How did the code pass validation tests but fail the final test with an assertion error? Panicking, maomao90 scrambles to debug the code.
The time now is 2:11 AM. Five minutes have passed since downloading the zip file. maomao90 fails to debug his code and is no longer allowed to submit problem D. maomao90 wasted 30 minutes of his time and is left frustrated and in tears.
Problem 1: Why is the validation test so weak?
Is the validation test intentionally weak, or is it a mistake by the problem setter?
If it’s intentional, what's the goal? To make participants suffer? Brute-force algorithms often pass validation easily but take far longer than five minutes for the final test. Why is that?
Problem 2: Why are participants allowed only a single 5-minute attempt?
Almost every other online judge allows multiple submissions when your solution is incorrect. Why does Meta Hacker Cup limit participants to just one try?
One possible reason is that if someone's code takes more than 5 minutes to run, they can wait until their code finishes running before making a second attempt and AC the problem even though their solution took much longer than 5 minutes to finish running. However, there's an easy solution to this:
- Instead of one input file, create three strong input files, each worth $\frac{1}{3}$ of the total points.
- Allow participants to download each input file individually, with a 5-minute submission window for each file.
- This way, if a participant fails to submit for the first input file, they can still debug and submit for the second and third, potentially earning $$$\frac{2}{3}$$$ of the total points.
- This approach would also strengthen the final test with three times more input data.
The time now is 2:12 AM. After a brief crying session, maomao90 starts on problem E.
The time now is 3:41 AM. maomao90 validates the solution for problem E but lacks confidence after the disaster with problem D.
The time now is 3:44 AM. After a final check, maomao90 downloads the zip file and runs the code for the final test.
The time now is 3:45 AM. maomao90 submits the output for problem E. There’s nothing else to do now, as problem D can’t be submitted. maomao90 is tired and wants to sleep, but at the same time, maomao90 wants to know whether his final output is correct. Unfortunately, the final verdict will only released after the contest...
Problem 3: Why is the final verdict delayed until after the contest?
Is it to reduce server load by judging only after the contest ends? The server doesn’t even need to compile or run code~--- it only has to compare two text files. Is that really too much for the server during the contest?
If the final verdict were provided immediately, along with the solution proposed in Problem 2, the contest experience would be far more pleasant. Yet, after 14 years, there’s still no improvement in the grading system. Why is that? Even Codeforces is experimenting with pretest=system test to prevent "Fail System Test" issues.
The time now is 4:00 AM. The contest finally ends, and maomao90 can check if he solved problem E correctly. Thankfully, it was accepted and he celebrates.
The time now is 4:01 AM. Looking at the leaderboard, maomao90 sees the number of WAs on problem D.
So many red crosses! maomao90 laughs, realizing many others faced the same weak validation issues on problem D.
Problem 4: Why doesn’t Meta Hacker Cup follow other online judges and run the code for us?
The ultimate solution to all these problems is simple: adopt the standard system used by most online judges, where participants submit their code, and the platform compiles and runs it. Why hasn’t Meta Hacker Cup implemented this?
Codeforces held its first round in 2010, using the current code submission system, and Meta Hacker Cup started in 2011. Why did Meta Hacker Cup opt for this convoluted system of downloading password-encoded zip files instead of following the code submission system that Codeforces uses?
Please upvote this blog if you faced similar issues or agree with the solutions mentioned. Hopefully, Meta will consider these suggestions and improve the system in the future. :(
I have to do nothing but upvote. I solved B (at least I thought so) but missed n = 3 and n = 4. They weren't even in validation cases. I slept 30 minutes before the end of contest due to sleep and in the morning BOOM! I didn't even qualify for Round 2 :(
I mean life does give suprises and I'm ok with it. It taught me to stress test more but a little bit fault goes to the system as well... I hope that they improve their system so no other contestant faces what I faced
They called me a hater. Its 2024 and a CP contest is still using such a silly submission format.
I don't agree with the point about weak validation test, since it's supposed to be just a sanity check for the format I guess, but everything else is 100% true.
Idea about several input files is especially great, since it opens up possibility to make 3 inputs with different complexity (easy\medium\hard) and encourage participants to write solutions for hard problems even if they are not optimal, so that they can at least get points for an easy input.
Also another minor issue is the requirement to have a facebook account, makes no sense. Meta already has standalone sites like metacareers for example which has separate and more simple account system, not sure why it can't be implemented here.
That's stupid. You could as well use the samples for "sanity check".
Making validation input almost equal to test like it's done on CF makes life easier for those who can't test their solutions properly. Instead of complaining about it maybe just git gud.
Hi, I agree with the "Why doesn’t Meta Hacker Cup follow other online judges and run the code for us?" part. I setup my entire system for the competition, including codes to increase stack size etc. I coded the solution for problem A, and upon downloading and opening the final input file (which was very large), my system crashed. By the time I could figure out a solution or switch to another device, the timer had already ended.
Are you familiar with how Google Code Jam used to operate?
If MHC were to change from the current format, I would prefer MHC to go the way of old Google Code Jam (with small and large input cases, and you still run your code locally on your own machine) instead of new Google Code Jam (where you submitted code for evaluation.)
Hmm why do you prefer running locally rather than submitting code?
It's fun and unusual. It's also double fun when you get an assertion failure while running on the final test, and have to fix it within a few minutes.
Additionally, I believe there's an important educational aspect.
I very much enjoy the aspect of 6-minute timer and sometimes scrambling to fix the solution. It's my favorite part of Hacker Cup.
It's comments like yours that make them still retain this submission format. Just because you're high rated, probably with a better pc and likely able to solve more problems even if your submission timer expires doesn't mean everyone will feel the same.
If you look at the topcoder format it is not that different. Only sample tests to test your code on, only know the verdict at the end of the contest. You can submit multiple times, but the only reason why you would resubmit is if you have found a hacking test on your own. Still people enjoyed topcoder for a long time. It's different from CF, true, but noqt bad. What makes it bad is if you just approach it as if it's CF and submit immediately once you pass samples/validation. In this format testing your code pays off, or you can make a gamble and submit for a faster solve time but less guarantee. There's strategy to this.
In problem D it was obvious validation was very weak as you could see there were two more cases added except the sample, no big cases and D could have lots of casework and edgecases. I was also gambling on D and lost, but I was not mad about not solving.