Most probably, smart people here might have already thought about this, but still..
During the system test phase of Codecraft' 20, I observed many submissions failing around the 100th test case or so. To me, this seems like waste of processing power, to let the codes run till 100 submission and then figure out its failing.
Isn't it possible to sort the test cases by: - decreasing order of number of non-ac submissions - in case of tie, increasing order of average time spent in judging it.
There could be an evaluator thread (for example), which does this after every 100 submissions or so (or something better)?
Is this something too difficult to achieve with the current codeforces infrastructure? One possible difficulty I can imagine is the scenario to show test cases to users. Are there more issues?
Well, in fact according to my experience setting one round most systests cases was ran by participant's submissions during the round, probably Codeforces could copy the results to the system testing phase and not really run those test cases again.
I think about that too, some thing like
Max_heapfor the test cases may be useful, I guess ~~Correct me if I am wrong