Most probably, smart people here might have already thought about this, but still..
During the system test phase of Codecraft' 20, I observed many submissions failing around the 100th test case or so. To me, this seems like waste of processing power, to let the codes run till 100 submission and then figure out its failing.
Isn't it possible to sort the test cases by: - decreasing order of number of non-ac submissions - in case of tie, increasing order of average time spent in judging it.
There could be an evaluator thread (for example), which does this after every 100 submissions or so (or something better)?
Is this something too difficult to achieve with the current codeforces infrastructure? One possible difficulty I can imagine is the scenario to show test cases to users. Are there more issues?