I'm a little curious about CF's back-end for each contest.
In my country, Taiwan, there is an "educational" certification for average-level and low-level students who may not talented in programming called CPE (Collegiate Programming Examination).
They hold 4 times every year. There are about 1500 students, partitioned into about 50 exam rooms, connected through Internet, to participate this contest at the same time.
Even I haven't participated in CPE yet, but in my experience in ACM-ICPC regional, the PC^2 system are somehow slow to wait for my verdict if there is a lot of submissions in a short time.
I know there are some well-known jury systems like DOMjudge, CMS (recently used by IOI), etc.
Recently, some students participated in CPE argue that the judge process is slow, and the CPE system is modified from the DOMjudge.
I'm wondering is there an efficient jury system to meet such heavy load.
Well, I know the contest rule in CF is different from the one in ACM-ICPC (the CPE rule is same as ACM-ICPC), also different from the Google Code Jam.
Is there a jury system using ACM-ICPC rule and can hold maybe up to 5000 even 10000 contestants?
Maybe the most important factor is hardware issue (how many computers to absorb the submissions) not just software (jury system).
It seems the Cloud Computing is the trend in recent years. So how to scale up the jury system to hold more contestants?
This is very interesting.
The UVa online judge seems to renew their website (but I don't know their current progress), I'm looking forward to see it!
AFAIK, judging on CF distributed betwen dozen of computers. Each computer can evaluate around 8 solutions simultaneously.
It's actually not so hard to calculate load on your system: suppose you want to hold 10k people, with average time to solve = 30 minutes, average test amount = 30 and TL = 2second. Therefore each solution will be tested for at most 1 minute. So each of your threads will be able to serve 30 people. So you need 10000/30 ~ 333 threads. With 8-threaded PC it requires 333/8 ~ 41 pcs. And one machine to make all interface stuff plus scheduling.
If you will reduce test amount to 5 pretests — divide that number on 6.
But calculations above not taking into account situation with easy problems, when system receives outrageous amount of solutions between 3 and 10 minutes.
Scaling up the judging process is trivial when it's distributed (as in DOMjudge): just throw more machines at it. The bottleneck problem with these kinds of contestant numbers is the webserver and underlying database.
DOMjudge has been used by IIT Amrita for their Aspirations contest in which about 4600 people competed (indiviually), see this presentation at ICPC WF 2014. The main modifications to make it run fairly smoothly were to use hardware load balancers, distributed MySQL databases for read/write queries and to disable the standard scoreboard.