Yes, I stand in Petr on this point ... Now I look back it seems more better if we give a hack as soon as possible, but not adding tests! But We were lack of experience about this issue, and didn't know we as a manager, can also hack just like other participant.

The tests are so weak, I didn't consider the random data's property, the tester also has responsibility, in fact, a few of them can solve this one. I hope that will give all of us a lesson about this issues, manager must aware of this in the future.

But in this situation, I'm afarid it'll be improper if we set his solution as a correct solution, because his code didn't maintains the derivative, and only work when all the delta is closed to a or b. It will be unfair for other participant and irresponsible for himself if we give him the score for this problem.

We are trying to discuss this point in more detail in the editorial.

Anyway, I felt so guilty about this. And I really appreciate his first-E last-A strategy during the contest. It is not only courage but a confident of the personal strength. After all, without him, the contest will become boring and boring ...

How can I do something to compensate you ... Jacob, please tell me ~~

No matter what will be decided, I suggest these tests shouldn't be removed from practice. It affects only your solving abilities and the fewer wrong solutions pass during practice the better.

It's my fault.I truly apologize for this,and promise I won't do this any more. Although I don't think Jacob should get the score..

I made 4 generators:gen gen6000 genab gen6000ab,and first three generators can only made weak data.

I don't have any experience.I only put 4 "gen6000ab" in all 100 test!!However,I put 80 "gen",the weakest generator...

His solution accepted these 4 "gen6000ab".But when I added 6 "gen6000ab",his solution got 3 WA..test 101 is one of them.

It's my fault because my data is too weak,but actually,I didn't made his solution WA on propose(I do not mean I don't want to do it,just...for example,I didn't run generators for 1000 times or make "anti" data."gen6000ab" is added before the contest).

And why I do it?Because the solution to this problem is so magical,and I'm so selfish that I really don't want to see a wrong solution got AC..

At last,I apologize for this again.Wish you all forgive me.

Here's a similar question that MikeMirzayanov has brought up offline:

What about bugs in tests that are discovered because some team submits a correct solutions but gets a WA, then judges look at the team's output and realize it's actually correct? Should we fix those?

Apparently my above reasoning leads to a "no" answer — since this is subjective, too (a lower-rated team getting a WA would probably not attract attention). However, the real-world practice has always been a "yes", for example my contests in Petrozavodsk did have rejudges caused by teams submitting correct solutions while my solutions were wrong.

TopCoder's answer to this question has usually been "yes, but the contest becomes unrated", although that was not true in a recent match.

I don't have a good answer here. Somehow a "yes" here doesn't sound as awful as the today's situation to me, but I can't explain why.

I believe adding test cases against a specific approach to be acceptable (however, it's better to do so early enough) — in fact, such practice is not uncommon among problem setters of serious informatics competitions in Slovakia.

I do not, however, agree with adding test cases in such a way that a specific solution would be sure to get a WA. The goal of programming competitions is not to get a correct solution, but to get a solution which passes all tests, and is therefore "correct enough".

So, were it up to me, feel free to add some test cases against Jacob's idea, but do it before looking at his program :D

I am absoulutely agree with you. The situation when you can cheaply change yours tests during the contest allow (and sometimes force) testers to not properly care about tests berore the contest. It is not good and results in such situations like with Jacob's problem E.

First I should apologize bcz I didn't test the Problem E carefully as a Tester so didn't point out the test-data is somehow weak.

But I still should make my point here, let us supposed, if there's a problem, but the test data is too weak, and some simple wrong greedy passed. Then it is really fair? Just to say, a good coder would realize quickly the simple greedy is wrong, and of course they won't try it. But an normal coder may simply try it for nothing to lose. What's more, maybe one good coder type like 200~300 lines code for the right solution, but an wrong greedy with 30 lines also passed, is that really fair?

You are right that a programming contest should be objective.BUT THE ROCK-PAPER-SCISSORS IS THE MOST OBJECTIVE ONE! why don't we just throw a dice to decide the winner, how objective it is?

I just mean that it should not only be objective, but also should be fair, which mean only let the correct solution passed.

it's my (and Seter roosephu) first CF round.We wanted to make it perfect. CF is not OI which encourages one to cheat for scores.i don't think a ACTUALLY WRONG solution should get accepted in system test. contestants hack others' submissions to point out others' mistakes and get scores as a reward. when there is a wrong submission but nobody can hack it,we problem setters do have the right to execute it.

Anyway,Jacob is the hero of the contest..

I think the only fair way is to test every possible input.

In my opinion, adding tests during the contest is something bad. But in the situation that the origional tests are not strong enough, it is worthy to do that bad thing. Idealistically, the tests should accept a solution if and only if which is perfectly correct. A solution which fails in the additional tests fails due to the incorrectness of itself, not some subjective reason.

I agree with you, Petr. One of the main reasons why programming competitions is so popular is its objectiveness which ensure the fairness. In most cases, you will never find an objective way to check each possible solution. As for the imperfect solutions, what should we do? Since we don't have the ability to discover all of them, we should , at least treat them equally.

Btw: If this happens again, someone would like to add a lot of meaningless information to protect his "score"(only a joke, don't be serious)

When the mistake with test data do happens, what to do then? Hacking one program is unfair for a single competitor. Let the program get AC is unfair for other competitors. Neither seems a good idea.

While I don't have a “42” answer on the matter, I'd like to point out an associated issue.

This thread may well have hundreds of comments and seem to come to a conclusion eventually. But the Codeforces team does not usually cover such cases publicly. Instead, some internal conversations take place and the decision is silently made. To an external observer, it is unclear whether such a thread influenced the final decision at all, and hard even to find out what the decision was in the first place.

In such interesting cases (contestants' appeals in general), I would like to have them publicly recorded along with the judges' answers and motivations. Generally, it is a good thing to do; to check that it is so in more common sports, I just googled for appeals in football and instantly found some public decisions.

When the judges' decisions and motivations are publicly available, we can expect them to act and decide in a similar way when a similar issue arises. For a contestant, it is a good thing to know in advance what to do and what to expect in some corner case of the contest rules.

When the judges' decisions and motivations are kept private to the judges' team, we have no clue what will the judges' reaction be next time. Even worse, ironically, contestants which engage in private conversations with judges on the matter gain a slight unfair advantage of knowing their motivations and knowing what to expect next time. So, it turns out that keeping the appeals private leads to more unfairness — the very same unfairness the appeals are designed to decrease!

i think it was appropriate to add the test data and hack Petr's submission of E before the contest ended as soon as possible.

Well I don't know about you all guys but I hope this kind of thing will not repeat.

I think it necessary to prepare all data for a test before it finishes.While solutions look not quite right may pass all test data, right solutions may not be able to pass some special data. So, It's fair to give the coder the points he deserve for fair and make his solution fail for justice. Of course, I think the mistake can be understood, and the adding of the last data make the problem more perfect.

In fact the answer is simple. Someones tried to design test cases that any wrong programs can't pass, this is just impossible. For example, if a qsort don't use randomized pivot, should we design tests to make it TLE?

SO the only fair way is to design tests before contests, then to all participant, the probability of their wrong solution passing the tests are the same. Design tests based on an existing solution is just unmoral.

I think some people who excel on programming are genius, but they don't know what is fair. They just think stronger is better. To try to make only "right" solutions passing the tests are just childish thinking, stop your chuunibyou, pls.

Anyway I think Codeforces need make a rule to make it clear that what a jury can do and what they can't do.

This round should be unrated for Jacob (_maybe_ for his room too).

At least he gained some contribution :)

I agree with Petr, the reason I participate in programming contests for many years is that: It is auto judged and the judgement process is public(Maybe not true for ACM). That's different with some math contests: it needs jury to understand the proof statements written by contestants and sometimes it's unfair.

And I know it's very hard to make a good contest under these conditions, because I have some experiences of writing problems (contests):

1. Sometimes making test cases is hard, for example: I proposed this problem: MagicMolecule in late 2011, but it was used in SRM571, just about 3 weeks before today. One reason is that it's not easy for making test cases to defeat some naive search algorithm. So I think the author (and admin) should't use a problem until he(she) think test cases are prepared well.

2. Sometimes a non-intend solution will pass the system test: In the Div-I hard problem of SRM557: XorAndSum, this short solution passed system test, which is totally out of expectation. I think it should pass even if it is indeed incorrect, because it passed all test cases we prepared before the contest. I think we should never add test cases when the contest begin, even if it will make the contest looks more 'fair'. If test cases are very weak, then they are. It will push the writer to do better in next time, but he(she) should't have the right to modify test cases during the contest.

3. Sometimes we find the reference goes wrong, for example: In SRM571, my solution of Div-I Hard: CandyOnDisk goes wrong. During the challenge phase, Petr submit a challenge and find the reference solution gives out a wrong output. The story after that is: admins fixed the reference solution and a re-judge was made. (is that fair?) It's hard to make a decision when this kind of things happen.

So I think we should make some rules before the contest: Does the writer have permission to modify the test case? What should we do if the reference solution goes wrong? What should we do if the system goes down during the last 5 minutes of challenge phase? ...

And I agree with Gassa's suggestion: at least we should publish all modifications made during the contest. (By the way, I heard that: test cases of ACM/ICPC Finals will be destroyed after the contest, is that true? If so, then why don't we publish them instead?)

I'm happy to see people discuss this kind of issues here.

CF is an interesting game,isn't it? Without rules we can't fully enjoy it. No matter the result is judges can add tests against submitted solutions during the contest or not , as long as rules are definite,it's fair,and I think we all can accept that. Don't take it seriously,just have fun.

This is the most interesting event I have seen in CF...

Standing in the perspective of both of Petr and WJMZBMR... I think that what they said both are reasonable.. So I think the most important thing is not argue with the result of this game. The passed things were history, CF should give a clear rule to solve the problem to make these kind of things not happen any more.

This is only a game. If it happen to me, I don't care about it if it's nothing to do with my money...

BTW, reading this blog and all of the replies is one of the way to improve English :)

Our whole Codeforces team has been tracking the discussion of the described case. We've considered the community opinions and we've added our own ideas and we now want to say:

1. In a perfect world we should use your ideas and completely agree with you. Unfortunately, the world tends to be imperfect. When a group of authors is really motivated to prepare the contest, then no such need arises. For instance, no similar idea has been thought of the Saratov ACM-ICPC subregionals as everything is ready much earlier and in full. But the rounds are regular and the authors are different. Not all of them are super skilled or super motivated. In this case the problem is weak tests.

2. The jury's goal is to face every situation deliberately and reasonably and act, whenever needed, to improve the contest as much as possible. In this case we are sure that the added test didn't contain any specific patters against this solution. Most probably, the point was fixing a bug in the tests after viewing this solution.

3. We consider it okay to fix a bug in a problem after some participant's feedback. In this case it was a solution attempt. Don't worry about biased jury. Trusting the jury is an important part in programming contests. What will become of these contests if we stop trusting Andrew Stankevich to work in the NEERC jury? Let's trust ACM-ICPC World Finals jury members. You shouldn’t ask “who knows who they are and how they choose problems?”

4. Petr has offered this principle, because it guarantees the impartiality and objectivity. But the human factor in the preparation and holding rounds always exists. Authors makes problems and problemsets according to their preferences. Maybe the author consciously chooses a problems, making them easy for his friends? Should we allow to participate his friend's in his round? Should we get the problems for a round from 5 different writers from 5 different countries? Or come up with some kind of limitation that reduces the possibility of such a partiality. Petr proposed the principle of the same kind. We should trust the jury as we trust participants.

5. It is also important to understand that we try to act objectively and impartially. Jacob submitted a wrong solution and received WA. The test had a general character and its solution simply pointed the testset's defect. This is of course an isolated case and we will hope it won't have any reason to happen again.

6. I want to repeat — this situation is an exception, rare untypical case.