Yes or no, depending on the circumstances. In practice, no.

If you keep submitting the same random solution and switching the seed until you get more points, you can get disqualified. This approach doesn't demonstrate any programming skill and it's very unlikely to work anyway.

Randomised solutions that give correct results and are supposed to give correct results should be ok. Even if you submit something and it works by accident, that should also be ok. (There's no reverse engineering in that.)

Such a rule was also proposed last year, but it was not used because team leaders didn't want it. Let's see what happens this year.

At first I thought this was just a stupid idea. I'm pretty sure a similar idea was presented last year and it was shot down for being vague and unenforceable.

But thinking a bit more, and reading closely, I think perhaps this rule is okay?

For example, I thought at first that the following case would result in unfair judgement: Say there is a problem with a tree. A contestant incorrectly believes that node 0 will always end up with even degree, so she adds:

assert(edges[0].size() % 2 == 0);

and ends up with a crash judging result. Suddenly (I thought) the contestant is disqualified for discovering details about the test data! But actually this is okay by the rule, because if she fixes her solution to handle all cases where node 0 has odd degree, then she has not "adapted to specific test cases" but rather solved all cases with node 0 having odd degree.

I hope that it will be clarified that doing this intentionally is also okay(?) That is, if you get WA, you are allowed to check what type of case it is getting WA on by submitting with asserts.

Maybe this is already obvious to everyone else and I'm just thinking too hard ><

(oh but also, there is still the problem of how to enforce the rule...)

edit: actually I guess they could just actually go through with testing solutions on another data set with a different seed, and then manually check any solutions that get a different score/different correct subtasks.

One should be rewarded for 10000 points for having such solution. In the new rule contestants may only submit at most 50 solutions for each task. People who can solve problem in highly test-data-dependent manners under such rule don't even exist.

That's a strange rule, and I'd like to see a rationale behind it. In absence of such a rationale, I'd say the rule is far from perfect:

1. Most importantly, it puts an additional burden on the contestants: they now have an additional poorly defined measure of solution quality to keep in mind. This better be justified by a real concern.

2. Next, it does not play well with certain types of problems, even ones which appeared before. For instance, IOI 2013's Art Class is essentially test-data-dependent. If such a problem appeared in IOI 2017, with the current wording of the rule, many contestants would be anxious about what they can and what they can't do in their solutions. Generally, the rule is problematic any time the problem has only few outcomes (say, a yes/no problem with very few test cases). Some contestants will be overthinking this rule in such problems, and it can't be fully helped.

3. Furthermore, there definitions are not clear. What would be an equivalent set of test cases if there was no randomness involved in generation of certain tests? How much is "significantly fewer"? If a contestant submits the same solution with different random seeds for 2, 3, ..., 50 times, how much repetitions is cheating? Or just how lucky they have to get on their second=last try to be considered cheaters?

4. Finally, what's the use case of the new rule which gives anyone a benefit? For example, if problem setters overlooked a certain flavor of test cases, the rule does not cover them, since changing the random seed usually does not help to cover another class of tests.

The keyword is `significantly fewer', aka. going from 100/100 test cases to say, 2/100 test cases. We felt that this rules out most of the adaptive submission behaviors such as using asserts to see what's wrong with the data and etc.

The most common form of this exploit is to use the feedback mechanism to extract hashcodes of testcases, and get much higher points by doing 'if data hash = !@#!@#, do this'. This lets contestants to:

1. guess and check when the answer to that test case is binary. (some contestants realized that this was possible for IOI15F Towns, but refrained from doing this)
2. distribute the running time of a n^2 time solution over multiple submissions, then use the hash code to narrow down to the `part' that solved that test data correctly. (this was done for IOI14F Holiday, and nearly succeeded for IOI16F Aliens too)

Just thinking. How many people (approximately, per year) have tried to exploit the IOI system in such a way?

In IOI, frequently some testdata have problem and affecting contestants. So, contestants might not believe correctness of testdata, make some assertions, lead some so-called "reverse-engineering" and then get disqualified. They are affecting students in wrong verdict like giving WA but AC and spends more time debugging, giving TLE but WA and looking for infinite loop or time complexity or many other cases.

Moreover in IOI, there are problems which have strange scoring criteria such as IOI 2015 scale, which same solution can be scored different depending on testdata. In this case, testdata is almost predictable and order is matter. I also tried random solutions for 1-2 points changing random seed. Is it also considered as cheating?

Problemsetter should make various solutions and testcases considering lots of corner-cases and solutions. Solution with intentional code with fileIO, using grader bug should be considered as cheat, but if not, some random solutions got accepted, it is mistake of problemsetter. I hope NOT contestants get disadvantage due to problemsetter.

Many contestants depends on full-feedback of IOI and there are some disadvantages of this system. This is also rule from that disadvantage. I think this rule is saying like we are going to change full-feedback system.

For fun, here is how to hack IOI 2015 Towns:

The problem is set up so that there exists a tree with N leaves. Initially, you don't know anything about the tree, but you can ask queries of the form getDistance(i,j), where i and j are leaves. You want to compute something about the tree while using at most c*N queries, for a fixed constant c. To prevent hacking, the problem was set up so that there were multiple test cases per file. But it was possible to isolate a failing file and only process the output for that particular file.

In a suboptimal solution, there are two phases: The first is to find the radius of the tree, and the second is to use the radius of the tree to compute the answer for the tree. A standard method to find the radius is to first find the vertex that is at the end of a diameter of the tree. Computing any individual phase could always be done, but computing both phases might exceed the allotted number of queries. So the idea is to compute only one phase at a time.

As Richard mentioned above, in 2015 it was possible to extract bits from the judge through the memory usage (which was 1GB if I recall correctly). Thus to communicate a number X back, you could declare an array of size X KB and look at the memory usage. Doing so, for each test where we needed too many queries, we could simply output the test index through the memory. Then, when we got to that test, we could compute the first phase, outputting a vertex that was at the end of a diameter. Hardcoding this vertex into the code, we could then run the second phase within the query limit. Doing this for each failed test case, it was possible to get full score on the problem in less than the submission limit.

If it is similar to last year, the judging system will not show you the result of all the cases, only the result of the first non-AC case, the same as CodeForces (WA on test 17, for example).

Isn't it already extremely difficult with 50 submissions and this style of mistake reporting, to hard-code or reverse engineer the data (as long as there are a significant number of test cases)?

To make it even harder, they can even stop telling you the test case you got it wrong if it is past the first 20. So you can't even know if your cheating worked, or if your greedy code is good or not.

Also, in the case of Holiday 2014 and Railroads 2016, it is totally the fault of the problem-setters to have poor data which allow certain greedy algorithms to pass...

It is unfair for contestants to worry about being disqualified for actions with no intent to cheat. Often asserts and such are very helpful to debug REAL bugs (as opposed to figuring out the case). This was why I was very uneasy when the rule was proposed last year, as I thought it was ridiculous to ban any sort of "asserts" which help me to maintain my code correctness and assure myself of the logical assumptions I'm making. Technically, it was still gaining information about the test data (e.g. if it RTE, I know why) but it would NOT be figuring out the exact answer to pass that specific case, it would be to fix a MAJOR bug.

The point is, there are many better ways from the contest organiser's position to prevent cheating, especially because this current rule is pretty poorly worded — "significantly" is extremely subjective, "specific" could mean anything from a certain type of case to the exact numbers in the case.

Once one becomes acquaintanced with the problem itself, it is pretty obvious that it is a problem (we do not want it to be easy for contestants trained in using this technique, to be able to easily solve many test cases without trying to solve the IOI task itself at all).

Also, I think it is pretty obvious that perfectly defining "what is this kind of cheating" in a fixed written rule is impossible in practice. So, I think that there are only two (three if we add "using a huge number of test cases so that this is not possible", but that is probably not practical at all) practical solutions (of course, "do nothing" is a no-solution, but possible):

1) Reduce the feedback given to contestants, so that this is not a problem at all in practice. For example, an ACM-ICPC World-Finals-like full feedback, where you get a "Time limit exceeded", "Wrong answer", etc verdict for the task (subtask) as a whole (as opposed to each single individual test case) is probably enough feedback to give a user a very meaningful judging result, but most likely not enough to perform any of these attacks.

2) Clearly (on an informal level) define the idea, spirit and real problem to solve behind the rule, and trust the ISC on deciding whether something is "clearly hacking the test cases on purpose with no intention at all to solve the problem algorithmically, but just exploiting the full feedback to solve those particular test cases" (or whatever careful wording of the main spirit of the rule is chosen).

Personally, I am fine with any of these (for example, I trust that the ISC will be very reasonable and follow the spirit of the contest: specially considering that they will have to explain any such disqualification to the whole assembly). But I would strongly recommend just going for the 1) approach, because it is fairly obvious that many people will not trust the ISC subjectivity (and it's perfectly fine: I see no reason why force the community to trust the ISC on such specific matters, if the community is not comfortable doing so and there is no absolute need). So, in practice, 2) will likely face strong opposition from the community again (just see the reactions to this post), while I don't think that most people will be strongly against 1) (I might be very wrong on this last assumption) if such feedback is reduced specifically having these problems into account, and "full feedback" (as in, meaningful useful feedback for every submission, like the ACM-ICPC world finals example) is retained.

Orthogonal idea: Why give info about every single kilobyte and millisecond used on every test instead of giving ACMish "AC/WA/TLE" for every subtask? I am a very big fan of instant feedback, but see no reasons to provide unnecessarily detailed info. I guess I am not bringing innovative ideas here and that has been very likely widely discussed, but I would like to hear reasons behind that very full feedback.

For me this rule is at least kind of awkward, if not "against the spirit of progamming competitions" (in Petr's word).

Feedback contains only [WA / TL / ..] + testcase ID and it is fairly safe: even if the output is Yes / No, you need about O(log(# of test case)) submissions to hook that test case so toally you need O(log(# of test case) * (# of test case you can't solve)) submissions, this can be much more than 50 if there are hundreds of test cases.

And still, people can cheat if he fails on only 1 or few test cases (since you say he must solve 'significantly fewer' test cases after changing test cases in order to become a cheater).

There are many other solutions:

1. Don't give feedback of testcase ID (if you think it makes the system not safe enough). Or you can give percentage of TL / WA / RE. Atcoder uses something similar (you can even know how many test cases you have). It sounds vulnerable but over years I never had a chance to 'cheat' using that feedback.

2. Re-generate test case for each submission. GCJ had this rule for small submissions and it works fine even if you can see the test data.

3. Just use stronger test cases, and don't use tasks that are vulnerable.

After all, this issue is better to be solved on the organizer's side than put responsibility to the contestants. And this rule need to be translated in many languages and people may get different interpretation of it.

IMO this rule will be removed again after a very ambiguous case appears where it's not clear if it should be applied or not.