Sounds interesting, though taking 10 attempts to solve 43% of all problems does not seem to be around the 85th percentile if you think about WA/TLE penalties (they did seem to take into account time penalties though).

Did they correct for test data leakage, since they mentioned that they fine-tuned on the Codeforces dataset? They mentioned that Gemini was the model that made all the difference, so I am curious as to whether that model was trained on code or editorials for the remaining problems (since it is easy to overfit on the tiniest of data when you have huge models).

Also curious about which divisions the contests on which AlphaCode 2 performed better than 99.5% of the participants were for. I would be really impressed if they could do well on the latter half of a Div1 contest without any test data leakage, since Div2/3 contests seem to have problems with more limited ideas than Div1 (especially Edu Rounds, which are supposed to be standard in some sense, and thus are more "learnable" by machine learning models).

To be honest, sampling a million concrete code samples and sifting through them to make clusters (which is what they do) sounds like a really bad approach for a human or any algorithm that claims to have "learnt" something — at some point it just devolves into guessing and tuning your ranking functions. This is in the same vein as "generative" models not generating things based on reasoning, but based on some transfer distribution that comes from probabilities, which is a far cry from being verifiable. TL;DR — in its current form, it seems like proof by AC and that the models are not really learning anything in a rigorous sense. I don't expect any PAC-style estimates relating these models to mathematical rigor any time soon, but would be pleasantly surprised if there exist such estimates. Of course, the ideal scenario is to solve the IMO Grand Challenge.

Edit: there is also the AIMO prize with the prize fund of $10^7.

Edit: Since this is the top comment, it is worth noting here that there is a strong chance of test data leakage, because of the following:

The AI solution advertised: https://codeforces.net/contest/1810/submission/234586634

Someone's solution: https://codeforces.net/contest/1810/submission/204262433

Someone else's solution: https://codeforces.net/contest/1810/submission/200078083

I'm told that the original code for the last of the above submissions is by a Chinese competitive programmer and the code was posted on a Chinese blog website (seems very plausible given that this happens for a lot of hard problems, though I do not have an exact link to validate the claim), so test data leakage from other websites is also possible.

I'm even sure there is a solution that is much closer to this solution than the above one, but still the structure being that similar (even though there are other solutions with a distinct structure) provides compelling evidence to start investigating whether these claims suffer from test data leakage or not.

I also looked into the other AI submissions (supposedly from their other alts), and they seem to be other minor changes on top of the other submissions.

Obviously, the best way to see whether their model can successfully solve competitive programming problems is to test their performance in real contests.

We also randomize targeted metadata included in the prompt, such as the problem difficulty rating and its categorical tags

Isn't that answer leakage?

Where is the appendix where it explains how the evaluation is done and how training data leakage was avoided? Did this gigantic foundational gemini model already see the editorials in its training data? Does solving 43% of problems really correspond to 1800 elo? (Solving ABC fast in div2 doesn't seem like it would.)

This would hold a lot more credibility if they just let the model run in a couple of live contests.

This report is very disappointing. The alphacode 1 paper was significantly higher-quality than this. Hopefully they release a much more thorough version.

Looks like this could be the model in question (thanks to this comment)

Curious, how it can flop on problems A-C and make 5+ WA submissions in a row (an average participant would have already quit at that point), but solve a 3000+ problem on a first try

Nonetheless, that is some impressive stuff, but until these llms learn how to apply binary search on the answer I think the competitive programming as we know it is relatively safe

Waiting for a more detailed report from DeepMind, or just an opportunity to test the Gemini model in action (which I believe is just a foundation for AlphaCode 2)

I wonder what it will be able to a decade later. So much improvement just happened in less than 2 years.

I think we can believe these data only when they make one account, take part in contests, and continuously achieve good ranks.

Yeah, and one more thing is that in the contest testing process now we must include AIs as well, at least for div. 2C and div. 2D.I know we can't do more with Div. 2A and B-level problems because now it seems more possible to solve them using AI.  

Remembering Alon Musk here, "Mark my words, AI is far more dangerous."

Gemini: Excelling at competitive programming

I think the problem statement they tested was from the past codeTon round4 Which is basically

So There is a possibility that it already knows the solution of this, as submissions and editorial are there (obviously from reading this data) as part of training data

So let's see if it can solve problems in live contest

At this rate, someday the term "Competitive Programming" will evolve from "competing to solve programming problems" to "competing to create AI to solve programming problems" lol

cAn aI s0Ive nPc Prob15m?

Here is the submission 229535002 for the 3200 rated problem 1810G - The Maximum Prefix shown in Google's video

for (long long i = 0; i <= n; i++) cin >> h[i];
for (long long i = 0; i <= n; i++) dp[0][i] = h[i];
for (long long i = 1; i <= n; i++) {
  for (long long j = 0; j <= n; j++) {
    dp[i][j] = ((p[i] * dp[i - 1][j + 1]) % mod + ((1 - p[i] + mod) * dp[i - 1][max(0LL, j - 1)]) % mod) % mod;
  }
}

Here is the 8 months old editorial for the same problem

for(int i = 0;i <= n;i++) scanf("%d",&h[i]);
for(int i = 0;i <= n;i++) f[0][i] = h[i];
for(int i = 1;i <= n;i++) {
  for(int j = 0;j <= n;j++) {
    f[i][j] = (1LL*p[i]*f[i - 1][j + 1] + 1LL*q[i]*f[i - 1][max(0 , j - 1)] ) % mod;
  }

Is it time to bring back this meme?

I just realized AdamantChicken2 shortens to AC2 -> AlphaCode 2.

whenever I read that AIs solve human tests (cp, university exams, school exams, etc) I cannot help but find the discussions misinterpreting the results.

Computer Programs are fast. If AlphaCode 2 can solve 43% of the problems within contest time, then it may not be able to solve 44% within 1 year. It already had near infinite time. Now imagine how good AlphaCode 2 would place in a contest that runs for a year. How good would the AI place if very unique problems with long solutions appear. Not good. Under these conditions, the AI may not even reach pupil status.

Also Note that according to the report, the AI is not expert-cm. The AI placed in the top 0.5% in 2 contests. Meaning it has contests where it places in red, but therefore also has contests where it does in fact places like pupil. This should not shock anyone, the report clearly states that the AI is basically good at guessing which of the millions solutions to random problems work for the current problem.

why they not simply let AI doing some rated contest that make sure problem is new but not let AI vp old contest spark unnecessary arguments

Thanks for the interest!

To address some questions in the comments:

Regarding possible data leakage: this was indeed a huge concern for us and we went through extensive lengths to avoid contamination. Please see this post on X for more details on how we did it. TLDR: we're confident that AlphaCode 2 did not see this specific problem (or its editorial/solutions); however, given that prefix sums appear regularly in competitive coding, the model has likely seen somewhat related (but distinct) problems and managed to adapt this knowledge to come up with its own solution.

An additional data point: AlphaCode 2 has a solve rate of nearly 100% on problems it was actually trained on, which is very far from its performance on any held-out contest we evaluated on.

On the potential leak of tags: we are using randomly-chosen tags for each sample we generate. Interestingly, for that particular sample the tags did not contain "dp" :)

PS: Extra kudos to TwentyOneHundredOrBust for figuring out the easter egg in the account name :)

So the summary is: The bot solved A, C and G from a Div1+2 round, nothing else. Its participation was virtual. Gs solution is extremely similar to authors solution. They promise there was no leakage. To me this is pathetic

High time to ask, "Is it the end of Competitive Programming? Should I quit grinding and hustling?"

why not evaluating on their own codejam/kickstart problemset :)

In This video, it is claimed that Gemini can solve a 3200 elo problem. Is this an existential crisis for newbie competitive programmers like me?

Interestingly, in AdamantChicken2's code, the "quick power" is written as "qmi" (quick mi), it seems that AdamantChicken2 also knows Chinese phonetics.

Would like to see future untrained problem performance

Not implying that this is impossible, but it actually seemed a bit unnatural to me that an AI could accurately do tasks that require high amount of precision for correctness, like solving a very specific math problem or a programming statement with a very high confidence rate.

For drawing, writing and all it makes sense, where just the overall pattern/structure matters...

I guess the comments above talk about plausibility, but I honestly don't care that much whether it is true or not. I am pretty sure it will happen at some point anyway, so whatever. And I am not surprised by this in any way! Because we know that the most important thing in solving competitive programming is practice, and that's exactly the thing that machines are much better at than we are. I genuinely believe that if you give a chicken a million years to practice, it can become a candidate master or a master. I am not trying to insult candidate masters with this comment, it is nevertheless hard to become one! I just mean it in the same way as that if a chicken studied chess for a million years, it would become better than Magnus Carlsen. It's just about human resources. If some 3500 legendary grandmaster would call a problem "an exercise" from their experience mountain, why a computer can not learn to do such exercises? And the reason why I still am not impressed is because indeed these are the problems you just need to learn to go through, and the real stuff begins later when a 3500 will have to start thinking. And I don't see AI being able to solve any of these problems any time soon so whatever...

If I understand correctly, this large language model is like a clever monkey typing in the infinite monkey theorem? Because it seems like it's just randomly putting together some code and then trying them one by one, but using some clustering method to speed up this process.

so guys should i stop grinding or no ?

Probably the rules of using code generators will have to be modified. Even if they publish the code and the weights of the neural networks, the majority of people will not have sufficient resources to run this code during the contest.

We don't need AI We need EI Emotional Intelligence