Balajiganapathi's blog

Codeforces problem recommender
By Balajiganapathi, history, 8 years ago, In English

Hi all,

While practicing I always have difficulty choosing which problems to solve next. So, I coded a site to look at the user's submissions so far and recommend what all problems to solve next. Since the ICPC season is at peak now, it can also analyze multiple users and recommend practice problem for the whole team.

View the site here: Code drills.

So far, it only supports cf (EDIT: Now also supports codechef and spoj). I plan to add support for other sites later. Besides recommending, it will also show some statistics of your submissions so far. If you scroll down after entering your handle, you can view the practice recommendation. I will explain some of them here:

ICPC: Best to use this with team i.e. enter all your teammates' handles too. It will try to select problems from different topics. This and all other recommendations are based on your past submissions. So, the difficulty of the set will vary according to your past submissions (i.e if you have solved a lot of hard problems, then you will get harder problem recommendations, so tourist's easy problem will be very different from my easy problem recommendation :) )

Mini Contest: Best for individual coders. 5 problems are recommended, again based on the coders' past submission.

Daily Practice: Suggests only 2 problems — an easy and a hard. This is useful for those who don't have that much time to practice.

Warmup: 2 easy problems for solving before some contest.

Easy/Medium/Hard: 10 problems of the chosen difficulty.

Topic based: Select a topic and start solving from easy to hard problems on that topic.

This was all coded in one day, so expect some bugs :) . I will try to keep adding more features and other sites for analysis. Comments and suggestions are welcome.

  • Vote: I like it
  • +583
  • Vote: I do not like it

Comments Comments (91)
| Write comment?
»
8 years ago, # |
  Vote: I like it +28 Vote: I do not like it

I'm just curious, what's the algorithm you used to determine the best problem? Is it by solve rate, difficulty, etc?

And how to determine the "cutoffs" for easy/med/hard? (If some cutoff values are used to determine these problems)

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +17 Vote: I do not like it

    The cutoffs are relative — based on the coder's solved submissions difficulty. Difficulty of a problem is approximated by the number of users who solved that problem.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it +5 Vote: I do not like it

      Wouldn't it cause problems from a few years ago be considered hard as codefoces was less popular than?

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
        ← Rev. 2   Vote: I like it +3 Vote: I do not like it

        Yes you are correct. Older problems may be considered harder. We are working on improving the recommendation algorithm to consider that. Interestingly, spoj has the exact opposite issue, older problems are solved more often so may be considered easier than they actually are.

        → Reply
        • »
          »
          »
          »
          »
          8 years ago, # ^ |
            Vote: I like it +9 Vote: I do not like it

          Another way to estimate difficulty would be to see the ratio of its solvers officially during contest vs the solvers of Div2A of that contest (this assumes that everyone who takes part is able to solve div2A easily). Div1 D and E can be compared with Div1 A (after appropriate scaling with respect to solvers of div2 C).

          This may give inaccurate results in case of Div2 E or Div1 D or E, which are sometimes easier but many weaker contestants only try them after contest. So another contributing factor can be to compare solvers after contest with the solvers div2 A after (or during) contest.

          → Reply
          • »
            »
            »
            »
            »
            »
            8 years ago, # ^ |
              Vote: I like it +6 Vote: I do not like it

            Some contests have unusually difficult Div2As, I think that it would be better to compare it to the number of participants.

            → Reply
        • »
          »
          »
          »
          »
          8 years ago, # ^ |
            Vote: I like it +4 Vote: I do not like it

          maybe this blog can help http://codeforces.net/blog/entry/46304

          → Reply
        • »
          »
          »
          »
          »
          8 years ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          You could try to approximate the difficulty of a problem based on the total number of AC submissions during a contest vs the total number of participants in that round.

          → Reply
»
8 years ago, # |
  Vote: I like it +13 Vote: I do not like it

Nice initiative!

→ Reply
»
8 years ago, # |
  Vote: I like it +8 Vote: I do not like it

Every time I refresh my page, the recommended problems are changing frequently (Even though I didn't submit any solution). Is that a bug or a feature?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +30 Vote: I do not like it

    The problems are chosen randomly after cutoffs, so it changes every time the page is refreshed. This is because, if you don't like a particular set, you can just refresh and get a new set. Also, due to this, I need not store anything in the backend :).

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Is it possible to save or download a particular problem set instead of manually adding each of them to favorites?

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        That is a nice suggestion. We will add it in the next release (probably by creating a permalink for each problem set).

        → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Codeforces shows that I have solved 748 problems, while the site shows that as 712. Any reason why?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +3 Vote: I do not like it

    Yes I noticed the discrepancy in my data too. I am not sure why that happens. I just use the data codeforces returns in its API and count the total number of distinct AC problems. Still looking into it.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it +9 Vote: I do not like it

      In this site we see the number of problems which we solved (without Gym ones) ... maybe that is why it is differ

      → Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +6 Vote: I do not like it

    Maybe Codeforces count Div1A and Div2C as two different problems?

    → Reply
»
8 years ago, # |
  Vote: I like it -63 Vote: I do not like it

I am a new programmer.Very few algorithms i knew.How I improve my coding skill and i will be a good programmer??

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +4 Vote: I do not like it

    practice practice and practice

    → Reply
  • »
    »
    8 years ago, # ^ |
    ← Rev. 2   Vote: I like it -15 Vote: I do not like it

    Here is a big group in bangla for competitive programming! :) as you are Bengali, You need to join this group and read+follow the pinned post.

    https://www.facebook.com/groups/bengaliprogramming/

    → Reply
  • »
    »
    8 years ago, # ^ |
    ← Rev. 2   Vote: I like it 0 Vote: I do not like it

    As you are new programmer. First you need to learn some essential algorithms.

    Competitive Programming 3 by Stevien & Felix Halim Is the best for beginner.

    Also you can learn from geeksforgeeks.com and so other sites. Just search google.

    And to improve your coding skills, you need to practice more and more :)

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it -20 Vote: I do not like it

      Rezwan bro, direct 3? may I know why didn't you preferred 1st and 2nd part of this book?

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        The newer editions are not intended to be read as sequels with regards to the previous editions-they are just that, newer and improved versions, containing everything the previous ones have, and much more.

        → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

The site looks awesome and is fast as hell.

But I'd request to include user's spoj performance for submission analysis for recommendations.

Thanks anyway!

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Thanks! I am planning to include other sites' data too. There are some issues to be solved in that (how to assign difficulty across sites) so I did not do it in this first iteration.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
      ← Rev. 2   Vote: I like it -41 Vote: I do not like it

      CHOOSE ONE SITE AS STANDARD(E.G. CF RATING)

      X= NO OF USER GOT AC;

      Y= NO OF USER TRIED;

      M= SUM OF THOSE USERS WHO GOT AC THIER RATING ON STANDARD SITE

      DIFFICULTY= (X/Y)*SQUARE_ROOT(M);

      BEFORE THIS YOU HAVE TO ASSIGN DIFFICULTY IN SAME WAY TO ALL PROBLEMS OF STANDARD SITE

      NOW DO YOUR ALGORTHM WHAT YOU WERE DOING.

      TALK WITH ahmed_aly

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it -10 Vote: I do not like it

        why am i getting downvotes plz reply

        → Reply
        • »
          »
          »
          »
          »
          8 years ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          Because you need more practice :P

          → Reply
        • »
          »
          »
          »
          »
          8 years ago, # ^ |
            Vote: I like it 0 Vote: I do not like it
          • Everything is in caps
          • There are lot of typos, it's hard to understand and there is no formatting at all
          • For me it's not clear, what is M
          • Difficulty calculation makes no sense at all, no explanation, why it should be used and why this assignation will be right. For me it looks like random formula
          → Reply
»
8 years ago, # |
  Vote: I like it +1 Vote: I do not like it

The recommendation also includes solved problems. Is that intended?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    No it is not :(. Will look into it. Can you tell me which tag this issue appeared on (i.e. icpc or easy or some problem tag)?

    → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

where to view how many submissions i got ac ?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Codeforces returns ac as "OK" so that is the number of submissions you got AC for.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      where can i see no total on your site

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it +1 Vote: I do not like it

        It is in the chart titled "Submission Results". The green slice in that pie chart labelled OK.

        → Reply
»
8 years ago, # |
  Vote: I like it -8 Vote: I do not like it

Have you used the Collaborative recommendation engine algorithm like "Programmer who solved Problem X also solved Problem Y." to provide suggestions ?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    No, we just consider the given coder's data. A collaborative recommendation engine will require a lot more infra and coding than this model. I can't decide if that will be useful here. The aim is to suggest problems to the coder which will be just beyond their difficulty level.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Yes You're right. Programmer's growth is not guaranteed in Collaborative model.

      → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      It would be nice if we can see what all problems were solved under each category if we click on the category in that pie chart.that will also help someone following others to see what problems others have solved under that category so that we can follow some experienced coders

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Nice idea, will try to implement it. Will take some time as we need to start storing data in db for that to work.

        → Reply
»
8 years ago, # |
← Rev. 2   Vote: I like it 0 Vote: I do not like it

Wow, this is awesome thank you for your hardwork. I wish you good luck. but I have 2 questions: first : on the daily practice tag it shows two problems a div 2 A and a div 1 E that is not good for practice I mean seriously div 1 E??? second : on the chart of submissions it shows 10 submissions as challenged what does it mean ??? :-)

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Hi, the intention behind daily practice is to solve a hard problem daily. The hard problem is chosen to be just beyond the coder's level. If we keep solving problems that are within our level, we can't improve. The easy problem is just there for warm up.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Yes you are right we should solve harder problems but we shouldn't jump a head of our selfs like for example I can solve div 2D usually so the problems that should be in daily practice are div 2 E anyway that's just what I think thank you for this hard work but... you still hadn't answered my second question :-)

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Challenged is when a submission in a contest gets hacked by others.

        → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        About that div2 E recommendation, we only consider the number of coders who solved a problem as the indicator of the problem difficulty. So this particular div2 E may have been as easy or just beyond an average Div2 D and so has been recommended to you.

        → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Nice one. I was looking for the same to practice .I have a query, Suppose a problem involves two categories (say dp and number theory) , did you include this problem in both categories ?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Yes it is considered to be in both categories.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      I found a bug . It is recommending some solved problems, maybe because it is considering div2 C and div1 A as two different problems.

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Yes Emphi reported the same bug. This is probably the cause of it. Will fix that soon. Thanks for reporting :)

        → Reply
»
8 years ago, # |
  Vote: I like it +10 Vote: I do not like it

I was looking for something like that for so long ... thank you for your efforts :)

→ Reply
»
8 years ago, # |
  Vote: I like it +8 Vote: I do not like it

wow... amazing

→ Reply
»
8 years ago, # |
  Vote: I like it +6 Vote: I do not like it

I went for warm-up suggestion, and the website suggested me a Div1-C and a Div1-E...

Anyway, I like the idea of this website. It would come handy for every competitive programmer.

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +9 Vote: I do not like it

    Oops! There are a bunch of bugs around the recommendations. Will try to fix all of them asap. Thanks!

    → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Every time I open the site and enter my user handle, I see a different list than what I saw the last time. So the site picks up random problems based on their difficulty level and are the problems solved or not. It's a really great initiative. Try to include spoj in it.

→ Reply
»
8 years ago, # |
  Vote: I like it +5 Vote: I do not like it

Wow, fantastic website. Hope daily pratice will have more problems because I am going in to take part in VOI so I need to pratice more and more. Or simply, I can chose how many problem to deal a day.

Anyway, thank you a lot Balajiganapathi.

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +6 Vote: I do not like it

    Hi, thanks :). If you have more time to practice daily, then "Mini contest" is the recommended one. "Daily Practice" is for those who want to solve just one problem a day.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
      ← Rev. 2   Vote: I like it +11 Vote: I do not like it

      Thank you very much. I just think that daily pratice is for someone who want to pratice everyday, everytime.

      → Reply
»
8 years ago, # |
  Vote: I like it +11 Vote: I do not like it

Awesome website and it's very fast. Thanks Balajiganapathi for this amazing site.

→ Reply
»
8 years ago, # |
  Vote: I like it +5 Vote: I do not like it

nice ! and thanks a lot :)

→ Reply
»
8 years ago, # |
  Vote: I like it +5 Vote: I do not like it

That's cool.

→ Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

it is suggesting me to solve a Div1A problem that i've already solved in Div2C. if you can solve this bug it would be much helpful. :) Balajiganapathi

→ Reply
»
8 years ago, # |
  Vote: I like it +11 Vote: I do not like it

Very good site in my opinion, but one thing you should consider is not to show Surprise type rounds. For example my daily practice included this problem.

→ Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Great work Sir. Please include Codechef too. It will be helpful.

→ Reply
»
8 years ago, # |
  Vote: I like it +3 Vote: I do not like it

Excellent idea. But I'm facing some serious issues here. I've solved problems from multiple handles. :( Now the recommendation will be 1/n times as accurate as it should be, where n=no. of my handles. Does inserting various handles produce a cumulative result, or average, median, etc.?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +12 Vote: I do not like it

    It will consider all entered handles as a single unit. So, it should work.

    → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

I noticed a bug: If we try to input the name of a user whose submission(s) is(are) "in queue", the site gives this error. I know it's a minor issue since the problems generally tend to stay in the queue for a very short period of time, but I thought I'd report the same.

→ Reply
»
8 years ago, # |
  Vote: I like it +5 Vote: I do not like it

I think that there is a small bug: your site suggests me solving this Div1 C problem but I have already solved this problem as Div2 E.
Anyway, great site.

→ Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

This doesn't work for codechef anymore :/

→ Reply
»
8 years ago, # |
  Vote: I like it +31 Vote: I do not like it

just curious here, how are strong areas and weak areas determined?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I guess ratio of AC to WA/TLE in each tag maybe? :/

    → Reply
»
8 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Nice job! Are you planning open-source your codes on your github page?

→ Reply
»
7 years ago, # |
  Vote: I like it +5 Vote: I do not like it

Hasn't been working for the past 2 or 3 days :( It says RTE

→ Reply
»
7 years ago, # |
← Rev. 5   Vote: I like it +5 Vote: I do not like it

The site doesn't work for CodeForces handles, but works for other Online Judges handles. It is nice. I wish it would categorize many problems according to topics. Including Problems solved from UVa and Open Kattis would be a great feature.

→ Reply
  • »
    »
    7 years ago, # ^ |
      Vote: I like it +1 Vote: I do not like it

    Fixed it for cf handles. CF changed their API response format hence the error. I will look into adding UVa and open kattis — should not be hard to add.

    → Reply
»
7 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Hey ! Wonderful work ! The website is useful to me and is helping me find new problems ...

Just one suggestion — Try to include Hacker Rank and Hacker Earth handles for analysis as well !

→ Reply
»
7 years ago, # |
  Vote: I like it +19 Vote: I do not like it

The site has been down for a few days now! Can you please look into it?

→ Reply
»
6 years ago, # |
  Vote: I like it +3 Vote: I do not like it

Is the site down? I am unable to open my Codeforces Account in it

→ Reply
»
21 month(s) ago, # |
  Vote: I like it +3 Vote: I do not like it

your site isn't working, can you fix it? it shows:

Oops! Something went wrong while processing your request. Mark this as a RTE against us :(

→ Reply
»
12 months ago, # |
  Vote: I like it 0 Vote: I do not like it

This site is down for long days. Is there any update regarding this problem?

→ Reply