Um_nik's blog

Changes in CodeChef Problemsetting
By Um_nik, history, 3 years ago, In English

Hello Codeforces!

For the last half a year I have been working on CodeChef and was proposing ideas on how to change the problemsetting practices (hopefully, for the best). For some time the changes were in brainstorming phase, but as of now many of them are implemented. Today we are ready to share the information with you. But first I would like to approach people who are going to close this blog without reading:

If you are not interested in CodeChef contests at all, I think you should change your mind. Monthly short contests (Cook-Off and Lunchtime) have interesting quality problems, on par with Codeforces in my opinion (not AtCoder level yet, but who are?). Long and Starters are more targeted to newbie participants, with more classical educational problems.

If you are interested in setting problems for regular contests, you might want to do it on CodeChef. Some reasons:

  • You don't have to set the whole round, as we work on per problem basis.
    • This may be especially interesting for proposers of hard problems — you don't have to come up with easy problems. You can also propose a lot of problems on your favourite topics, as we won't put all of them in one contest.
  • Your problem will be reviewed by me, discussed with me and then maybe enhanced. You might have different views on my persona, but it is hard to deny that I'm pretty good at this stuff.
  • In normal circumstances your problem will be reviewed less than a week since submitting proposal. (You might have to wait for it to be used in contest though)
    • We are pretty low on harder problems, so if you propose a (good) Medium+ problem you can expect it to be used soon.
  • We pay more. I know, I know, we do it not for the money. But it is a nice bonus, isn't it?

With that sorted, let's take a closer look at what have we changed.

Some of the changes are internal and it's hard to show them, but hopefully those changes will simplify the life of contest admins and they will have more time and energy to work with problemsetters and oversee the contest preparation.

  • We have separated reviewing problem proposals and admin-ing (coordinating) contests. Now we have a special person, whose job is to review proposals and try to improve them. That person is me right now, although I hope that this system will outlive me, as it is more fair and leads to better problems. Yes, better problems, as my job is not only reviewing the proposals, but also thinking on ways how to make them better. Most of the short contests in 2021 had at least one hard problem significantly improved during the review.
  • Making problems better requires collaboration with setters, which starts from communication. We used emails before, and it was... not ideal. Now we have created a dedicated Discord server, where we create private channels for me to communicate with setters. On the same Discord server we create channels with all the setters for a particular contest, where they will be able to talk with admin, tester, and generally people working on the contest. Moreover, if there is an issue that requires some managing help (like providing access) it will be easy to ping managers. I will not put a link to the server here, and I ask setters who has it to not do it too: we want only people who are interested in problemsetting there.
  • Many people hate Campus (the system which was used for preparing problems on CodeChef), and rightfully so. We don't like it either. That's why we have gotten rid of the entire UI and have built a new problemsetting portal from scratch. We have implemented many essential features which were previously lacking (bulk uploading of test files, bulk editing of time limits, and judges, statement history versions, copyable sample IO, etc), and are working on adding the missing features as well. We know we still aren't as good as Polygon, but we are working towards it. We ask setters to prepare the problem on Polygon, and then port them to CodeChef.
  • This is an internal thing, but I'm very proud of it. In general, we have a big waiting list of problems that are approved but not yet used for contests. When we need to make the next contest, an Admin is assigned, and he then chooses the problem from the waiting list. This waiting list was stored in a Google Sheet, it was hard to maintain and even harder to choose problems from it, and mostly we had to rely on our memory to choose problemsets. Now the waiting list will be stored in Notion. We will have to migrate the waiting list manually, and this work is far from done, but the future of stuck-in-the-waiting-list problems looks a bit brighter now, as we will be able to actually consider them while choosing problemsets.
  • The CodeChef problem setting portal is still not a very user friendly system. And to use it you have to somehow learn how to do it. There is a guide for it... accessible only by a link you have to somehow get. Also this guide is written several years ago. Yeah. Actually, there are more different guides, sometimes contradicting each other. But it's not much of a problem, you have close to 0 chance to find them. Well, this had to stop. Either by CodeChef dying out because the link to the guide was lost and now nobody knows how to upload problems, or by us making an effort. We have chosen the second way, and it's Notion to the rescue again. Introducing CodeChef Public Guides. It's very new, but at least there are links to all the old guides I could find. We are planning to write more articles on all things related to CodeChef problemsetting, and maintain them in relevant state. Right now there are articles on step-by-step process of setting a problem and what (not) to do when proposing a problem. I'm planning to write an article on good practices when working with Polygon next.
  • We have increased our judge server capacity and have not had a queue issue in a long while. We are in the process of scaling it 4x (and more) in the coming months, which will allow our setters to not have to worry about keeping the testfiles to a minimum.

In case I have managed to convince you to become a problem setter on CodeChef, here is the link. But please read this first.

What changes would you like to see on CodeChef, either in problemsetting or in general? Please help us make CodeChef better by filling this form.

  • Vote: I like it
  • +632
  • Vote: I do not like it

Comments Comments (53)
| Write comment?
»
3 years ago, # |
  Vote: I like it +94 Vote: I do not like it

I just want to thanks Um_nik , from the time he joined codechef , none of the contest has went unrated and problem quality has also improved , also also proposed problem is getting reviewed very fast unlike earlier , idk if everything is possible due to Um_nik only but still a lot credit goes to him

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +106 Vote: I do not like it

    This is actually not true, the very first contest where I was contest admin had severe queue issues and was unrated, a lot has changed since. Not all of it is due to me, but thanks!

    → Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +27 Vote: I do not like it

    Why post it with an alt though? XD

    Also um_nik, thanks for your efforts as codechef contests have become more interesting recently.

    One thing that I'd love to see is that there should be multiple testcases shown in the verdict, sometimes it is very irritating to see just one testcase pop up as WA. I have filled it in that form.

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
        Vote: I like it +4 Vote: I do not like it

      What do you mean exactly?

      → Reply
      • »
        »
        »
        »
        3 years ago, # ^ |
        ← Rev. 3   Vote: I like it -56 Vote: I do not like it

        Hii Um_nik there are 3-4 rated contest for div2 I believe per month, If it can be increased to 6-7 would be great. Correct me if i am wrong somewhere.

        Also, Is not the 10 days too long for a long contest ? I believe 7 is still good. Like atcoder heuristic. Everyone gets a chance to attempt at least once in week.

        In short:

        • Shorter long contest (7 days).
        • More div2 contests (6-7) per month.
        • Try working out a better front end design for website. (front page is a bit messy -- I personally never go through it -- okay, I never go through any of these on any of websites :P).
        • Do not know if it is feasible or not, but if user editorials can be encouraged like in leetcode (I mean you can sort them by most upvotes).
        → Reply
        • »
          »
          »
          »
          »
          3 years ago, # ^ |
            Vote: I like it +17 Vote: I do not like it

          You make it sound like making contests is a daily homework assignment

          → Reply
    • »
      »
      »
      3 years ago, # ^ |
      ← Rev. 2   Vote: I like it +23 Vote: I do not like it

      Yeah in a short contest they don't show verdicts of individual TC's just WA or TLE or AC. We want something like Atcoder or CF. Like how many were AC or up to which tc it ran if others are skipped?

      → Reply
      • »
        »
        »
        »
        3 years ago, # ^ |
          Vote: I like it +20 Vote: I do not like it

        If I'm not mistaken, you get full feedback in IOI style scoring contests (Long, Lunchtime). For ICPC style scoring only the verdict is shown, that's true. But that's just a design choice, I don't see why it is much worse than providing no. of the failed test. And since usually there is multitest on CodeChef, in the case of WA it won't say much. I understand that it is less convenient for participants, but I don't think that it's something that needs to be fixed.

        → Reply
        • »
          »
          »
          »
          »
          3 years ago, # ^ |
            Vote: I like it +5 Vote: I do not like it

          It would be nice to have that addition (not fix), but we don't necessarily need that feature / design.

          → Reply
»
3 years ago, # |
  Vote: I like it +19 Vote: I do not like it

Just for your information. When I was a CodeChef admin, I have tried to create something similar to your CodeChef Public Guides here. Most likely you will find no additional value there now, but at the very least I think that the generic input file formatting validator might still be useful.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +24 Vote: I do not like it

    Thanks!

    → Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +26 Vote: I do not like it

    Btw one note about that input validator. The way it is written now, it doesnt work properly to detect '\r\n' if ran on a windows system. This is because sys.stdin.readlines() on windows automatically switch out '\r\n' to '\n'.

    The fix is to use sys.stdin.buffer.readlines() which returns a list of byte strings. 

    lines = sys.stdin.buffer.readlines()

if any(line[-2:] == b"\r\n" for line in lines):
	raise Exception("The input file contains Windows-style line endings!")
...

    sys.stdin is just a wrapper for sys.stdin.buffer that converts the raw bytes read from sys.stdin.buffer into unicode (while also stripping away the \r if ran on a windows system).

    → Reply
»
3 years ago, # |
  Vote: I like it +8 Vote: I do not like it

Are you planning to increase the frequency of Div-1 contests?

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +38 Vote: I do not like it

    We hope that we will be able to, but it won't be possible without increasing the number of high-quality hard problems from you guys.

    → Reply
»
3 years ago, # |
  Vote: I like it +9 Vote: I do not like it

Um_nik Everthing is fine, but I think the admin should do the plagiarism checks first. Most of the problems solutions are available on Youtube , Telegram etc along with ideone links for both Lunchtime & Cookoff. Hence many have lost their interest in giving these contests. The last plagiarism check was done as far as I remember was on August 2020. After that nothing has been done. Link

I think instead of doing the plagiarism checks after every 2 years, if its done after every contest then it would make a lot of sense.

→ Reply
»
3 years ago, # |
  Vote: I like it +1 Vote: I do not like it

Can you add something to notify if the solution fails the sample case and not give penalty for the same? This is helpful in problems having multiple outputs and interactive problems.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +1 Vote: I do not like it

    Currently, we don't run your solution on sample cases in the system at all.

    → Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +25 Vote: I do not like it

    It would be nice to have this feature, but you can work your way around that, and it's not the end of the world if you waste 1 or 2 minutes verifying your solution on samples (this is much less than the WA penalty).

    → Reply
»
3 years ago, # |
← Rev. 2   Vote: I like it +244 Vote: I do not like it

drawing

→ Reply
»
3 years ago, # |
← Rev. 2   Vote: I like it +18 Vote: I do not like it

I request Codechef for another feature that is the announcement. As of now, announcements can only be seen on the dashboard of the contest. But it is better if it comes as a pop up on any page of codechef (like CF). Sometimes we miss announcements and see them very late.

→ Reply
»
3 years ago, # |
  Vote: I like it +90 Vote: I do not like it

Meanwhile TopCoder...

→ Reply
»
3 years ago, # |
  Vote: I like it +21 Vote: I do not like it

→ Reply
»
3 years ago, # |
← Rev. 2   Vote: I like it 0 Vote: I do not like it

I know this is a difficult task but would it be possible for you to give some feedback for rejected problems as well (on mail)?

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +19 Vote: I do not like it

    When I submitted a problem in May, Um_nik personally gave feedback about why the problem was rejected (too much implementation). However it seems August onwards the feedback is limited to a email stating that the problem has to be turned down as it could not pass the review process. It would be beneficial for new problem setters if they were given feedback about the problem so that they could try to come up with problem that are in line with the reviewer's expectation.

    The feedback also helps in unexpected manners. For example when I submitted my second problem I entered a wrong email address. However Um_nik personally reviewed the problem and decided to forward me the email (assuming correctly that I was the problem author) stating that my problem was approved as simple. This was so good of him to do and something I am grateful for.

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
        Vote: I like it +48 Vote: I do not like it

      Yes, I will need to improve in this area. Before we had to respond to all the proposals in email, while now I just press one button to either generate a channel in Discord or send an automatic rejection letter. I would love to provide reasoning for all rejections, but most of the time it's just "it's boring" or "it was used before: link". It takes time which is not very productive and "disrupts the flow of reviewing".

      I will surely answer if you respond to the rejection email demanding explanation.

      → Reply
»
3 years ago, # |
  Vote: I like it +87 Vote: I do not like it

Lifehack: if you came up to some interesting problem but doesn't came up to nonshit solution, just suggest it to codechef and check good solution from Um_nik!

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +55 Vote: I do not like it

    He knows what he's talking about :) Except his initial solution wasn't shit, but my was cooler
    I was actually a bit scared that it might be a good tactic to cheat on some Long contests...

    → Reply
»
3 years ago, # |
  Vote: I like it +1 Vote: I do not like it

The quality of problems of Codechef is incredibly high nowadays. Thank you Um_nik.

→ Reply
»
3 years ago, # |
← Rev. 3   Vote: I like it 0 Vote: I do not like it

What does "Pre-requisite(s) required to solve the problem" in the form above mean (sry im not a native speaker)Um_nik

→ Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Is it fine if we share links to text files containing the respective data for — Problem Statement, Constraints, Expected Solution etc. in the Form?

→ Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

How we know that our idea rejected or accepted sir ?? i am new, so asking :'(

→ Reply
»
3 years ago, # |
  Vote: I like it +36 Vote: I do not like it

Are there any plans to restore the classic 2010-2020 long challenges, with exact problems strong enough for DivA (DP on chordal graphs, Faigle-Kern, queries on top trees) and an optimization tie-breaker?

I think these long challenges were what really made CodeChef unique. They drew some very strong participants, and each month you can still see some reds doing the new Div3 long challenges. I think they are all hoping for the real long contest to come back.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +2 Vote: I do not like it

    Based on Um_nik's Things I don't know post I feel like he's not a huge fan of stuff that requires advanced algo/ds knowledge.

    Aside: Can you link the problem for queries on top trees? I've been looking for some intro problems for learning top trees after ecnerwala mentioned it in: https://codeforces.net/blog/entry/94726?#comment-838868

    → Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I can't answer that. I'm not qualified to judge problems on obscure data structures, problems based on articles and optimization problems.

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
        Vote: I like it +16 Vote: I do not like it

      OK.

      This CodeChef page for problemsetters still has information about Super-Hard and Challenge problems: https://www.codechef.com/problemsetting/

      Is it the case that CodeChef continues to receive such problems intended for a DivA long challenge, but there is nobody qualified to judge them so they get rejected/ignored?

      → Reply
      • »
        »
        »
        »
        3 years ago, # ^ |
          Vote: I like it +10 Vote: I do not like it

        I think we haven't received proposals for any of these two categories in half a year.

        → Reply
        • »
          »
          »
          »
          »
          3 years ago, # ^ |
            Vote: I like it +6 Vote: I do not like it

          If you received enough good problems for these categories, would you be willing to organize a DivA rated Long Challenge?

          I think a possible reason for not receiving problems is that it's not clear if CodeChef is even interested. It takes a lot of effort to create such problems. A clear message saying "yes, we will organize a DivA Long Challenge if we get problems" would be good to see.

          → Reply
          • »
            »
            »
            »
            »
            »
            3 years ago, # ^ |
              Vote: I like it +26 Vote: I do not like it

            I'm sorry, I can't answer that. There are different issues with Long contests as they were before, I don't know if I'm allowed to disclose information about them.

            → Reply
            • »
              »
              »
              »
              »
              »
              »
              3 years ago, # ^ |
                Vote: I like it +9 Vote: I do not like it

              That's very sad to hear. :(

              But thank you for the clear answer.

              → Reply
»
3 years ago, # |
  Vote: I like it +9 Vote: I do not like it

Monthly short contests (Cook-Off and Lunchtime) <...> on par with Codeforces

not AtCoder level yet, but who are?

applies transitivity

You'd better hope Codeforces worshippers don't read too much between the lines :)

→ Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

One of my problems were approved by you at March 30, 2021. Then I applied for campus access, sent signed NDA but didn't get any response after sending NDA.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Right. Yeah. There is a huge issue with porting old proposals to our new database. It will take some time.

    → Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

CodeChef has nice problems.

→ Reply