efimovpaul's blog

Stress-testing
By efimovpaul, history, 3 years ago, In English

Hello, Codeforces!!


Guys, how do you stress-test your solutions? I know at least three possible ways to do it.

1)Right in your solution file
Write second stress-solution ( definitly correct, but quite slow ), generator, checker, ect. right in your solution file as functions and call functions from main.
I personally use this variant, it is fast and convinient. But there is one problem using this method.. When you have your solution fixed, you have to comment all your stress-code.. I have a trick: just make copy of your file before adding stress-testing functions and add them in your second file, after debugging you will have to just fix jour solution in the first file.

2)Python script
Some people write Python scripts to stress their solutions. You have two programs: solution with bug and stress-solution. So you can just generate tests, lauch your solutions, check answers, etc. from Python script.

3)Bash or maybe Bat scripts
Method is idiologically similar to the 2)Python script, but you write bash/bat scripts.


Which method do you use? Or maybe you have your own techniques, share in comments.

  • Vote: I like it
  • +54
  • Vote: I do not like it

Comments Comments (36)
| Write comment?
»
3 years ago, # |
  Vote: I like it +95 Vote: I do not like it

Too much stress can be detrimental to their performance. Codes need a caring and minimal stress environment to foster their growth.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +1 Vote: I do not like it

    aahahha, nice joke
    But I think you are right, sometimes it is better to try to talk to it, look, search for bug just watching code

    → Reply
»
3 years ago, # |
  Vote: I like it +32 Vote: I do not like it

Big eye big brain method: Try to come up with nasty inputs and run them.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it +3 Vote: I do not like it

    Yeah!! Of course you are right. It is supposed that you have already tried some tricky cases before you write stress tests.

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
      ← Rev. 2   Vote: I like it +6 Vote: I do not like it

      Oh, I misread. In China, when participating in the provincial/national Olympiad, we do something that is called '对拍' which is basically writing a brute force solution and a generator, then using a bat script to connect those three and try to find a case that fail on the better solution. You have to be absolutely sure that the brute force solution is correct though. That's probably a mix of (1) and (3) mentioned above.

      → Reply
      • »
        »
        »
        »
        3 years ago, # ^ |
          Vote: I like it +8 Vote: I do not like it

        hm.. Is using Bat script to combine solutions and generator more convenient than just calling them from main program (of course I am talking about cases when all functions are written using one programming language) ?

        → Reply
        • »
          »
          »
          »
          »
          3 years ago, # ^ |
            Vote: I like it +3 Vote: I do not like it

          Well that's probably because we don't use 'void solve()' or something like this, we write a ton of stuff in main(). Maybe I can try calling from main program later...

          → Reply
          • »
            »
            »
            »
            »
            »
            3 years ago, # ^ |
              Vote: I like it 0 Vote: I do not like it

            When I have tried writing all staff in other function and kerping main function minimalized, I found it really convenient

            → Reply
»
3 years ago, # |
  Vote: I like it +11 Vote: I do not like it

I use FastOlympicCoding's stress in sublime text. And it's usage is like the second one. All we need is a correct solution, a wrong solution and a generator file.

→ Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Hi! Firstly I use ubuntu so the bash script may be diffrent if you use windows.I published the codes here.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    in Windows it is Bat scripts, they are a bit different, but I think it Will be interesting to try to use your script on Linux, thank you!!

    → Reply
»
3 years ago, # |
  Vote: I like it +6 Vote: I do not like it

Check this

→ Reply
»
3 years ago, # |
← Rev. 2   Vote: I like it 0 Vote: I do not like it

Here's the bash command that I use for stress testing.

Command
→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    do you understand what is happening there? I will be glad if you give me some good resources to learn bash commands, I was quiet surprised when I found out that it is not easy to find full bash-learning courses and videos.

    → Reply
»
3 years ago, # |
  Vote: I like it +8 Vote: I do not like it

Using https://polygon.codeforces.com/ is also an option

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    how can I use CF Polygon to stress-test solutions?

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Generate tests and ran an invocation, as you would normally do when preparing a problem.

      → Reply
      • »
        »
        »
        »
        3 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        a, yeah, I got it. Do you really use this method during contests?

        → Reply
        • »
          »
          »
          »
          »
          3 years ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          I don't really use stress testing during contests at all) But I use this method during practice

          → Reply
»
3 years ago, # |
  Vote: I like it +3 Vote: I do not like it

All I do is write brute force and use this and this

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    you do not even write scripts connecting these utilities? is stress-testing fast enough using this method?

    → Reply
»
3 years ago, # |
  Vote: I like it +8 Vote: I do not like it

I've developed my own system for stress testing, which also allows me to do a ton of customizations, and run multiple tests in parallel.

cfstress.com

Although behind the scenes, it'd fall under category 3.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    wow, nice job!! is only online version available? online versions are not allowed to use during some olympiads, maybe you have something like "portable version"?

    → Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    in addition to the previous comment, it asks the number of CF problem, so is it important for problem to exist on CF? you get input format from there? is it possible to make your utility work using just problem's statement text?

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      No, it's not important for the problem to exist on Codeforces (for example, I also use it for Atcoder Beginner Contests). But that feature is only available on the portable version.

      However, don't get the impression that it uses Machine Learning to parse problems and generate test automatically. It doesn't. But I have created various scripts, header files and macros that allow me to feed the data in a fixed format and it generates the test cases off of that. There's definitely a little coding required for each problem, but it's just 5 to 10 minutes of effort per problem. (If it takes more than 15 minutes, it's not worth stress testing IMO).

      For example, for the last Edu Round, it took ms around 30 minutes to create generators for all 6 problems.

      → Reply
      • »
        »
        »
        »
        3 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Yeah, I got it, so 30 minutes for 6 problems is a good result.

        BTW: Did you try to use ML somewhere in Competitive Programming?

        → Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

I use CodePal

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    it is only for VS code, isn't it?

    → Reply
    • »
      »
      »
      3 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Yep. You dont have to use all the featues of the extension. For example, I mainly use CodePal for stress testing and Atcoder Library Support.

      → Reply
»
3 years ago, # |
← Rev. 3   Vote: I like it +3 Vote: I do not like it

I can suggest IFDEF for C++, and if statement check for python. If you write ur code in python, you can create decorator which checks if code is used locally (for example using os.getlogin()), and place it right before ur stress tests and they will be skipped. 

import os

def ignore(func):
    if os.getlogin() == "doreshnikov":
        func()
    else:
        pass

@ignore
def stress():
    print(123)
→ Reply
»
3 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Why not to always use first option + #ifdef? The only downside is that you should be a bit more careful with random and multitests, but if those problems arise, then you are probably doing something wrong.

→ Reply
  • »
    »
    3 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    yes, I think that it is a good solution to use ifdef some people offered it, I will try this one. Thanks!

    → Reply