Блог пользователя Not-Afraid

Автор Not-Afraid, история, 4 года назад, По-английски

Hello Codeforces community.


Tl;dr: The script is specifically for *Unix and can be used to stress test `C++` solutions. It can be easily modified to work with python or java. To use it in windows you can download Ubuntu terminal which is officially available on Microsoft Store and then use it or you can modify the script to work in cmd by changing the .out files to .exe and a bit of syntax for cmd.

Note: If you don't want to read this lengthy blog, just go to this repo and follow the instructions of readme.

First let me tell you what stress testing is if you don't know, Stress testing is technique using which we can run our solution (which we think might fail/is failing) on a number of random test cases and match it's output with the output of a solution which is a brute force solution or accepted solution of someone else.
If you don't know why and when to use stress testing you can use this detailed article on Stress Testing by ADJA.

I want to share my script for stress testing your solution.

Requirements:
1) A solution which we want to test.
2) A brute force solution which gives correct solution.
3) A generator for generating test cases according to the problem.

About script:

code

Go through the code once, i have added comments and you will understand most of the part. To understand the script all we need is just basics of the language bash. So we have three .cpp files:


1) gen.cpp // our cpp solution for generating test cases 2) solution.cpp // our orignial solution 3) brute.cpp // our solution which uses brute force and guaranteed to give correct output

About test case generator:
For generating test cases you can either use "testlib.h" here (i personally don't use it because it takes 6-7 seconds on average to compile in my pc(idk there might be ways to reduce it) and also i need to remember the methods to generate things.) or write your own since most of the time you just need arrays, strings, trees or simple graphs or simple some integers.

I use the below code for generating test cases most of the times(file gen.cpp):

Code

You can use above code and modify it according to your needs. Just go through the code once and everything is self explanatory(i have added some comments too). Usage of above gen.cpp code:


1) to generate an array calling: gen_array(10, -5, 10); it will return an array(vector more specifically) with length 10 and elements in the range [-5, 10]. 2) to generate a tree calling: gen_tree(10): will return a tree with 10 nodes. 3) to generate a simple graph calling: gen_simple_graph(10, 12); will return a simple connected graph with 10 nodes and 12 edges. You can add things as you need them or use testlib which is the best if you know how to use it.

Usage:
Download this repository manually or by using git clone on terminal.

  • Copy your original solution which you expect might fail in the file solution.cpp.
  • Copy your brute force solution which is expected to give correct output in the file brute.cpp.
  • Change the gen.cpp file so as to generate test cases according to the question.

Now open your terminal in the directory where file s.sh resides and run command:
$ bash s.sh
or
$ ./s.sh
If it shows permission denied then give it execute permission using:
$ sudo chmod +x s.sh.

In file s.sh you can change the value of variable max_tests as you wish on how many random test you want to run the solution.

Verdict: the verdict of running file s.sh on every test case is either Accepted if your solution's output matches the brute solution output or Wrong Answer and will show the input on which the solution failed, the output of your solution and expected output according to the brute force solution on terminal and the script will be terminated.
If you wish to terminate the script at any moment you wish use the command ctrl + c in terminal.

quick demo: Here the output of first 3 test cases matched with the brite and as soon as it didn't, it printed the test case, output, and expected output to the console and terminated. Now you can debug your original solution and check where it goes wrong.
( )

You can watch this video by Errichto to understand more about stress testing.

Any suggestions or corrections are welcomed.

Apologies for my poor English.

UPD1: Updated the script to take brute and main file via flags, no. of testcases as optional, added help/usage flag and updated readme for better readability & usage. Tested on Mac & Ubuntu, bash 5.x. Please refer updated README here for usage.

  • Проголосовать: нравится
  • +52
  • Проголосовать: не нравится

»
4 года назад, # |
Rev. 2   Проголосовать: нравится 0 Проголосовать: не нравится

Suggestion — Maybe include the option to add a checker, because many problems have multiple possible solutions. Keep a default checker as a direct comparison of solutions, but allow the user to write his own.

»
4 года назад, # |
  Проголосовать: нравится 0 Проголосовать: не нравится

If you want to test cpp / c / java / python, you can checkout this

»
4 года назад, # |
  Проголосовать: нравится 0 Проголосовать: не нравится

How do I generate a binary string using the generator? please answer..

  • »
    »
    4 года назад, # ^ |
    Rev. 4   Проголосовать: нравится +5 Проголосовать: не нравится

    I think this should do the job:

    Спойлер


    or maybe like this:

    Спойлер
»
4 года назад, # |
Rev. 2   Проголосовать: нравится 0 Проголосовать: не нравится

sorry for a silly doubt but this wont work in cases when there are multiple possible solutions right

  • »
    »
    4 года назад, # ^ |
      Проголосовать: нравится +5 Проголосовать: не нравится

    Yes, you are right. But for such case you can write custom checker to verify the solution instead of using diff.
    I never really needed it, so never wrote one else i would have definitely updated the script.

»
4 года назад, # |
  Проголосовать: нравится 0 Проголосовать: не нравится

Hey, I downloaded the ubuntu subsystem on Windows 10 and I tried to run this script. It gave an error like this -

Error

I use g++ 9.2.0 but still this error. Any way around this? Any help is appreciated!

  • »
    »
    4 года назад, # ^ |
      Проголосовать: нравится +2 Проголосовать: не нравится

    Its probably saying that you dont have g++, try running g++ --version once there, it should show the version if its successful

    • »
      »
      »
      4 года назад, # ^ |
        Проголосовать: нравится 0 Проголосовать: не нравится

      I didn't realize I had to install g++ in Ubuntu too. How stupid. Now it works. Thank You!

      • »
        »
        »
        »
        3 года назад, # ^ |
          Проголосовать: нравится 0 Проголосовать: не нравится

        zass hey did it work after installing g++ like do i need to set path again i mean how to install g++ in ubuntu too ?? we already installed it right for normal ide i mean ...please tell me

»
4 года назад, # |
Rev. 2   Проголосовать: нравится +9 Проголосовать: не нравится

In case someone was looking for a stress testing script with a checker, I found this blog pretty useful.