Hello!

I have been writing a new test cases generator framework for IOI-style problems for several months. It should be quite stable now, so I would like to share it to you.

First of all, let me tell the source of inspiration of this new framework.

testlib

(https://code.google.com/p/testlib/)

I discovered MikeMirzayanov's testlib library when I prepared Codeforces Round #192 (a.k.a. Trollforces round). It is a nice library (thank you Mike!). I wanted to use it for generating test cases for our national training camps, but it seems that it is not quite suitable for IOI-style problems which have subtasks etc. So, I created a wrapper on top of it.

tokilib

(https://github.com/fushar/tokilib)

It is a wrapper on top of testlib. I actually wrote about it in a blog post. It essentially will call testlib's generation many times for generating multiple test cases, and testlib's validation many times for supporting subtasks.

It has been very useful and quite easy to use. We have been using it for ~ 1 year for our training camps and real national OI. It really helped the problem setters to collaborate on writing test cases for the whole problemset. Their feedbacks are always positive.

tcframe 0.3.0 (NEW)

(https://github.com/ia-toki/tcframe, https://tcframe.readthedocs.org/en/latest/)

Using the previous two solutions, we still need to write two files: a generator, and a validator. Can we somehow just merge them and only write a single file?

After brainstorming and designing for several months, I came up with a solution that I think is quite clean and neat! I completely rewrote it from scratc. It now does not depend on testlib anymore. I gave it a new name: tcframe.

Let's consider a very simple A + B problem:

Description

You are given two integers A and B. Compute A + B!

Input Format

The first line contains two space-separated integers A and B.

Output Format

A single line contains the answer.

Sample Input

7 42

Sample Output

49

Subtask 1

• 1 <= A, B <= 1000

Subtask 2

• 1 <= A, B <= 1000000

This is a sample generator program using tcframe framework that generates test cases for the above problem.

#include "tcframe/tcframe.hpp"
using namespace tcframe;

class Problem : public BaseProblem {
protected:
    int A;
    int B;

    int result;

    void Config() {
        setSlug("aplusb");
    }

    void InputFormat() {
        LINE(A, B);
    }

    void OutputFormat() {
        LINE(result);
    }

    void Subtask1() {
        CONS(1 <= A && A <= 1000);
        CONS(1 <= B && B <= 1000);
    }

    void Subtask2() {
        CONS(1 <= A && A <= 1000000);
        CONS(1 <= B && B <= 1000000);
    }
};

class Generator : public BaseGenerator<Problem> {
protected:
    void Config() {
        setBaseDir("tc");
        setSolution("./solution");
    }

    void SampleTestCases() {
        SAMPLE_CASE({
            "7 42"
        }, {1, 2});
    }

    void TestGroup1() {
        assignToSubtasks({1, 2});

        CASE(A = 1, B = 1);
        CASE(A = 5, B = 7);
        CASE(A = 10, B = 100);
        CASE(A = 1000, B = 1000);
    }

    void TestGroup2() {
        assignToSubtasks({2});

        CASE(A = 1001, B = 1001);
        CASE(A = 2000, B = 1500);
        CASE(A = 41728, B = 771823);
        CASE(A = 1000000, B = 1000000);
    }
};

int main() {
    Generator gen;
    return gen.generate();
}

One cool feature of this framework is that the problem specification class is really similar to the problem statement! I have brainstormed several possibilities for the syntax, and I think this is the best so far.

When run, the above program will produce the following output:

Generating test cases...

[ SAMPLE TEST CASES ]
  aplusb_sample_1: OK

[ TEST GROUP 1 ]
  aplusb_1_1: OK
  aplusb_1_2: OK
  aplusb_1_3: OK
  aplusb_1_4: OK

[ TEST GROUP 2 ]
  aplusb_2_1: OK
  aplusb_2_2: OK
  aplusb_2_3: OK
  aplusb_2_4: OK

And all generated test case pairs will be available in tc directory.

Now suppose that we made some mistakes in the generator class: instead of CASE(A = 1000, B = 1000), we accidentally wrote CASE(A = 1000, B = 10000). When the program is run, it will output a nice error message:

Generating test cases...

[ SAMPLE TEST CASES ]
  aplusb_sample_1: OK

[ TEST GROUP 1 ]
  aplusb_1_1: OK
  aplusb_1_2: OK
  aplusb_1_3: OK
  aplusb_1_4: FAILED
    Description: A = 1000, B = 10000
    Reasons:
    * Does not satisfy subtask 1, on constraints:
      - 1 <= B && B <= 1000

[ TEST GROUP 2 ]
  aplusb_2_1: OK
  aplusb_2_2: OK
  aplusb_2_3: OK
  aplusb_2_4: OK

That's it. You can find more complex examples on the documentation.

Anyway, please do try it yourself for creating test cases for your problems! I really need feedbacks from you, whether as bug reports or feature suggestions. Note that this still not a 1.0 version yet, so some syntaxes may change in the future.

I released this framework as open source, under MIT license. You can use it for whatever you want.

If you have any questions or feedback, please post a comment on this thread

Thanks! I hope this framework will be useful for you :)