I've created a Bash script to evaluate your own test cases:

#!/bin/bash
N=`ls *.in | wc -w`
for ((i = 1; i <= N; i++)); do
    echo ----- case $i -----
    diff -wb <($1 <$i.in) $i.out || break
done

To use this script, create test cases in the format 1.in, 1.out, 2.in, 2.out, 3.in, 3.out, ...

You need to pass your executable as first argument

By default, it will try doing tests sequentially until it finds a test case where it fails, then it will print the difference between your output and the correct output. If you want it to test every case, remove the part that says || break

I've also created a script that can create these files for you, which may be more convenient

#!/bin/bash
cases=`ls *.in 2>/dev/null` && N=`echo $cases | wc -w` || N=0
((N++))
echo Enter the contents of $N.in and press Ctrl+D when done
cat > $N.in
echo Enter the contents of $N.out and press Ctrl+D when done
cat > $N.out

You may replace cat with nano, vim, geany, etc if you prefer.

You may add both of these to your PATH, for instance you can create the folder ~/bin, add both files there, and then run:

echo PATH=$PATH:$HOME/bin >> ~/.bashrc

Then, if the judge is in ~/bin/judge and the case creator in ~/bin/mkcase, you can simply run judge ./program to test your program and run mkcase to make a new case

Both scripts use the Bash double parentheses construct that lets you write more C-like expressions, which isn't that well known.