A simple bash script would suffice. Something like this:

for f in *.in
do
  test_name=${f%.in}
  if ! ./solution <$test_name.in >$test_name.myout
  then
    echo "$test_name: RE"
  elif diff -bB --strip-trailing-cr $test_name.out $test_name.myout
  then
    echo "$test_name: AC"
  else
    echo "$test_name: WA"
  fi
done

(Obviously if the answer is not unique, you'll have to replace diff with a manually implemented checker.)

Alternatively, you could use my caide software which has other features in addition to tester.