There are lot of competitions which do not allow to submit your solution after contest's end(CEOI,COCI,IOI...), but they publish test data. I think it is very hard to manually compare output data with diff and it would be much more simpler if there was program for that. Isn't there something like that? Isn't there a nice coder who could code that for us? :D :D Sorry for english.
DMOJ
PEG
Assuming that test data are *.in/*.out files:
Upreckon is a very advanced local testing system, which allows you to specify input data filenames with either Python code or regular expressions (for most of the tasks, you'll just specify a mask like 'test.#.in' and the list of possible values for #).