HATE: Harness for Algorithm Testing and Evaluation
One of the most tedious tasks in writing software is making sure it works as expected -- and this goes some way to explaining why many programs have bugs in them. Although testing will never become a terribly exciting task for most people, it is possible to reduce the tedium of repetitive testing by writing test scripts that exercise programs.
In practice, test scripts are usually put together quickly, used to check out one piece of software, and then discarded. Experience shows that test scripts are often more prone to bugs than the software they are supposed to be testing! What is needed is a framework into which different tests may be fitted: this simplifies writing them and encourages their re-use.
Ensuring that a program is as bug-free as possible is not necessarily the whole story, especially in a research environment: one often needs to compare different algorithms in order to assess which is optimum. In this case, the different algorithms are typically applied to identical data and their performances compared statistically. This evaluation process also benefits from automation.
These tasks are precisely what HATE addresses: HATE is a test harness, a program that is intended to take the tedium out of testing and evaluating programs.
Some effort has been taken to ensure that HATE is totally independent of both the program being tested and the series of tests to be applied to the test program. The intention is to provide a single tool that may be used for both testing and evaluation; to encourage well-designed objective testing; and, by allowing researchers to apply the same sets of tests to their own programs, to make it easy for researchers to compare techniques on an informed basis.
The major features of HATE are:
- HATE is written in a high-level scripting
language and may be used unchanged on any platform that the
language supports. This includes all flavours of Unix; Linux; Windows
NT, 95, 3.n; and the Macintosh.
- Test scripts may be used unchanged on all platforms.
- Test scripts are independent of the software being tested or
- Programs are interfaced to HATE by means of a short interface
procedure; thereafter, all relevant existing test scripts may be
used with them.
- The decomposition into test script and interface procedure, and
the portablity of HATE, encourage the sharing of scripts between
developers or researchers.
- HATE is able to generate its output in a range of forms, including HTML and LaTeX tables.
HATE was developed principally to aid the evaluation of computer vision algorithms. There is increasing realization within the vision research community that this type of work is essential in converting computer vision from a "black art", with algorithms being tested on only a few images, to sound engineering practice. It is hoped that the vision community will develop test scripts that allow comprehensive comparisons of existing algorithms to be performed, and that new algorithms are subsequently compared with the existing body of results.
Further information is available on:
- downloading HATE
- installing HATE
- HATE's modes of operation
- using HATE
- interfacing programs to HATE
- writing test scripts
- the HATE program itself
- incorporating HATE into your own programs
- possible future enhancements
HATE is not public-domain software: the author retains copyright. However, it is made available free of charge on the conditions that:
- it may only be passed on to third parties without charge and in a complete and unchanged form;
- the author is not liable for any harm or injury that arises from the use of HATE, either directly or indirectly.
If you wish to use HATE in your own work, please inform its author, Adrian F Clark (email@example.com). A simple message containing
REGISTER HATEas the subject of the mail message will suffice. You will then be informed of any bug-fixes or new releases of the software.
If you find HATE useful in your own work, the author would be happy to receive a picture postcard from your locale. The principal citation for HATE in the literature is currently
P. Courtney, N. A. Thacker and A. F. Clark
Algorithmic modelling for performance evaluation
Machine Vision and Applications, in press
Note that HATE is work in progress. Feedback on both the software and this documentation are welcomed: provided you're connected to the Internet and are viewing this document with a Web browser, you can give feedback by clicking the author's email address at the bottom of this document. (Feedback is especially welcomed on the layout of the tables that HATE generates, as the author doesn't think they're quite right yet.) Suggestions for enhancements (or, better, working code) and test scripts to share with other researchers are also welcomed.