HATE: An example

Adapted from EPSRC Summer School on Computer Vision Practical Assignment #2: Performance Evaluation

June 1996

The purpose of this assignment is to make you think about how one might evaluate vision techniques and to familiarize you with the use of HATE, the Harness for Algorithm Testing and Evaluation.

1. HATE is available to you via the URL:

     http://vase.essex.ac.uk/projects/hate/
   

   Feel free to look through the HATE source code---it is written in Tcl and is reasonably easy to follow---to see how it works.

2. After unpacking the HATE distribution, you will find an example sub-directory containing three C programs that are intended to be used for evaluating the performance of edge detectors. These are a data generator, a Laplacian edge "detector," and a program for assessing the output of the edge detector. Also in this directory are interface.tcl, which interfaces these programs to HATE, and edge.tcl, a script which contains a series of tests to run in order to characterize edge detector performance.

3. Install HATE on your system.

4. Compile the programs (there is a Makefile, though you may have to adjust it a little for your system), and run the tests by typing:

      hate -evaluate -verbose edge.tcl
   

5. Look at assess.c, the program that assesses the output of the edge detector--you'll see that it uses a very naive mechanism for determining whether or not a pixel contains a true edge pixel. Do you consider this to be a fair or appropriate assessment mechanism?

6. Design and implement (by modifying assess.c) a better strategy for evaluating an edge detector.

7. Does the script, edge.tcl, contain a reasonable set of tests for evaluating the edge detector? Enhance the script with some tests of your own that demonstrate the advantages of your assessment strategy.