A short quiz on vision system evaluation

Adrian F. Clark

1. Which test is most appropriate for comparing algorithms' performances?

   • Gauss's test
   • Laplace's test
   • Canny's test

   • McNemar's test

   McNemar's test is the most appopriate test for comparing algorithms: it is a chi-squared test with one degree of freedom for paired data.

2. You are developing software for the police to show mugshots of suspects to the witness of a crime. Which of the following is the best approach to take?

   • minimize the number of false positives

   • maximize the number of true positives, even if the false positive rate is high

   • maximize the number of true positives
   • minimize the number of false negatives

   For this type of application, we want to show anyone who stands any chance of being the perpetrator; it doesn't particularly matter if we have a high false positive rate as long as the true positive rate is high.

3. Which corner of a ROC curve indicates the best performance?

   • upper right

   • upper left

   • lower left
   • lower right

   We want the smallest number of false positives for the largest number of true positives, so the best performance is the upper left corner of the plot.

4. When using McNemar's test, what do we do if we want to see whether one algorithm's performance is better than another's?

   • Look up the Z-score in two-tailed tables
   • Look up the Z-score in Normal distribution tables
   • Look up the Z-score in binomial tables

   • Look up the Z-score in one-tailed tables

   If we want to know that one algorithm's performance is better than another, we must use one-tailed tables.

5. You are developing a automatic passport system for use by immigration, where pictures of people are compared to those in their passports. Which of the following is the best approach to take?

   • maximize the number of true positives, even if the false positive rate is high
   • maximize the number of true positives
   • minimize the number of false negatives

   • minimize the number of false positives

   For this type of application, we need to keep the number of false positives as low as possible; otherwise, we would admit lots of people who don't look like the picture on their passports.

6. What is a false negative?

   • A false result from an algorithm that should have succeeded

   • A negative result that is false
   • A false result from an algorithm that should be negative
   • A false result from an algorithm

   A false negative arises when an algorithm reports failure (or crashes) when it should have succeeded.

7. What are the axes of a ROC curve?

   • TP and TN
   • TP and FN
   • FP and FN

   • TP and FP

   An ROC curve plots the number of FPs against the number of TPs.

8. When using McNemar's test, what do we do if we want to see whether algorithms' performances differ?

   • Look up the Z-score in Normal distribution tables
   • Look up the Z-score in one-tailed tables
   • Look up the Z-score in binomial tables

   • Look up the Z-score in two-tailed tables

   If algorithms' performances differ, it doesn't matter which is better than the other, so we use two-tailed tables.

9. What is 'ground truth'?

   • the true values obtained by an algorithm
   • values obtained by an algorithm that are known to be true

   • data known to be correct

   • images of the ground

   Ground truth are data (usually images) for which the correct answer is known; they are used for training and testing algorithms.

10. When evaluating vision systems, it is normal to:

    • train on all data but test on only the test set
    • train on the training set and test using both training and test sets
    • use the same training and test sets

    • have different training and test sets

    We know that algorithms work better on the data they were trained on than on unseen data; hence, we use different training and test sets.