A quiz on the machine learning experiment

Adrian F. Clark

1. What is the accuracy of MLP on digit '4' of MNIST?

   • 0.97
   • 0.99

   • 1

   • 0

   MLP (and some other techniques) report a perfect score on MNIST, so the accuracy is 1.

2. On which digit or digits of MNIST does eigen out-perform SVM?

   • 0

   • 0 and 8
   • all of them
   • 1 and 7

   eigen out-performs SVM on digit zero, though the performance difference is not significant.

3. How do you tell ml.py where to find any images it needs?

   • by editing the task file

   • using the -data command-line qualifier

   • using the -dataset command-line qualifier
   • by editing the program

   ml.py supports the '-data' qualifier for you to tell it where the images are to be found.

4. When comparing the performances of RF and SVM on MNIST, for how many digits is the performance difference statistically significant?

   • 7

   • 9
   • 0
   • 10

   RF out-performs SVM on all digits of MNIST but those for 0, 1 and 6 are below the significance threshold of 1.96.

5. What is the main purpose of this experiment?

   • to see whether machine learning is better than hand-crafting solutions
   • to learn about machine learning
   • to try out deep learning

   • to give experience in comparing the performances of algorithms

   This experiment uses a variety of machine learning approaches, spanning conventional and modern, and its real purpose is for you to see which is better and how the training database affects performance.

6. Which of the following is NOT supported by ml.py?

   • convolutional neural network

   • genetic programming

   • support vector machine
   • multi-layer perceptron

   Genetic programming (GP) is not yet supported by ml.py. I use GP a lot in my own research and so one day will get round to interfacing it to the program.

7. How many different classes are then in the ORL5 task?

   • 40

   • 6
   • 10
   • 5

   There are 40 subjects in the ORL database so this is a 40-class problem.

8. Which of the supported learners is the slowest to run?

   • CNN

   • eigen

   • WISARD
   • MLP

   At least with the sketchy implementation in ml.py, eigen is by far the slowest of the learners to run...though it is probably the fastest to train.

9. The eigen learner can use 'manhatten' to match feature vectors. What does this compute?

   • the sum-squared differences between features (aka Pythagoras's theorem)

   • the sum of absolute differences between features

   • the correlation coefficient of the feature vectors
   • the angle between the feature vectors

   The Manhatten or 'city block' distance is the sum of the absolute differences. It crops up quite often when assessing the similarity of things.

10. What is a 'knowledge base' in ml.py?

    • a compiled program
    • a type of data structure

    • a trained machine learning system

    • a set of rules

    ml.py refers to a trained machine learning system as a 'knowledge base'.