Laboratory 6: Identifying biscuits

Introduction

Bearing in mind how many times the lectures touch on broken biscuits, it was bound to happen: an experiment that involves you recognizing whether images containing biscuits are complete --- either circular or rectangular --- or problematic, such as broken, which we lump into the category reject. Your starting point is a program called bikky.py which does the following to each image given on its command line:

1. the image is read in as grey-scale;
2. it is thresholded using a fixed value;
3. the resulting binary image is tidied up using morphological operations;
4. contours are found around each foreground object;
5. each contour is processed;
6. some text is written on the image.

You run the program with a command such as:

python bikky.py -d biscuit-001.jpg

With the -d qualifier, It displays output such as that shown below:

as well as writing a line of output in the terminal window that gives the image filename and the outcome. (We shall return to this later.)

The program is incomplete in that it does not classify the contours found into the three categories mentioned above --- that is your job. The program and a set of images are to be found in the zip-file for this experiment.

Streamlining development with task files

It is tiresome to run bikky.py on all the images and try to keep track of how effective it is --- isn't that precisely the kind of thing computers are meant to do? That is right of course, and so a neater approach is available using something called a task file, which essentially lists all the input images and their expected outputs. You tell the program to use the task file biscuits.task with the command:

python bikky.py biscuits.task

The content of the task file is plain text so you can read through it. (You'll likely see me generate a task file during a lecture.) Note that the -d qualifier isn't given here --- if you do, you'll have to look at a couple of hundred images as the program runs.

What kind of biscuit is it?

As discussed in earlier laboratory scripts, OpenCV has the ability to find contours in thresholded images, similar to but better than the region labelling approach described in Chapter 5 of the notes. It is able to compute a variety of features of any contour; see for example

https://docs.opencv.org/master/dd/d49/tutorial_py_contour_features.html

for a description of some of them --- and be aware that not all measures are described. Using contour features or measures derived from them, you should be able to determine whether a particular contour is a circular or rectangular complete biscuit or that the biscuit should be a reject. (Hint: have you written routines in sxcv.py that are potentially useful here?) You should determine the category and store it in the variable outcome so that bikky.py can annotate the image for display and write it out.

Assessing how well the bikky.py works

It was mentioned earlier that each image causes a line of output to be produced. When you run bikky.py on a task file, it generates output for subsequent analysis by a program called FACT. The way to do this is:

python bikky.py biscuits.task  > biscuits.res
python fact.py analyse biscuits.res

in a Unix (Linux, macOS) environment. (Those of you who think the command line is archaic and irrelevant should be aware that many real programmers use it a lot. You might be starting to see why we work this way.)

When invoked in analyse mode, FACT outputs a table of accuracies and a class confusion matrix; at this juncture, you are most interested in the overall accuracy. If you want to see what the definitions of some of the other measures output are, use the --detail=2 command-line qualifier. As usual in the Unix world, the invocation:

python fact.py -h

yields some help.

Improving the segmentation

You will see from bikky.py's displayed output that it does not threshold the biscuit from the background particularly well, often finding spurious foreground regions. If you look at the code, you will see it uses a fixed threshold. You might be able to improve contour detection by changing the value of the threshold it uses, or you may find that an adaptive threshold works better. This functionality is available in OpenCV, so explore that and see if you can improve the program's thresholding stage to find the biscuits more reliably. If you do that, you might also be able to reduce the amount of morphological processing. You can assess whether any changes you make are an improvement by checking the overall accuracy in the FACT output.

Alternatively, you might find that segmentation works better using colour. As the biscuits may be different colours but the background is always the same, the best approach is to identify the background --- anything which is not background is then biscuit. You should have taken this approach in an earlier laboratory.

Do show your solution to a demonstrator.

All the laboratory scripts