\chapter{GP for Low-Level Vision Tasks (DRAFT)}
\label{ch:low-level}

The previous Chapter discussed at length how Genetic Programming might best be put to the task of classification, that is finding patterns in amongst data. The data sets included the StatLog Sat Image, which contains multispectral terrain readings from aerial photographs, or the Wisconsin Breast Cancer set, which consists of a number of measurements taken by physicians. In this Chapter we begin to focus on problems in computer vision, and examine how GP classifiers may be employed to solve them. That is, alas, too high level for now - we must first explore exactly how one might create those datasets in the first place. The StagLog and WBC measurements seem rather obvious, given their specific domains, but exactly which measurements should we take from images in order to solve generic problems involving vision?

As it happens, most non-evolved vision systems comprise two or more stages. Following any preprocessing operations, the next step is usually concerned with the detection of features of interest within the image. A classifier may then be used to make sense of the output of a particular feature across a number of different situations. Exactly what makes an interesting or useful feature is a point of discussion in itself. Some algorithms, such as SIFT/SURF were developed to find generic points of interest within an image, which seems quite promising. A range of other feature detector families are concerned with detection of all kinds of features, such as corners, edges, blobs, and geometric shapes. Although the kind of features detected by SIFT are quite robust, making them suitable for image matching and 3D reconstruction work (and indeed they have been used for recognition too), one has to consider the extent to which different objects can be described using such features. It may be these points of interest are in fact \emph{too} generic to work well in our experiments. 

While our system itself must be capable on working on problems that it hasn't seen before, there is nothing to stop it from developing domain specific features, provided that it can do so automatically. Our system will, therefore, develop or tune features for a particular problem in order to achieve a given level of accuracy or reliability. This is common practice in many areas of engineering and in also practical computer vision. A drawback of these features is their potentially limited scope for re-usability, although this further motivates our research into finding ways of developing such features automatically. This Chapter will outline the approach we've taken, and how it may be used in a variety of vision tasks. A few examples of feature detectors developed using our technique are presented toward the end of the Chapter.

%TODO: Mention requirements.

\section{Segmentation}
Our feature selection technique is based upon the classification of pixels in the image into particular classes, some of which may be of interest, and others which can be discarded. However, due to noise, compression artefacts  or other blemishes in the image the classification of a just one pixel may not provide a very reliable indicatator of the class of that area of the image. However, if one classifies all the pixels in the image, regions may appear that give a good indication of trends within the image.

The process of discovering regions in an image according to their apparent material is called segmentation. If an object consists of one material, and the other objects in the scene are made up from another, then segmentation can be used to distinguish the two. Once every pixel has been labelled, the image can then be divided into a number of contiguous regions which might relate to separate objects (providing the classification process or image itself are not inherently random). The computer vision developer may then choose to discard, investigate or compare different regions. Segmentation, therefore, is one method for combining many pixels (which individually carry very little contextual information) into fewer groups which may represent a piece of information within the image more concisely. 

Segmentation has a large range of applications. It has been used to aid the visualisation of mineral or geological features when processing multispectral images. It is also commonly used in facial recognition in order to discard the areas that don't appear like human skin, therefore reducing false positives. In a wider sense, segmentation can often be used to distinguish the pixels in the foreground from those in the background, implicitly finding the boundaries between certain objects (the Watershed algorithm is one example).  

Although segmentation can be used to distinguish objects by material, as alluded to by the above examples, it may also be used as a feature detector, capable of identifying certain objects by both their shape, or texture, or colour (or a combination of the three), as will be demonstrated later. Segmentation, therefore, may be a slight misnomer as producing image segments is just one capability of this flexible approach. Because of its flexibility, and ability to use raw pixels as training data, segmentation was chosen as our feature detection method.

Of course, if the segmentation system is to be capable of making decisions about a pixel's colour, texture and wider context, then it must be provided with reliable useful information. The choice of pixel features that provide this information are discussed next.

\section{Features}
At the lowest level of the segmentation process is a classifier, which looks at each pixel individually and makes an independent decision for each one. Having discussed some of the ways in which Genetic Programming may be used to classify an object, based upon an $n$ dimensional feature vector, we now turn to the features themselves, the choice and implementation of which is crucial to the reliability of the classification and thus the segmentation as a whole.

Genetic Programming affords the developer the advantage of providing a reasonably large library of features to the classifier which will, in general, take advantage of the most useful features, given a certain problem. In order to make the segmenter capable of solving a variety of different problem types, two feature types were identified:
\begin{itemize}
 \item Colour Information about the intensity and colour of a particular pixel
 \item Texture Information regarding the pixels surrounding the pixel in question.
\end{itemize}

In addition to the kind of information delivered by the feature itself, the feature should also be reasonably invariant to changes in illumination. Some texture features may additionally be invariant to rotation or other image transformations. As the segmenter is run on every pixel of the image, it is necessary that all features be as computationally efficient as possible.

\subsection{An Aside on Sensory Considerations}
It isn't possible to discuss digital images without mentioning the problem of colour constancy. While light sensors (including our eyes) only receive the ``colour'' of a beam of light, expressed in terms of its wavelength, that colour is actually the product of at least two sources: the irradiance of the light(s), and the reflectance of the surface being viewed. The human vision system is able to recover the the surface's reflectance, what we describe as ``colour'', with relative accuracy, regardless of the light source, a phenomenon known as colour constancy. While it is sometimes difficult simply to calibrate camera white-balance in order to reproduce colours accurately, the development of software that can \emph{perceive} colours accurately is even more tricky. 

Developing colour constancy algorithms is just one part of the huge task of developing a sensor that can compete with the masterpiece of engineering buried within every human's vision system. The discipline of colour science is rather beyond the scope of this thesis, but a couple of ideas will be discussed briefly here. Although the most common approaches are imperfect \cite{colourconstancygood}, they may offer some useful algorithms that can be implemented relatively easily within the requirements of this vision system. Two simple algorithms are in common use, both in their own right, and as part of larger retinex algorithms\footnote{Other methods, especially Neural Nets, have also been shown to produce good colour constancy solutions, but their coverage is outside the scope of this thesis.}.

Ebner \cite{ebner:2001:colour-constancy} used Genetic Programming to evolve an artificial retina which is capable of recognising the true colour of materials.  His retina consisted of a 2D array of elements, each of which had a red, green and blue sub-element. Each element corresponded to one pixel on the image.

The first algorithm, white-patch retinex, assumes that the true colour of the light can be identified by looking at white patches in the scene (which are assumed not to affect the irradiance). White patches are relatively easily identified, as they are the usually the brightest. When the light's colour is known, then the colour of the rest of the image can be divided by that value to recover the ``true'' colour of the scene's materials. The white patch algorithm is used by many digital cameras for calibration, although it is dependent upon there being something white in the scene - if a non white patch is used mistakenly, the algorithm may produce strange colours.

The second algorithm, grey-world assumption, is based on the assumption that the average colour of images should be about grey. Therefore if one finds the actual average colour of an image, each pixel can be adjusted such that the average colour becomes mid grey. Although slightly more computationally expensive than the white patch algorithm (the image needs to be read twice, once to calculate the average colour, second to adjust the values, the technique can produce reasonably good results, (see Fig. \ref{greyworld}) on certain images.

Figure \ref{fig:cc} shows the author's implementation of a basic grey world retinex algorithm on two images that have been ``lit'' with different lights. While it is impossible to know what the ``true'' appearance of the image, minus the lights is, the algorithm appears to work reasonably well, here at least producing the kind of persistent perception necessary to make accurate and robust decisions. While the author does not claim to have produced an ideal solution to the problem, this kind of step will be necessary for creating generic vision systems which may have to work in non illumination-controlled environments.

\begin{figure}
\begin{centering}
\includegraphics[width=0.5\textwidth]{images/greyworldcolourconstancy.eps}
\par\end{centering}
\caption{The two images at the top have different tints, caused by different lighting conditions. The images on the bottom show the images after being adjusted by our simple grey-world-assumption algorithm. The two bottom images are much more comparable than the top two.}
\label{fig:cc}
\end{figure}

Machine vision systems may also fail in environments where the range of intensities is too large for the camera's exposure system to accommodate. Although people are able to perceive both very bright and relatively dark objects within the same scene without too many problems, cameras tend to correctly expose one while underexposing or overexposing others, yielding images with either completely white-out skies, or dark silhouette foregrounds. This is because although the human eye is able to control exposure in every sensor, current cameras can only control the exposure for the whole image, which can prove a severe limitation.

Until camera sensor technology improves in this regard, computer vision software in environments with non-controlled lighting conditions will always be at a disadvantage. However, there are some other algorithms which can improve the camera's exposure in software. As it is generally the case that illuminaion affects large areas, and a surface's texture is much more granular, an estimate of the average exposure of a pixel can be gleaned from looking at a weighted average of the pixels around it. If the weighted average is treated as the level of illumination, then the pixel's intensity can be updated accordingly such that all pixels in the scene have a more equal illumination. This is the basic idea behind Land's Retinex Theory\footnote{Edwin Land, founder of Polaroid and prolific researcher developed his retinex theory after conducting experiments which showed people can perceive colours in an image even when that colour isn't used at all by the illuminant. Although this 50 year old discovery undermines parts of the prevailing understanding of how we see colour, Land's research is still largely absent from textbooks.} The author has implemented a simple example of this algorithm, the results are shown in Fig. \ref{fig:exposure}.

\begin{figure}
\begin{centering}
\includegraphics[width=0.5\textwidth]{images/exposure-adjustment.eps}
\par\end{centering}
\caption{The two images at the top are both poorly exposed. However, it is possible to recover the image to an exposure more akin to the perception of a human. Some colour information is lost, however, due to the 8 bit nature of the image.}
\label{fig:exposure}
\end{figure} 

This completes this author's brief foray into the world of colour constancy. While there is no guarantee that the processed images shown here are accurate (although there is equally no guarantee that our personal perception of colour is accurate, or indeed the same as other people's perception), at least these techniques can aid the perception of an image by making some parts visible, and providing a degree of the consistency which is vital for the robustness of learned vision approaches.

\subsection{Colour Features}
Having discussed how colours might be more accurately computed using colour constancy and exposure algorithms, we turn to the features themselves, which provide information about the colour of a single pixel to the segmenter.

The traditional way to express colours in computing environments, dictated by the display hardware, is in terms of its red, green, and blue components. However, RGB values do not necessarily provide immediately useful information because the intensity, or brightness, of the pixel can only be ascertained by a formula using each of the three components in the triplet ($0.3R + 0.59G + 0.11B$). Fortunately, colours in the world of software may be expressed in many different ways. A popular technique for expressing RGB colours is in terms of HSI, which defines a colour according to hue, saturation and lightness/intensity components. By dividing the colour into components relating to the shade of colour, the colour's intensity and the overall lightness, HSL colour components make potentially more succinct information about the pixel's colour available to the learning system. The translation of RGB to HSL is also relatively straightforward.

Since GP does not impose any particular requirement to keep the feature set size beneath a particular level, both colour triplets were made available to the GP system. If the individual is able to make accurate distinctions using only RGB, then the computer is spared the (admittedly small) extra burden of calculating HSL. Otherwise the HSL may prove more useful in less straightforward cases.

\begin{itemize}
\item Red
\item Green
\item Blue
\item Hue 
\item Saturation
\item Lightness
\end{itemize}

%TODO: What about average colours, spread over several pixels to make sure that noise isn't interfering too much?

\subsection{Texture Features}
This section will deal with different features which may encode clues about a pixel's texture. While colour information can be ascertained from a single pixel (provided there is not significant noise in the image), patterns in the image (described here as textures) are only apparent when multiple points are compared. Therefore for the segmenter to discover some kind of texture information about each pixel it needs to inspect the pixel's neighbours.  
So, given samples of a particular texture, we have to find features that describe it with reasonable accuracy. Examples may include the difference between neighbours intensity values which may give an indication of the contrast of the texture, the arrangement of neighbours may that reveal the directionality of patterns within the texture. The rate of intensity change across a series of points could reveal the roughness or smoothness of the texture. The crucial question is: which amoung a pixels' neighbours will give an accurate indication of a particular texture or shape, and how might they be used? 

One solution is to provide the segmenter with the intensity of every pixel surrounding the current one, and leave the GP program to develop some strategy on its own. Given arithmetic operators and random constants, it is perfectly possible that the GP system could evolve a form of convolution mask. Since no data is lost, it is possible to develop a measure that accurately represents the texture. In theory, the GP system would be free from arbitrarily defined features and thus capable of producing solutions tailored exactly to the training data. The problem is that a relatively large number of pixels are required in order to produce a discernible pattern, so the evolved programs would have to be relatively large in turn. Given sufficient time, a learning system may discover useful ways of combining pixel data to develop texture features, but it may longer than the Genetic Programmer is prepared to wait (especially as the processing would be interpreted and without any form of optimisation). This kind of approach, attractive in some senses, is thus unlikely to be a practical solution.

Turning to established texture analysis techniques from the computer vision field, there are many techniques that can be drawn upon. Grey Level Cooccurence Matrices(GLCM) are used to encode probabilities between different intensity levels given a particular spatial relationship between two pixels, for instance the relationship between one pixel and the pixel directly below or above it. The GLCM can be used to calculate a variety of different texture metrics, including:

\begin {table}[tp]
\centering 
\begin {tabular}{ l p{10cm} }
\toprule
Contrast 	& Contrast is measured by the sum of GLCM probabilities, weighted according to their distance from the diagonal point of the GLCM (the further the distance from the diagonal, the higher the contrast).\\ \midrule
Angular Second Moment &	Calculated from the sum of squared probabilities, high values provides a measure of the ``orderliness'' of the texture since high probabilities (definite patterns) will be emphasised, and low probabilities will be minimised.\\ \midrule
Entropy 	& Gives an indication of the ``disorderliness'' of the image by summing the natural log of each probability, which gives highest weight to low probabilities.\\ \midrule
GLCM Mean & Measures the average probability of whatever relationship the GLCM is measuring. Horizontal and vertical GLCMs may produce different means.\\ \midrule
GLCM Variance & Measures the Standard Deviation of GLCM probabilities and is similar to the entropy measure. A completely uniform image will have a GLCM variance of zero.\\ \bottomrule
\end {tabular}
\label { table.label } 
\end {table}

While the GLCM is relatively easy to understand and calculate, its size is dependent upon the quantization level of the image. For 8 bit images, the matrix will be 256x256, which is resource hungry, especially if the process must be repeated for every point in the image. If the texture window itself is quite small then the matrix may be sparse. The practical reaction is to quantise the image into smaller pieces, which has the drawback of de-sensitising the matrix to more subtle changes.

\paragraph{Texture Feature Requirements}
The discussion so far has alluded to several requirements or specifications for a good pixel based feature. These are:

\begin {table}[tp]
\centering 
\begin {tabular}{ l p{10cm} }
\toprule
Dimensionality Reduction 	& The feature should reduce the dimensionality of the pixel data by processing groups of pixels together in order to arrive at a statistic. Leaving this work to the GP learning system makes the search space too wide.

Techniques for speeding up certain kinds of repeated evaluations on an image have been established within the computer vision community. In recent years the usage of an \emph{integral image}\cite{Crow:1984:STT} was popularised by the work of Viola and Jones. An integral image, as its name suggests, is a summed table produced from an original image in which each point at (x,y) consists of the sum of points' intensities from the (0,0) origin. Using such a table it is possible to calculate arbitrary image sums using only addition and subtraction operations.

\\ \midrule
Use of Colour 	& Since the majority of textures are just as clear in black-and-white as they are in full colour, the texture operators here also work in black and white, using only the intensity of each pixel as input.\\ \midrule
Efficiency 	& Since each feature may be used at least once per pixel, its processing should be as fast as possible to ensure that the learning process can proceed and that the eventual program can operate in practical circumstances. \\ \bottomrule
\end {tabular}
\label { table.label } 
\end {table}

\section{Training Procedure}
Segmentation works by making a decision about each individual pixel. Therefore in order to train the segmenter, a number of images need to be supplied containing pixels whose class has already been established. One way to do this is to create a binary mask which defines which pixels in the image belong to one class and which do not. Our experience appears to show that marking up every pixel in an image is not always a good idea. Some parts of the image cannot be assigned to one class with absolute certainty. If the user does fully mark an entire image, the training data may contain contradictions that, while broadly invisible to the human editor, confuse the learning system.

Binary masks are also tiresome to create and do not readily permit the marking of more than two classes. For this reason the author developed a software interface that allows the user to mark up parts of an image by painting classes over it as one would do in a graphics application. The software is completely integrated with the author's genetic programming classification system and thus makes the production of segmenters quite intuitive. The software will be explained in more detail later in this thesis.

Assuming for now that the learning system has access to a pile of pixels for which the class is known, the results of texture and colour features are precalculated for each pixel and the GP system is allowed to work on classifying them. 

%TODO: I'm not sure what the point of this section is - can I think of something more exciting to say? Otherwise move the whole lot to the vision system chapter?
 
% TODO:  Everything below here is a bit junk at the moment.

\section{Window Detection}
Segmentation does suffer from certain drawbacks. If objects do not have clearly defined boundaries, the output of a segmentation system may be unreliable or unhelpful. It can also only relate information based on material, so the objects in a scene ought to be all of the same class. However, for certain problems this doesn't work - if one is trying to evolve a face detector, a segmenter may well be able to find the regions of skin. The segmenter won't, however, be able to distinguish a person's hands from his or her face. It is therefore necessary to consider detectors which consider relationships between groups of pixels more carefully. Solutions to these problems will be the topic of later Chapters.

\section{Feature Sets}
How many lines are there in a human face? A single one around the whole face? Separate ones for each eye? Nose? Mouth? Wrinkles? Hairs? There is no correct answer, so sometimes the information yielded from line or edge detection is not that useful. Some information, however, can define a face pretty well. Its a fair guess that of two adjacent rectangles, one over the eyes and one on the forehead, the one over the forehead will often have less variance in intensities. Low level information like this can be found using simple features based on pixel intensity information. These features can give important clues about texture or other features not as strongly indicated as edges or lines. 


\subsection{Pixel Features}



\subsubsection{Features for Texture}

\subsubsection{Haar Features and other Parameterised Features}

\paragraph{GLCM Features}
... Information on Grey Level Co-occurence matrices

\section{Feature Selection}
Before features can be used in classification, it is sometimes necessary to involve an additional stage of feature selection in which a decision is taken as to which features are suitable for solving a particular task and which are unlikely to be helpful in classification.

Additionally, in the case where the features are parameterised (the haar features are defined in terms of the size of rectangles and other factors and thus represent a large ``family'' of potential features) it may be necessary to search for good values using techniques, for instance using a Genetic Algorithm. These techniques will be discussed in depth later.

Some linear algorithms, such as Neural Nets, are susceptible to unhelpful features, and generally also require the features' outputs to be normalised so that the magnitude of values returned by a particular feature doesn't effect the output. In non linear techniques, such as Genetic Programming, it is not always necessary to perform this stage. Indeed the GP process offers several different means of discovering acceptable parameters while the evolution is in progress. However the concept of ``dimensionality reduction'' -- where irrelevant features which would otherwise add noise to the training data while adding little new information are removed -- can certainly help speed up the search process. 

In many cases it may be the computer vision researcher who decides which features should be included, based on intuition or experience. However, sometimes the scale of the data is overwhelming or the correlations difficult to distinguish. For this reason various algorithms have been developed in order to perform this task automatically, including:

\subsection{Existing Approaches}

\subsubsection{Linear Discriminant Analysis (LDA)}
The aim of LDA (in its simplest form) is to find the one dimension which best
separates examples from two classes. This makes LDA suitable for feature
extraction. More information to follow here. Also to mention Principal Component
Analysis.

\subsubsection{Princical Component Analysis (PCA)}
... information about PCA

\subsubsection{Kernel Discriminant Analysis}
... information about KDA

\subsubsection{Branch and Bound}
... information to follow

\subsubsection{Floating Search Method}
... information to follow

\subsection{Feature Construction by Genetic Programming}
Genetic Programming is a flexible learning technique. Given a set of raw data, it is capable of evolving a reasonable solution to a problem, given time and the appropriate tools (those tools will be discussed later). In order to generate a high-level solution to a particular problem, the raw, generic data requires processing in some way to make it suitably useful. 

The kind of features discussed earlier in this chapter are mostly parameter based, and therefore represent families of potential features. Some of these features will be useful for a particular task, and other will not. While Genetic Programming may be able to tune these parameters and find a suitable overall solution in a single pass, other approaches have been suggested which may be more efficient; some are introduced below.

\paragraph{Two Stage Evolution}
As the granularity of a GP system usually only extends as far as individual programs, in default GP implementations it is not easy to retain those parts of a tree which work very well. The system may evolve an excellent feature but if used incorrectly, the feature (and the rest of the program) may not be included in further generations (it should also be remembered that GP does not necessarily guarantee that even the best individual will be able to reproduce). In such a situation, the bluntness of the fitness measure causes the system to loose otherwise useful information. For this reason it may be useful to separate the evolution into smaller stages so the fitness function is more precise and less information is lost. 

A natural approach is to split the evolution into two stages. In the first stage, just the feature set is evolved. As complex objects, such as a human face, generally cannot be completely described by one feature, in the second stage the GP system learns how to combine a set of pre-evolved features in order to deliver an optimised result.

The evolution of the feature set is in essence a parameter optimisation problem, for which techniques such as Genetic Algorithms have been previously employed. As Genetic Programmins is essentially an extension of Genetic Algorithms, a GP approach may also be taken.

\paragraph{Coevolution}
A second approach also splits the evolution into two parts, but aims to learn them in parallel rather than sequentially.

\section{Making GP More Efficient}
In this section, various techniques for improving the general efficiency of Genetic Programming are explored. Desirable techniques are those which have some positive impact either on the speed of evolution or on the quality of the results produced. 

\subsection{Automatic Classifier Selection using Minimum Standard Deviation / LDA}

\subsection{Automatic Classifier Selection using Range Typing}

\subsection{ERC Optimisation}

\subsection{Training Data}
Genetic Programming, in general, works using a supervised learning paradigm. That is to say that the fitness of an individual is measured according to its performance against known training samples. The collection of such training data will be discussed later in Chapter \ref{ch:vision-system}, but here we discuss here the ways in which GP can be adapted to learn as much from the training data as possible.

It is widely recognised in pattern recognition that training data is likely to contain some level of bias. It may be that some parts of the problem are represented very well by a particular set of training data, while other parts may may be represented by fewer samples. Some training data may be significantly easier to solve than others. 

These situations present two problems. Firstly they may introduce local minima in the solution landscape, into which learning systems may easily become stuck. Secondly the bias of training data sample may not reflect the true bias of data unseen environments, which will reduce the predictive performance of the learned system.

\subsubsection{Ensemble Methods}
The general purpose of ensemble methods, which are widely used in pattern recognition, is to generate a series of different estimators through manipulation of the training data which improve the overall performance. Two exist: bagging and boosting. These two methods have been widely used to create more accurate classifiers. As meta algorithms they can be used to improve the accuracy of any set of weak classifiers. As such we'll not discuss them further here.

\subsubsection{P Learning}

\subsubsection{Training Data Weighting}
Genetic Programming may be used to generate the weak classifiers used by the ensemble methods, but we do want to ensure that the learning system is as efficient as possible in the first place. This reduces the number of weak classifiers required to reach a level of accuracy, and speeds up the learning process itself, which may be hampered by a boosting procedure.

GP is susceptible to local minima because it is a lot easier to break a working individual than to make it that much better that it can continue to make incremental improvements. If the training data can be weighted such that more difficult samples attract greater rewards, then the GP system may be able to break out of local minima more easily. The downside is that this may make the learning process less stable.

The difficulty of a particular item of data may be approximated according to how often it is wrongly classified. The benefit of this approach is that it is domain independent, and can be added into any learning system in which Genetic Programming is involved. Although it appears to be a logical approach, it may introduce instability into the learning process. A number of experiments were conducted to investigate its effect empirically.

The experiments were conducted on a widely used machine vision data set: the CBCL face database (version 1), which consists of thousands of face and non-face images, divided into training and test data sets. Training set comprises 2,429 faces and 4,548 non-faces, the test set: 472 faces and 23,573 non-faces.

``Regular'' Genetic Programming without any training data weighting, and the weighted version were both evolved over the course of 50 generations. The process was repeated 10 times each in order to gather an average measure of performance. The same random seeds were used for each. 

\paragraph{Fitness Function}
Choosing a suitable fitness function is a key problem in Genetic Programming. In a face detection task the problem poses a dilemma - whether to reward highly sensitive classifiers which detect every face, or to reward highly specific classifiers which produce very few false positives. The CBCL data set is difficult to please in some ways, as the training and test set have very different ratios of face and non-face. If each item of training data is given equal weight, then the classifier could evolve a solution with accuracy exceeding 98\% simply by returning false for everything. For the purposes of this experiment we gave both types equal weight - that is to say that sensitivity and specificity could each account for up to half of the individual's score.

\paragraph{Results}
The following results were found after investigating training data weighting and ERC optimisation. \footnote{training-data-weighting-results2.ods}.

In general it was found that ERC optimisation and training data weighting had little overall impact on the results for the training data. Interestingly, both approaches appeared to improve the score on the unseen data when measured according to the same metric.