\chapter{Investigation into the Competitiveness of GP with other Techniques (DRAFT)}
\label{ch:gpcomparison}
In this Chapter the competitiveness of GP learned systems with other state-of-the-art vision approaches will be measured, and what degree of genericity has to be sacrificed in order to be competitive with specific approaches. This will be established on three problem domains.

\section{Face Detection}

\section{Classification}
Although Classification by GP is covered in much greater detail elsewhere in this thesis, we present here results on a public domain classification benchmark: the Statlog SatImage data set
\cite{Asuncion+Newman:2007}.  This database was generated from Landsat Multi-Spectral Scanner image data; it consists of the values of each pixel in a $3\times3$ neighbourhood across four spectral bands, thus producing 36 features in total.  Using these data, the central pixel is classified into one of six classes pertaining to the kind of soil present.  There are 4,435 items of training data and 2,000 samples of test data.

% Results from http://www.is.umk.pl/projects/datasets-stat.html

The best result on this data set, as published by the Statlog Project, was attained by a k-nearest neighbour algorithm with an error of 0.094. A Learning Vector Quantization (LVQ) neural net takes third place with an error of 0.105. A Bayes' classifier has an error of 0.147. Although the results from the LVQ method do not include its standard deviation on runs, a result of around 90\% on the data by a GP based approach would demonstate GP as being competitive. 

% TODO: GP results in this area - there must be some!

The SatImage data set, in common with many sets of real world data, consists of multiple classes with different prevalences. Of the six classes in the SatImage set, some have over 1,000 samples, and some have as few as 400. If one is to adopt a multiclass approach to the problem, then it is necessary to develop a single classifier that can tell each class apart. 

Critical to the training of this classifier is the fitness function for the GP process. As there is no particular concept of a mistake having higher cost than a correct hit, the fitness function may be expressed simply as:

\[
f = \frac{hits}{N}
\]

As is often the case in computer programming, in order to make an improvement, it is occasionally necessary to pull things apart before putting them back together again. Since this function offers equal weight to all training samples, but therefore unequal weight to each class it may result in a fitness landscape which is more exploitative than explorative. While the bias may prevent smaller classes from being properly included, the smaller classes are sufficiently small not to have a significant effect on the overall quality of the classification. For this reason a second fitness function was developed:

\[
f=\sum_{c=1}^{c=C}\left(\frac{mistakes_c}{N_c}\right)
\]

This function gives equal weight to each class, so there is no bias toward or against any particular class. This may help yield better solutions by GP.

% documents > Thesis-Experiments.ods > Tab 3
\subsection{Initial Results}
Initial results using Basic GP and Dynamic Range Selection ( to be discussed later) are presented below. The results are averages over 3 runs.

\begin{table}
  \begin{center}
  \caption{Statlog SatImage Results by GP Programs}
    \begin{tabular}{l|r|r|r|r}
      \hline\noalign{\smallskip}
      Technique & Training \% 	& Average \% 	& High \\
      GP (FF1)  &   	      	&  72.07 	& 76.19 \\
      GP (FF2)  &   	      	&  73.84 	& 81.39 \\
      \hline
    \label{tab:statlog}
    \end{tabular}
    \end{center}
\end{table}

Initial results do reveal that the second fitness function appears to yield better results on average, so our remaining experiments will use it. The results, however, are still significantly off the mark, so two further GP approaches were used.

\subsection{Boosting}
Although results in the 80\% territory are not competitive with the best classifiers, it may be possible to boost a number of classifiers up to the desired accuracy using AdaBoost. This was tried in the second experiment.

In this experiment the classifier was unable to outperform the results of GP alone.

\subsection{Binary Decomposition}
In a problem with a relatively large number of classes, it may be difficult to find a function that outputs distinct values for each class. In this case it may be better to decompose the problem into a series of component binary problems and see what level of accuracy can be achieved on a series of simpler problems. Once each problem is solved, an overall solution can be tied together. Such a process may take longer (as it requires more GP runs), but may yield better results. The results of this approach were collected in the third experiment.

%\subsection{Both}


