TClass was applied to both the Auslan and CBF tasks to get some idea of its accuracy compared with other techniques. Unfortunately, there is no central repository for temporal classification datasets (unlike conventional classification). This limits how rigorously the accuracy of TClass can be evaluated.
For the CBF task, all five PEPs were applied to the only channel. The global mean was the only global feature used. 266 instances of each class were generated. For ease of comparison with previous results, 10-fold cross validation was used.
For the Auslan task, all five PEPs were applied to the x, y and z channels. Increasing, Decreasing and Flat were applied to the remaining rotation and finger bend channels. The mean was calculated for each channel, as were the global maximum and minimum for the x, y, and z channels. Two tests were done, one with a small dataset consisting of 10 signs, and the other with a larger dataset with 95 signs. To simplify comparison with previously published results, 5-fold cross-validation was performed.
A naïve segmentation approach to feature extraction was used to allow comparison to TClass's performance. Each channel was divided into 10 segments along the time axis. The mean for each segment was computed and used as a feature. This created 10 features for the CBF task and 80 for the Auslan task. The same global features used with TClass were also combined with the segment features before being processed with the learners.
Two learning algorithms were compared: naïve Bayes and C4.5. The error rates of these experiments are shown in table 4.
Table: Error rates expressed as percentages (mean 
std error of the mean) on the learning tasks.
Table 4 shows some interesting patterns.
Firstly, the ``segment'' approach to feature extraction performs
surprisingly well, especially in the small Auslan task where there is
no significant difference
between it and the TClass approach. At 2.4 per cent error it
outperforms the best known published results for the CBF task,
including complex approaches involving local discriminant bases using
wavelets (3.75 per cent) [Saito, 1994] and trend episode analysis
(2.98 per cent) [Manganaris, 1997] but is beaten by TClass's 1.9
per cent error rate.
TClass has a better error rate overall compared to the naïve segmentation technique, regardless of the underlying learner.
Learner performance is consistent with previous results [Domingos and Pazzani, 1997]; in that in the Auslan task, which is characterised by few data (only 16 instances to train on per class), many classes and many attributes, naïve Bayes outperforms C4.5. On the other hand, in a domain with many data (240 training instances per class), few channels and few classes, such as the CBF task, C4.5 outperforms naïve Bayes.
The author had previously attempted both the small and large Auslan tasks using domain-specific, labour intensive, ad hoc feature extraction and iterative analysis of the errors to guide the inclusion of additional features, followed by feature selection and a nearest neighbour classifier [Kadous, 1995]. TClass exceeds the performance of this approach for the small Auslan task (2.5 per cent error rate vs 8.0 per cent error rate), but lags behind in the large Auslan task (24.0 per cent error rate vs 17.0 per cent error rate). Considering the PEPs are generic, this is hardly surprising.
It is also interesting to note the positive impact of using an ensemble of classifiers based on class. For example, in the large Auslan task, with TClass features and naïve Bayes learning, the average error rate of an individual class classifier was 36.1 per cent, whereas the combined classifier had an error rate of 24.0 per cent. Similarly in the CBF task, the average error rate of individual class classifiers was 5.6 per cent, compared with the error rate of the combined classifier of was 1.9 per cent.