Pascal VOC-style object recognition

Problem

Given an image, say whether it contains a bicycle, bus, car, cat, cow, dog, horse, motorbike, person or sheep. This was the Pascal project in the Scene Segmentation and Interpretation course of the VIBOT MSc at the Universitat de Girona, using the PASCAL VOC 2006 data: 256 training images and 523 test images, scored by the area under the ROC curve (AUC) for each class.

What I did

With Dousai Nayee Muddin Khan, over five weeks, I built and compared classification pipelines in MATLAB with VLFeat and PRTools.

Bag of words. Because about 90% of training images are negatives, we built the visual vocabulary from only the first 8 positive images per class, 80 images in all, with hierarchical k-means. That cut the dictionary build time without losing information.

Features. We started with colour SIFT, added a texture channel (local standard deviation over a 7×7 window), then moved to dense HOG on the RGB image with spatial ordering of the patches. 500 visual words beat 400, 800 beat 500, and 1,000 added nothing but compute time.

Classifiers. The default L2 nearest-neighbour ratio, multi-class SVM, a discriminative restricted Boltzmann machine, k-NN, naive Bayes and a homogeneous-kernel-map Chi2 SVM. The Chi2 kernel SVM of order 3 won consistently.

CNN features. Finally we replaced hand-crafted features with activations from the fc7 layer of a pretrained AlexNet, fed to the same kernel SVM.

Result

On the 523-image test set, the best hand-crafted pipeline (dense HOG, 800 words, Chi2 kernel SVM) reached an average AUC of 0.835, and AlexNet features with the same classifier reached 0.939. The gain was largest on the hardest class, person, which went from 0.589 to 0.844. The pretrained network also needed no training time: about 10 minutes of testing, against roughly 30 minutes of k-means and 20 minutes of training and testing for the HOG pipeline on a laptop CPU.