Sign Classification using Local and MetaFeatures

1
Sign Classification using Local and Meta-Features
Marwan Mattar Allen Hanson Erik
Learned-Miller Computer Vision
Laboratory University of Massachusetts Amherst
2
vidi

• In collaboration with,
• Piyanuch Silapachote and Jerod Weinman
• Same system, different recognition component
• Does not use color information
• Different features / matching method
• A second classifier to help us find non-signs
• In the previous talk, we covered
• Motivation
• Previous Work
• System Overview
• Sign Detector

3
outline

• Importance of Signs
• System Overview
• What is Different?
• Recognition
• Experimental Results
• Conclusion

4
motivation

• Signs contain an immense amount of information

5
system overview
Saint Brigids Church
6
whats different?
7
recognition overview

• Overview of the recognition problem
• Training set sign database
• Test Instance an image region (detector output)
• Classification which sign is the image region
• A sign or sign class is the same physical sign

8
image features

• Two classes of image features
• Global features (i.e. shape / global texture)
• Local features (i.e. local patch descriptors)
• Local features are better suited for this domain
• More robust to occlusion and clutter
• Do not require a segmentation
• Local features are composed of
• Interest point detector
• Feature descriptor

9
sift example
10
recognition
Local Feature Matching
135
98
76


11
representation

• Each image in the database is represented as a
  set of local feature vectors
• We extract the same local features from the query
  image
• The matching process uses the local features from
  two images to obtain a similarity measure between
  them

12
local feature matching

• Find point correspondences (or matches) between
  two images
• Number of matches from image A to B (mAB)
• Given two sets of feature vectors (one for each
  image)
• find the number of vectors in image A whose
  distance to its nearest neighbour (in feature
  space) in image B falls below a pre-set threshold

13
matching example
Image A
mAB 50
Image B
14
mAB ? mBA

• Number of matches from image A to image B is not
  the same as that of B to A
• Bi-directional similarity measure between images
  A and B MAB MBA (mAB mBA) / 2

15
matching process
150
245
95
5
15
10
Query image
25
55
Match
30
Database
Match Scores
Class Scores
16
recognition
Meta-Features

• 10 match score differences
• highest score
• highest probability
• entropy
• 13 features total

Binary Classification
Output

• Either,
• most likely class
• classify as non-sign

Decide if most likely class should be chosen
17
meta-feature classifier

• Choosing the class with the highest score every
  time is dangerous
• An image region might not necessarily belong to
  one of the signs in the database

18
meta-features

• Our intuition is that there should be a large
  separation between the first class and the
  remainder
• We compute 13 features from the class scores that
  capture that information
• 10 class score differences (SD)
• Max class score (MS)
• Max posterior probability (MP)
• Entropy (H)

19
meta-features extraction
55
250
120
0.33
10
130
130
0.17
45
120
0.16
35
10
65
75
0.1
10
SD
65
0.09
20
10
15
5
55
0.07
10
20
35
0.05
5
0
20
0.03
5
65
10
250
0.01
MS
250
5
0.01
0.33
MP
120
0
0
H
0.76
Class Scores
Sorted Class Scores
Posterior Probability
Meta-Features
20
classification

• We have a binary classification problem
• Choose the most likely class
• Classify the region as non-sign

21
outline

• Importance of Signs
• System Overview
• What is Different?
• Recognition
• Experimental Results
• Conclusion

22
experimental design
23
data sets

• 35 class
• Contains 3325 images (95 for each class)
• Images are taken at 5 different times of the day
• Images exhibit in-plane rotation and lighting
  changes
• 65 class
• Contains 650 images (10 for each class)
• Taken at one specific time of day
• Images exhibit out-of-plane rotation
• Signs are manually segmented from the background

24
experiment 1

• Using 35 class data set
• Training set 5 instances from each class
• Testing set remaining 3150 instances
• Use only the match / class scores for
  classification
• Using match score 99.5 accuracy
• Using class score 99.5 accuracy

25
sample result
21
Match scores for top five matches
21
21
21
Query image
20
26
experiment 2

• Using 65 class data set
• Run 5 fold cross validation
• Use only the match / class scores for
  classification
• Using match score 90.4 accuracy
• Using class score 92.8 accuracy

27
sample result
18
Match scores for top five matches
15
13
13
Query image
13
28
experiment 3

• Using 65 class data set
• Omit 35 random classes from the training set
• Run 10 fold cross validation
• Using a Support Vector Machine classifier
• Use the class score and the meta-feature
  classification
• Accuracy 90.8
• Without the meta-feature classification we would
  not expect to do better than 39

29
time complexity

• m average number of local features per image
• n number of images in the database
• Image matching O(mlog(m))
• Local feature classification O(nmlog(m))
• Run time for one instance is less that one min
• Matching against 650 images
• Up to 500 features per image
• linear nearest neighbour search
• C/C implementation (no attempt to optimize)

30
conclusion

• We have presented a sign recognition system
• Very robust
• High accuracy
• Ability to classify query images as non-sign
• Fast running time

31
future work

• Use other local features in addition to SIFT
• Spatial information
• Look into other possible methods for classifying
  query images as non-sign
• Further optimization
• Color information
• Log time nearest neighbour search

32
Thank You!
Questions?