Computer Assisted Visual InterActive Recognition (CAVIAR)

1
Computer Assisted Visual InterActive
Recognition(CAVIAR)

• Jie Zou
• RPI ECSE DocLab

Advisor Prof. George Nagy Committee Prof.
Qiang Ji Prof. Robert B. Kelley Prof. Mukkai
Krishnamoorthy
2
Agenda

• Introduction
• Related research
• CAVIAR methodology
• Interactive segmentation
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

3
Agenda

• Introduction
• Related research
• CAVIAR methodology
• Interactive segmentation
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

4
Motivation

• All operational systems require human assistance
  (preprocessing, handling rejects).
• CAVIAR makes parsimonious use of human visual
  talent throughout the process rather than only at
  the beginning or the end.

5
Scope of CAVIAR

• Visual pattern recognition only
• Each CAVIAR system addresses a specific domain
• Many class classification
• Low throughput

6
Research Goals

• Allocation of human and machine responsibilities
• Mathematical model
• Framework and design principles
• Prototype CAVIAR systems

7
Agenda

• Introduction
• Related research
• CAVIAR methodology
• Interactive segmentation
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

8
Content-Based Image Retrieval

• Typical search of CBIR system
• Submit a query image.
• Specify the relative importance of the features.
• Relevance feedback (label the retrieved images as
  acceptable or not-acceptable).
• Iterates until user finds the desired image.

9
CBIR vs. CAVIAR
CBIR
CAVIAR
Subjective retrieval
Objective classification
User judges retrieval results
Statistical decision boundary
Machine weights features
User weights features
Narrow domain
Broad domain
Relevance feedback
Relevance feedback
Model adjustment
10
Flower Recognition

• Little research on automatic flower recognition
• M. Das, R. Manmatha, and E.M. Riseman, Indexing
  flower patent images using domain knowledge,"
  IEEE Intelligent Systems, vol. 14, no. 5, pp.
  24-33, 1999.

11
Face Recognition

• Started in 1960s. Now, most active pattern
  recognition application
• Eigenface, dominant method
• Geometrical feature models are appropriate for
  interactive recognition

12
Agenda

• Introduction
• Related research
• CAVIAR methodology
• Interactive segmentation
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

13
Psychophysics

• Attneave (1954) the nature of redundancy in
  visual stimulations, and information is
  concentrated along contours.
• Miller (1956) plus or minus 7

14
Allocation of Human and Machine Responsibilities
Conventional System
CAVIAR
15
Notation
CAVIAR state
Model parameters
Features
Index vector
Training set
Label
16
Formal Description (1)

• Finite state machine
• Initial state created by
• Model building
• Feature extraction
• Indexing

17
Formal Description (2)

• Model manipulation leads to a state transition
  from state n to state n1
• Model building ,
• Feature extraction
• Indexing

18
Formal Description (3)

• The task can terminate at any state by
  identification.

19
Illustration (Video)
20
Agenda

• Introduction
• Related research
• CAVIAR methodology
• Interactive segmentation
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

21
Notation

• Parametric boundary
• Exact boundary
• Foreground region or
• Background region or
• Radius vectorintersectsat , andat .

22
Training Color Distributions
23
Training Circle Parameter Distributions
24
Training Deviation of Circular Model From Exact
Boundary
ß5.52
25
Automatic Segmentation Circle Partition
Use a circle to isolate a region, which contains
mostly flower colors.
26
Automatic Segmentation Generate Boundary
Likelihood Map
Distance to the circle
Magnitude of color gradient
Boundary pixels should be close to the circle,
and have high color gradient.
27
Automatic Segmentation Deformation on BLM

• Circle center a foreground seed, and four
  corner pixels background seeds
• Foreground and background regions compete with
  each other to expand.
• Eventually, converge to the watershed of the seed
  pixels on the BLM.

28
Advantage of BLM over Gradient Map
29
Examples of the Result of Automatic Segmentation
30
Interactive Correction (Video)
31
Segment Flower Pictures with Interactive
Correction

• 1078 flowers from 113 speciesBorjan Gagoski and
  Adam Callahan
• 5.7 seed pixels, 15.2 seconds per picture Greenie
  Cheng

32
Agenda

• Motivation
• Related research
• Interactive segmentation
• CAVIAR methodology
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

33
Flower Database

• 320 by 240 resolution
• Highly variable illumination, and complex
  background
• 216 samples from 29 classes for development
• 612 samples from 102 classes for evaluation

34
Rose Curve Model

• Parametric curve withsix parameters.
• Flowers are composed of petals, which
  havecircular symmetry.
• When n0, rose curvereduces to circle.

35
Classification Features
number of petals.
the ratio of outer to inner radius.
first three order moments of the hue
andsaturation histograms
36
CAVIAR-Flower GUI
37
CAVIAR-Flower (Video)
38
Evaluation

• CAVIAR compared to human-alone and machine-alone
• Machine learning
• Decision directed approximation
• Finite state machine calibration

39
Experimental Protocol

• Thanks to Borjan Gagoski

40
CAVIAR Compared to Human-Alone and Machine-Alone

• Significantly reduce the recognition time
  compared to human-alone
• Significantly increase the accuracy compared to
  machine-alone

41
Accuracy of Initial Automatic Recognition
T2 5 labeled
T3 1 labeled
T4 1 labeled 2 pseudo
T5 1 labeled 4 pseudo
42
Rank Order after Initial Automatic Recognition
T2 5 labeled
T3 1 labeled
T4 1 labeled 2 pseudo
T5 1 labeled 4 pseudo
43
Time of Interactive Recognition
T2 5 labeled
T3 1 labeled
T4 1 labeled 2 pseudo
T5 1 labeled 4 pseudo
44
Accuracy of Interactive Recognition
T2 5 labeled
T3 1 labeled
T4 1 labeled 2 pseudo
T5 1 labeled 4 pseudo
45
Observations about Machine Learning

• Initialized with a single training samples per
  class.
• Self-learning user classified pseudo-labeled
  samples improve the performance.
• Performance of T5 is close to T2, suggesting that
  instead of initializing with many training
  samples, we can trust the systems self learning.

46
Calibration of Finite State Machine

• 52 samples are immediately confirmed.
• 90 samples are identified by 3 adjustments.
• The probability of success on each adjustment is
  just over one half.

47
Summary of CAVIAR-Flower

• Parameterized rose curve to model the flowers.
• Display the rose curve and let user adjust it if
  necessary.
• The evaluation of the system shows advantages of
  CAVIAR system.

48
Agenda

• Motivation
• Related research
• Interactive segmentation
• CAVIAR methodology
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

49
CAVIAR - Face

• Not to implement a state-of-the-art face
  recognition system.
• To demonstrate the wider applicability of CAVIAR
  methodology.
• 400 FERET pictures of 200 subjects (ba and
  bk series)

50
Face model and features

• Face model contains only two pupils.
• An automatic facial feature detection program
  locates the other 26 points.
• Thanks to Yan Tong, Zhiwei Zhu, and Dr. Qiang
  Ji.

51
Face model and features

• Similarity transform all 28 points to place two
  pupils at (-1,0) and (1,0).
• The normalized coordinates of the 26 points are
  the features for classification.

52
Interactive Recognition
53
Interactive Recognition
54
Experiments

• Three types of experiments human browsing only,
  BAAR, and CAVIAR
• 200 training samples, one from each person
• 200 test samples, one from each person
• 15 subjects, each classifies 40 faces

55
CAVIAR Compared to Human-Alone, Machine-Alone,
and BAAR
56
Summary of CAVIAR-Face

• A simple face recognition system
• Still clearly shows the advantages of CAVIAR
  approach

57
Agenda

• Motivation
• Related research
• Interactive segmentation
• CAVIAR methodology
• CAVIAR flower recognition system
• CAVIAR face recognition system
• Conclusions

58
Contributions

• Domain specific geometrical models to mediate
  between human and computer
• Allocation of human and machine in interactive
  visual recognition task
• Model building (primarily human)
• Feature extraction (primarily machine)
• Classification (collaborative)
• Finite state machine
• Two experimental CAVIAR systems

59
CAVIAR Design Principles

• Machine tries its best to infer the model and the
  candidates.
• Machine should display its results, and let the
  user correct any errors.
• Machine always accepts human correction.

60
Observations

• CAVIAR system can significantly reduce the time
  compared to unaided human, and significantly
  increase the accuracy compared to unaided machine
  (without years of RD).
• CAVIAR system can be initialized with a single
  training sample per class.
• CAVIAR system demonstrates self-learning ability.
• CAVIAR is better than BAAR, and provides more
  opportunities for machine learning from human
  intervention.

61
Directions For Future Work

• Other applications
• Mobile CAVIAR
• Collaborative learning
• Machine suggestion to human

62
Other applications Face

• Face under head rotation, occlusion, and changes
  of illumination and facial expression
• Elastic bunch graph model, and others

63
Other applications Skin Diseases

• Nearly 1000 diagnoses
• Big image atlases available
• John Hopkins dermatology image atlas
• University of Erlangen

64
Other applications Fish
Alabama Shad
Black Crappie
Atlantic Sturgeon
Blue Gill

• U.S. Fish wild life service

65
Other Applications

• Only a few successful visual recognition
  applications OCR, fingerprint, ??.
• CAVIAR may make others viable.
• Education

66
MobileCAVIAR
Courtesy to Abhishek Gattani
67
Mobile CAVIAR

• Client-server architecture, with a hand-held
  computer as a client, connecting to an Internet
  server.
• The easy accessibility of multiple pictures poses
  an interesting information fusion problem.

68
Collaborative Learning

• When using CAVIAR, also collect training samples
• Pattern identified by some users can benefit peer
  users
• Extension of the concept of Open Mind Initiative

69
Machine Suggestion to Human

• Currently, machine suggestions only annoy users.
• Eventually may be able to make useful suggestions
  to acquire or refine particular features.

70
Thank you!

• Questions and Comments

71
Motivation

• Successful visual pattern recognition systems
  require several years RD.
• All operational systems require human assistance
  (preprocessing, handling rejects).
• Pronounced difference between human and machine.
• Human gestalt tasks, object-background
  separation, etc.
• Machine computing complex features, evaluating
  conditional probabilities, etc.

72
CAVIAR Introduction
It may be more effective to make parsimonious use
of human visual talent throughout the process
rather than at beginning or end.
73
Human and Machine Visual Perception

• Visual perception is a kind of computation.
• Two theories of human visual perception.
• Recognition by components (RBC).
• View-based recognition.
• Machine visual perception is still in its infancy.

74
Human Computer Interaction

• HCI is a wide topic.
• Graphic user interface
• Attentive user interface
• Exploratory data analysis
• Graphically display the data.
• For designing a classifier, rather than actual
  classification.
• Pattern recognition systems with human in the
  loop
• Preprocessing
• Handling rejects

75
CAVIAR vs. CBIR

• CAVIAR is objective classification, CBIR is
  subjective retrieval.
• Users arbitrarily emphasizing particular features
  is not a good idea in CAVIAR.
• CAVIAR is on narrow domain, CBIR is usually on
  broad domain.
• Relevance feedback is useful in CAVIAR, but more
  precise human-computer communication can be
  established through domain-specific models.

76
Unsupervised Decision-directed Approximation

• Assume that the decision of the classifier is the
  true label of the unknown.
• Works well if
• Little overlap among the class-conditional
  densities.
• Initial classifier is reasonably good.
• Used in our experiment.
• Benefits from human intervention.

77
Allocation of Human and Machine Responsibilities
78
Allocation Human and Machine Responsibilities

• We give up strong segmentation for parametric
  model fitting.
• Model building is primarily a human
  responsibility.
• Feature extraction is done primarily by machine.
• Classification requires human-machine
  collaboration.
• Human adjusts the model instance.
• Final human confirmation.

79
CAVIAR Compared to Machine-alone and Human-alone
(1)

• Cost function is a convex combination of error,
  E, and time, T.
• Machine-alone EEM, T0.
• Human-alone E0, TTH.
• CAVIAR Ei, Ti.

80
CAVIAR Compared to Machine-alone and Human-alone
(2)
81
CAVIAR Compared to Browsing-After-Automatic (BAA)

• The user browses to find the correct candidate.

82
CAVIAR Compared to Browsing-After-Automatic (BAA)

• In principle, CAVIAR can never be worse than BAA.
• The practical scenarios are much more
  complicated, and very difficult to model.
• CAVIAR provides more opportunity for machine
  learning from human intervention.

83
Automatic Image Segmentation

• Intensively studied for decades, no
  off-the-shelf solution.
• Segmentation methods includes
• Clustering.
• Edge-linking.
• Region splitting and merging.
• Hybrid optimization.
• General image segmentation is extremely
  difficult.

84
Automatic Segmentation Circle Partition
a0.025.
Use a circle to isolate a region, which contains
mostly flower colors.
85
Interactive Image Segmentation

• From human initialization, deform automatically
  to the object boundary.
• Snakes or active contours.
• Seeded region growing.
• Human and computer collaborate step by step.
• Intelligent scissors.
• Intelligent paint.

86
Image Segmentation Evaluation

• Analytical methods.
• Analyze the principles and properties.
• Empirical methods.
• Compare the results to the ground-truth
  segmentation.
• We believe that segmentation can only be
  evaluated according to its purpose.

87
Strong Segmentation Vs. Weak Segmentation

• Strong segmentation
• Division of the image into regions region F
  contains the pixels of object O, and nothing
  else. FO.
• Weak segmentation
• Partition of the image into conspicuous object
  region F without locating the precise boundary.
  FO.
• Strong segmentation may not be necessary for
  pattern recognition.

88
Outline of the Interactive Segmentation Procedure

• Training
• Color distributions.
• Circle parameter distributions.
• Statistics of the deviation of the exact boundary
  from the circle.
• Automatic segmentation
• Fit circle by maximizing posterior.
• Generate boundary likelihood map.
• Watershed of seeds is the boundary.
• Interactive correction
• Shriek or expand the foreground region by
  introducing more seeds.

89
Interactive Correction
90
Evaluation of Automatic Segmentation

• 174 test samples, 187 training samples from 29
  classes.
• Region discrepancy (RD) 25.
• Boundary discrepancy (BD) 8.7 pixels.
• Classification rank order (RO) 3.84.

91
Rank Order is Not Correlated with RD and BD
92
Experiments Comparing Strong and Weak Segmentation

• We compared classification results of strong
  segmentation and two (rose curve and circle) weak
  segmentations.
• Test 612 unknown samples, with 510 training
  samples.

93
Automatic Segmentation is Not Reliable.
The classification result of each kind of
manualsegmentation is much better than that of
the corresponding automatic segmentation.
94
With Reliable Segmentation
The improvement in classification from
strongsegmentation is limited even if reliable
segmentationcan be achieved.
95
When Reliable Segmentation cannot be Achieved
The simpler weak segmentation is preferred if
reliable segmentation can not be achieved.
96
Strong Segmentation may not be necessary for
Pattern Recognition

• Strong segmentation is generally difficult to
  achieve either automatically or interactively.
• Strong segmentation is obviously over-fitted to a
  particular sample, not the class that the sample
  belongs to.
• Our experiments empirically show this.

97
Summary of Interactive Image Segmentation

• A procedure for model-based interactive
  segmentation.
• The segmentation results approaches the exact
  boundary with little human efforts.
• Extensions.
• For highly textured image.
• Adjusting circle after human clicking.
• More elaborate models for both parametric
  partition and boundary likelihood map.

98
Analytical Solution to Fitting a Rose Curve
99
Training
100
Recognition
101
Accuracy of Initial Automatic Recognition
102
Rank Order after Initial Automatic Recognition
103
Top-3 Accuracy of Initial Automatic Recognition
104
Time of Interactive Recognition
105
Accuracy of Interactive Recognition
106
Rank Order After Automatic and Interactive
Recognition

• Subjects do adjust the rose curves, which reduces
  the median rank orders.

107
CAVIAR Compared to BAA

• Initial automatic classification is good. The
  average rank order is 6.6 (2 clicks).
• The subjects are first-time user, not able to
  always find the right strategy.

108
Training

• Manually entered the positions of 28 points for
  each of 200 training pictures.

The difficulty of face recognition with a
geometrical face model is due to unreliable
automatic feature point detection.
Human correction of eye centers is already very
helpful.
109
New Human-Computer Interfaces

• Quality of model parameters.
• Associate confidence measures to model
  parameters, which relates to classification
  features.
• Assume the parameters adjusted by human have high
  confidence.
• Occlusion can be communicated to the computer.
• Categorical features.
• Divide a family into clusters. (male/female).
• Category-specific classifiers.