Parametric measures to estimate and predict performance of identification techniques

1
Parametric measures to estimate and predict
performance of identification techniques
STATISTICAL METHODS FOR COMPUTATIONAL
EXPERIMENTS IN VISUAL PROCESSING COMPUTER
VISION NIPS 2002

• Amos Y. Johnson Aaron Bobick

2
Setup for example

• Given a particular human identification technique

3
Setup for example

• Given a particular human identification technique
• This technique measures 1 feature (q) from n
  individuals

- 1D Feature Space -
4
Setup for example

• Given a particular human identification technique
• This technique measures 1 feature (q) from n
  individuals
• Measure the feature again

- 1D Feature Space -
5
Setup for example

• Given a particular human identification technique
• This technique measures 1 feature (q) from n
  individuals
• Measure the feature again

Probe
Gallery
- 1D Feature Space -
6
Setup for example

• Given a particular human identification technique
• This technique measures 1 feature (q) from n
  individuals
• Measure the feature again

Target
Probe
Gallery
For template
- 1D Feature Space -
7
Setup for example

• Given a particular human identification technique
• This technique measures 1 feature (q) from n
  individuals
• Measure the feature again

Target
Imposters
Probe
Gallery
For template
- 1D Feature Space -
8
Question

• For a given human identification technique, how
  should identification performance be evaluated?

9
Possible ways to evaluate performance

• For a given classification threshold, compute
• False accept rate (FAR) of impostors
• Correct accept rate (HIT) of genuine targets

10
Possible ways to evaluate performance

• For various classification thresholds, plot
• Multiple FAR and HIT rates (ROC curve)

11
Possible ways to evaluate performance

• For various classification thresholds, plot
• Multiple FAR and HIT rates (ROC curve)
• Compute area under a ROC curve (AUROC)

Probability of correct classification
12
Possible ways to evaluate performance

• For various classification thresholds, plot
• Multiple FAR and HIT rates (ROC curve)
• Compute 1 - area under a ROC curve (1 -AUROC)

13
Problem

• Database size
• If the database is not of sufficient size, then
  results may not estimate or predict performance
  on a larger population of people.

1 - AUROC
14
Our Goal

• To estimate and predict identification
  performance with a small number subjects

1 - AUROC
15
Our Solution

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

16
Our Solution

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

Probability that an imposters feature vector is
within the measurement variation of a targets
template
17
Our Solution

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

Probability that an imposters feature vector is
closer to a targets template, than the targets
feature vector
18
Our Solution

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

EC 1 - AUROC
19
Expected Confusion

• Probability that an imposters feature vector is
  within the measurement variation of a targets
  template

20
Expected Confusion - Uniform

• The templates of the n individuals, are from an
  uniform density
• Pp(x) 1/n

P(x)
Pp(x)
1/n
x
- 1D Feature Space -
21
Expected Confusion - Uniform

• The measurement variation of a template is also
  uniform
• Pi(x) 1/m

P(x)
Pi(x)
1/m
Pp(x)
1/n
x
- 1D Feature Space -
22
Expected Confusion - Uniform

• The probability that an imposters feature vector
  is within the measurement variation of template
  q3 is the area of overlap
• True if m ltlt n

P(x)
Pi(x)
1/m
Pp(x)
1/n
x
- 1D Feature Space -
23
Expected Confusion - Uniform

• The probability that an imposters feature vector
  is within the measurement variation of any
  template q
• True if m ltlt n

P(x)
Pi(x)
1/m
Pp(x)
1/n
x
24
Expected Confusion - Gaussian

• Following the same analysis, for the
  multidimensional Gaussian case

25
Expected Confusion - Gaussian

• Following the same analysis, for the
  multidimensional Gaussian case
• True if the measurement variation is
  significantly less then the population variation

Probability that an imposters feature vector is
within the measurement variation of a targets
template
26
Expected Confusion - Gaussian

• Relationship to other metrics
• Mutual Information
• The negative natural log of the EC is the mutual
  information of two Gaussian densities

27
Transformed Expected-Confusion

• Probability that an imposters feature vector is
  closer to a targets template, than the targets
  feature vector

28
Transformed Expected-Confusion

• First We find the probability that a targets
  feature vector is some distance k away from its
  template

29
Transformed Expected-Confusion

• Second We find the probability that an
  imposters feature vector is less than or equal
  to that distance k

30
Transformed Expected-Confusion

• Therefore The probability that an imposters
  feature is closer to the targets template, than
  the targets feature (for a distance k) is

Target
Imposters
k
31
Transformed Expected-Confusion

• Therefore The probability that an imposters
  feature is closer to the targets template, than
  the targets feature (for any distance k) is

Target
Imposters
k
32
Transformed Expected-Confusion

• Therefore The expected value of this probability
  over all targets templates is

x
33
Transformed Expected-Confusion

• Next Replace the density of the distance between
  a targets feature-vectors and its template q

34
Transformed Expected-Confusion

• Answer Probability that an imposters feature
  vector is closer to a targets template, than the
  targets feature vector

35
Transformed Expected-Confusion

• This probability can be shown to be one minus the
  area under a ROC curve
• Following the analysis of Green and Swets (1966)

36
Transformed Expected-Confusion

• Integrate With these assumptions

37
Transformed Expected-Confusion

• Integrate With these assumptions

38
Transformed Expected-Confusion

• Integrate With these assumptions

39
Transformed Expected-Confusion

• Integrate With these assumptions

40
Transformed Expected-Confusion

• Integrate Probability that an imposters
  feature vector is closer to a targets template,
  than the targets feature vector

41
Transformed Expected-Confusion

• Compare EC with 1 - AUROC

EC 1 - AUROC
42
Conclusion

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

Probability that an imposters feature vector is
closer to a targets template, than the targets
feature vector
43
Conclusion

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

Probability that an imposters feature vector is
within the measurement variation of a targets
template
Probability that an imposters feature vector is
closer to a targets template, than the targets
feature vector
44
Conclusion

• Derive two parametric measures
• Expected Confusion (EC)
• Transformed Expected-Confusion (EC)

Probability that an imposters feature vector is
within the measurement variation of a targets
template
Probability that an imposters feature vector is
closer to a targets template, than the targets
feature vector
45
Future Work

• Developing a mathematical model of the cumulative
  match characteristic (CMC) curve
• Benefit To predict how the CMC curve changes as
  more subjects are added