Parametric measures to estimate and predict performance of identification techniques

1
Parametric measures to estimate and predict
performance of identification techniques
Amos Johnson Aaron Bobick
2
Question?

• For a given identification technique, how should
  identification performance be evaluated?

3
Whats Identification?

• Recognition
• Who is this person?
• One to many matching problem
• Verification
• Is this person, who they claim to be?
• One to one matching problem

4
Whats Identification?

• Recognition
• Who is this person?
• One to many matching problem
• Verification
• Is this person, who they claim to be?
• One to one matching problem

We address the verification issue
5
Verification

• Possible ways to evaluate verification
  performance
• False acceptance rate (FAR) of impostors
• False reject rate (FRR) of a genuine target
• Plot a ROC curve of various FAR and FRR rates
• Etc

6
Problems

• Database size
• If the database is not of sufficient size, then
  results may not estimate or predict performance
  on a larger population of people.
• Database content
• If the subjects in the database are not
  representative of the general population, then
  results may be bias.

7
Problems

• Database size
• If the database is not of sufficient size, then
  results may not estimate or predict performance
  on a larger population of people.
• Database content
• If the subjects in the database are not
  representative of the general population, then
  results may be bias.

We address the issue of database size
8
Our Approach

• What do we want?
• To estimate and predict verification performance
• What do we have?
• A small, yet representative, database of subjects
  
• How do we do it?
• The expected confusion and transformed
  expected-confusion metrics

9
Our Approach

• So what is special about these metrics?
• They are parametric, so they will converge to
  their steady-state value with far less subjects
  then the non-parametric measures currently being
  used.

10
Outline for Remainder of Talk

• Expected confusion
• Estimates uncertainty of identification
• Transformed-expected confusion
• Finds probability of incorrect identification
• Experiment Evaluation
• Synthetic data
• Data from our human identification by gait
  technique

11
Expected Confusion

• To estimate the uncertainty in identification
• We need to estimate the
• Probability density of a feature vector x
  for a population
• Probability density of a feature vector x
  for an individual

12
Expected Confusion

• Uniform density as an example
• Measurement x0

13
Expected Confusion

• Uniform density as an example
• Measurement x0
• Pi(x) 1/M

14
Expected Confusion

• Uniform density as an example
• Measurement x0
• Pi(x) 1/M
• Pp(x) 1/N
• N gt M

15
Expected Confusion

• Uniform density as an example
• Measurement x0
• Pi(x) 1/M
• Pp(x) 1/N
• N gt M
• The uncertainty in identity is the area of
  overlap of these two densities

16
Expected Confusion

• Average uncertainty in identity, or Expected
  Confusion, is the expected value of the area of
  overlap over an entire population

17
Expected Confusion

• Our model
• Individual variation
• Population variation
• Densities are estimated directly from the data
• Population variation reasonably greater than the
  individual variation.

18
Expected Confusion

• Analytically applying this analysis to the
  multidimensional Gaussian case

Define The ratio of the average individual
variation of a feature vector to that of the
population variation of the same feature vector.
19
Expected Confusion

• For example Estimate a population volume from
  samples

x2
x1
20
Expected Confusion

• Find the average individual volume

x2
x1
21
Expected Confusion
x2
x1
22
Expected Confusion

• Relationship to other metrics
• Mutual Information
• The negative natural log of the EC is the mutual
  information of two Gaussian densities

23
Expected Confusion

• Intuitively, identification techniques with a low
  expected confusion will also have a low
  probability of incorrectly discriminating between
  individuals.
• By applying a transformation to the expected
  confusion we can find that probability.

24
Transformed EC

• Probability of incorrect identification
• Find probability by
• Relating EC to area under a ROC curve

25
Area Under a ROC curve

• Define
• q is a genuine targets ID template

26
Area Under a ROC curve

• Define
• q is a genuine targets ID template

Target
27
Area Under a ROC curve

• Define
• q is a genuine targets ID template

Imposters
28
Area Under a ROC curve

• Define
• Probability of making a correct decision as to
  whether a given sample is the target or imposter
  for threshold k

Threshold k
29
Area Under a ROC curve

• Define
• Probability of making a correct decision as to
  whether a given sample is the target or imposter
  for threshold k

Threshold k
30
Area Under a ROC curve

• Probability of making a correct decision over all
  ID temples and thresholds
• Assuming ID temples are from Pp(q)

31
Area Under a ROC curve
32
Transformed EC

• To derive an analytic relation between EC and
  AUROC, we make some simplifying assumptions
• d-dimensional Gaussian densities for the
  individual and population densities for a ID
  feature vector
• Spherical covariance matrices for the individual
  and population densities,
  , respectively
• Statically independent features
• Population variation reasonably greater than the
  individual variation

33
Transformed EC

• We start with the probability of making a correct
  decision

34
Transformed EC

• We negate the equation, to find the probability
  of making an incorrect decision

to
from
35
Transformed EC

• Next, approximate the density of the distance
  from a particular individuals feature vectors
  from its template q with a general density for
  any individual

to
from
36
Transformed EC

• This density can be represented by the square
  root of the chi-square density

37
Transformed EC

• Next, we need to derive the probability that an
  imposters feature vector is within k of an ID
  template q

38
Transformed EC

• Assuming that the population density is constant
  over the range that the individual density is
  non-zero then it can be approximated by

39
Transformed EC

• With the stated assumptions, we solve this
  equation and arrive at

40
Transformed EC

• Since we assumed, spherical covariance matrixes
  with independent features,

41
Transformed EC

• Since we assumed, spherical covariance matrixes
  with independent features

42
Transformed EC

• The relationship to area under the ROC curve is

43
Transformed EC
44
Outline of Talk

• Expected confusion
• Estimates uncertainty of identification
• Transformed-expected confusion
• Finds probability of incorrect identification
• Experiment Evaluation
• Synthetic data
• Data from our human identification by gait
  technique

45
Outline of Talk

• Expected confusion
• Estimates uncertainty of identification
• Transformed-expected confusion
• Finds probability of incorrect identification
• Experiment Evaluation
• Synthetic data
• Data from our human identification by gait
  technique

46
Outline of Talk

• Expected confusion
• Estimates uncertainty of identification
• Transformed-expected confusion
• Finds probability of incorrect identification
• Experiment Evaluation
• Synthetic data
• Data from our human identification by gait
  technique

47
Experimental Evaluation

• Synthetic Data

48
Experimental Evaluation

• Synthetic Data
• Generate ID templates from population Gaussian
  distribution

49
Experimental Evaluation

• Synthetic Data
• Generate ID templates from population Gaussian
  distribution
• Use each ID template as the mean of an individual
  Gaussian distribution to generate individual
  data-points

50
Experimental Evaluation

• Synthetic Data
• Remove ID templates
• The Gaussian densities have
• Spherical covariance matrixes
• Independent features
• Where,
• Show results for dimension d 1 to 5

51
Experimental Evaluation
52
Experimental Evaluation
53
Experimental Evaluation
54
Experimental Evaluation

• Convergence of vs. 1 AUROC

55
Experimental Evaluation

• Convergence of vs. 1 AUROC

u
10 subjects 1 AUROC 190 subjects
56
Experimental Evaluation

• For synthetic data
• The EC decreases as the dimension increase
• The transformed EC converges to its steady-state
  value faster than 1-AUROC

57
Experimental Evaluation

• Static body parameters
• Measure 4 distances
• Head to ground (d1)
• Pelvis to head (d2)
• Foot to pelvis (d3)
• Left foot to right foot (d4)
• Measured during gait cycle
• Form 4 feature vectors

58
Experimental Evaluation

• 3D-motion capture database
• 20 subjects walking
• 6 sequences per subject
• 6 walk vectors per subject

59
Experimental Evaluation
60
Experimental Evaluation
61
Conclusion

• What did we want?
• To estimate and predict verification performance
• What did we have?
• A small, yet representative, database of subjects
  
• How did we do it?
• Expected confusion
• Uncertainty of identification
• Transformed expected-confusion
• Probability of incorrect identification

62
Questions Answers