Measuring Usability of Biometrics Review of Experiences at NPL

1
Measuring Usability of BiometricsReview of
Experiences at NPL

• Linda Johnstone Sorensen
• Linda.Sorensen_at_npl.co.uk

2
Outline

• 10 Years ago
• BIOTEST project views on Usability Testing for
  Biometrics
• 19972008
• Observations on usability measurement during
• NPL Performance Usability Testing 1999
• NPL Performance Evaluations 2000 2005
• UKPS Biometrics Enrolment Trial
• 2008 ?
• How would we update proposals from BIOTEST?
• What works well ? What doesnt?
• What else should be included?

3
BIOTEST 1996-1997

• Collaborative EU Project
• Objective Develop methodologies to measure
  biometrics systems performance
• Focussed on
• Accuracy (I.e. error rates such as FMR, FNMR)
• Security (I.e. robustness to spoofing, etc.)
• Usability (Measuring ease of use etc.)
• 10 years later
• Methodologies metrics for assessing usability
  remain the least well established

4
BIOTEST (1997)Usability measure proposals

• What to measure?
• Ease of enrolment use
• Acceptability of enrolment use
• Invasiveness of enrolment use
• Levels of Supervision
• Enrolment risks
• Exceptional enrolees
• Not defined as usability
• Physical characteristics
• Dimensions of device
• Interfaces
• Environmental conditions
• Context of use

• How to measure?
• Quantitative measures
• Effectiveness
• E.g. successful enrolment
• Efficiency
• E.g. proportion of unproductive time
• Time taken vs that of experienced user,
• Qualitative measures
• Expert assessment
• Subject/Operator feedback

5
Usability testing of biometric systems for ATMs

• Evaluation conducted in 1998/99
• Assessment of verification accuracy and usability
  
• to guide a consideration of implementing
  biometrics in an ATM system
• 2 fingerprint 2 face recognition systems
• Opportunity to apply some of the methodology
  developed in BIOTEST

6
Usability testing of biometric systems for ATMs

• Verification accuracy assessed by Scenario Tests
• Emulating sample of typical bank customers 200
  test subjects from staff on Teddington site and
  some relatives
• Emulating enrolling bank clerk NPL staff on
  project team
• In-depth usability assessment
• 20 subjects (demographic balance)
• Enrolment, training, verification
• Separate observer
• Videotaping of the interactions
• Subjects interviewed before and after trial
• Open questions
• Short questionnaire for all 200 test subjects
• Closed questions

7
Change in opinions during the trial
Opinions Before using devices After the trial
Not comfortable using biometrics Face 4 / 20 Fingerprint 4 / 20 Face 6 / 20 Fingerprint 3 / 20
Preferred biometric Face 4 / 20 Fingerprint 4 / 20 No preference 12 / 20 Face 5 / 20 Fingerprint 9 / 20 No preference 6 / 20
Other perceptions Fingerprints perceived as easy to forge
Other perceptions Fingerprints viewed as more stable than faces
Other perceptions Confidence in system improved by rejections (when doing something wrong)
8
Comparison with other methods of verification

• Users asked how biometric devices compared to
  using a PIN
• 8/20 reported that biometric devices felt safer
  than PIN
• 9/20 positive to biometrics not requiring
  memorisation
• 10/20 said they would be willing to use the
  biometric system to take out cash at an ATM
• The reservations expressed loss of confidence due
  to problems experienced during the trials and
  hence

9
Enrolment and verification problems observed

• Face recognition
• Height
• Problems with eyes (e.g. infections)
• Wearing items such as glasses, hats, sunglasses
• Variations in hairstyle
• Time taken to enrol/verify

• Fingerprints
• Poor quality fingerprints (e.g. due to manual
  labour or accidents)
• Finger placement (e.g. just the tip of finger on
  sensor)
• Removing finger before image capture is complete

10
NPL 2005 evaluationImpact of usability for
operators

• NPL Biometric Evaluation 2005
• Pier 2-3 Handheld Iris camera
• Holding the camera steady
• Expected to be difficult
• Found easy after a practicing
• Intrusiveness
• Expected that subject would find the experience
  intrusive (camera held close to face).
• Findings Operators also feel uncomfortable
  holding the camera so close
• Performance differences between operators not
  significant

11
NPL 2005 evaluationImpact of usability for
operators
CorrectThicker reference lines on forehead and
chin

• NPL Biometric Evaluation 2005 3D-face enrolment
• Of the errors incurred, most attributable to poor
  enrolment
• Operator
• Instructs subject throughout
• Raises/lowers camera
• Checks subject
• Positioning
• Pose
• No smiling/talking
• Fringe
• Problems not always clear on on-screen display

Incorrect Too far Too close (d
(due to height)
12
NPL 2005 evaluationImpact of usability for
operators

• NPL Biometric Evaluation 20052D-face enrolment
• Warnings operator advice for non-optimal images
• Resulting in
• Perfect matching performance
• No false matches and
• No false non-matches
• Longer enrolment times

Shows multiple quality measures whether these
are adequate
Shows whether algorithmcorrectly locates eyes
Recommendation toaccept or retake
13
UKPS biometrics enrolment trial 2005

• 2003 Feasibility study on biometric ID cards
• Main performance unknowns are around usability by
  all sections of the population
• 2004/5 Biometrics enrolment trial
• Focus on
• Enrolment verification durations
• Customer perceptions and reactions
• Exception cases
• Demographic differences
• Included a significant proportion of disabled
  usersdemographically balanced quota
  groupvolunteering members of the public

14
Error rate by demographic group
Outlier groups showmore usability problems
1st attempt enrolment errors 1st attempt enrolment errors 1st attempt enrolment errors
Face Iris Fingerprint
Age 18-24 2 21 27
Age 35-34 3 18 29
Age 35-44 4 17 27
Age 45-54 4 21 29
Age 55-59 5 26 30
Age 60-64 5 30 31
Age 65 4 41 34
Hearing impairment 10 51 35
Learning disability 12 56 63
Physical impairment 12 52 50
Visual impairment 12 65 36
15
Problem with measuring participant experiences
and perceptions

• What drives feelings of positivity towards having
  facial, iris and fingerprint biometrics recorded?
• Positivity measured as
• Level of concern about the technique after
  demonstration
• Favourability towards its adoption
• See biometrics as strengthening of the security
  of ones passport
• Preventing identity fraud
• Preventing illegal immigration/working
• Combined measure of a participants experience and
  attitude towards the biometric devices

16
Context of use seems more influential than
experience of use

• Drivers for positivity analysed as
• Time taken for biometric capture
• Level of intrusion
• Ease of positioning
• Level of initial concern
• More than twice as influential as ease of
  positioning and time taken

17
UKPS Biometrics Enrolment Trial

• Actual time taken vs. User feedback on time taken
• User response not particularly correlated with
  actual time taken

18
Findings on quantitative usability measures

• Effectiveness
• Measured by error rates
• Failure to Enrol, False Non-match rate, False
  Match Rate
• These are and mainly determined by exception
  cases
• Exception cases (oldest / youngest / tallest /
  shortest / disabilities) often reveal more
  usability issues than typical users
• Efficiency
• Appropriate to measure use with
• Habituated subjects (familiar with using the
  systems) as well as
• Unhabituated subjects, with without operator
  assistance
• Many tests use mainly unhabituated subjects with
  assistance

19
Findings on qualitative usability measures

• Expert Assessment
• Useful part of many evaluations revealing errors
  due to usability
• Assessor independent of the operator/subject
  interaction
• Some (though not all) issues can be assessed by
  experts without observing real use
• Possibility of checklist?
• Subject/Operator feedback
• Limited usefulness as indicator of usability
• Users goals not always the same as the systems
  goals
• Feedback is influenced by factors other than
  operational effectivenessE.g.
• Pre-conceptions
• Novelty of the experience

20
Outstanding issues - 1

• Acceptability of a biometric system
• Classes of users
• People problems some groups of people have
  problems with due to physical appearances
• Technology may be challenging for certain user
  groups
• Ergonomic concerns for specific user groups
• User behaviour
• Lack of behaviour compliance when people dont
  do what you want them to do (ideal system do not
  require explanations).
• Operator concerns with biometric systems

21
Outstanding issues - 2

• How do we combine measures of usability in
  biometric systems?
• We cannot always see what is most usable from
  measures of performance alone
• We cannot see what is most usable from user
  feedback alone
• Most errors observed in trials with biometric
  systems are not errors of the system, but errors
  in its use!
• Trust
• How do we include trust?
• Trust in the technology
• E.g. people like to see a failure once in a
  while, to confirm that the system works
• Trust in the system
• E.g. security of the system
• Storage of biometric data remotely or in
  personal chip-card

22

• Thank you!