M.A.Sc. Thesis Presentation Automated Reading Assistance System Using Point-of-Gaze Estimation

1
M.A.Sc. Thesis Presentation Automated Reading
Assistance System Using Point-of-Gaze
Estimation

• Jeffrey J. Kang
• Supervisor Dr. Moshe Eizenman
• Department of Electrical and Computer Engineering
• Institute of Biomaterials and Biomedical
  Engineering
• January 24, 2006

2
Introduction

• Reading
• Visual examination of text
• Convert words to sounds to activate word
  recognition
• We learn appropriate conversions through
  repetitive exposure to word-to-sound mappings
• Insufficient reader skill or irregular spelling
  can lead to failed conversion assistance is
  required

Objective Develop an automated reading
assistance system that automatically vocalizes
unknown words in real-time on the readers
behalf. The system should operate within a
natural reading setting.
3
What We Need To Do Step 1

• Identify the word being read, in real-time
• Detect when the word being read is an unknown
  word
• Vocalization of the unknown word

4
Identifying the Word Being Read

• Identify the viewed word using point-of-gaze
  estimation
• Point-of-gaze is
• Where we are looking with the highest visual
  acuity region of the retina
• Intersection of the visual axis of each eye
  within the 3D scene
• Intersection of the visual axis one eye with a 2D
  plane

5
Point-of-Gaze Estimation Methodologies

• 1. Head-mounted 2. Remote
• 
  (no head-worn components)

6
Head-mounted Point-of-Gaze Estimation
pupil centre
corneal reflections

• Based on principle of tracking the pupil centre,
  and corneal reflections to measure eye position
• Point-of-gaze is estimated with respect to a
  coordinate system attached to the head

7
Point-of-Gaze in Head Coordinate System

• Point-of-gaze is measured in the head coordinate
  system, and placed on the scene camera image

8
Locating the Reading Object

• The position of the reading object is determined
  by tracking markers

9
Mapping the Point-of-Gaze

• Establish point correspondences from
• the estimated positions of the markers in the
  scene image
• the known positions of the markers on the reading
  object
• Homographic mapping of point-of-gaze from scene
  camera image to reading object coordinate system

10
Identify the Reading Object

• Extract the barcode from the scene camera image
  to identify the reading object (e.g. page number)
• Match barcode to database of reading objects to
  determine what text is being read

11
Identifying the Word Being Read

• Using the mapped point-of-gaze, identify the word
  being read by table lookup

12
Sample Reading Video
13
Sample Reading Video
14
Mapping Accuracy
15
Point-of-Gaze Estimation Methodologies

• 1. Head-mounted 2. Remote
• 
  (no head-worn components)

16
Remote Point-of-Gaze Estimation

• Point-of-gaze is estimated to a fixed coordinate
  system
• C centre of corneal curvature
• P point-of-gaze

computer screen
IR LEDs
eye camera
17
Moving Reading Card

• How can point-of-gaze be estimated to a
  coordinate system attached to a moving reading
  object?

18
Estimate Motion
t1
t0
R, T
19
Use a Scene Camera and Targets
t1
t0
Scene Camera
20
Calculate Two Homographies
t1
t0
H0
H1
Scene Camera
21
Decompose Homography Matrices
t1
t0
R0, T0
R1, T1
Scene Camera
22
Calculate Motion of 2D Scene Object
t1
t0
R, T
R0, T0
R1, T1
Scene Camera
23
Point-of-Gaze Accuracy
24
What We Need To Do Step 2

• Identify the word being read, in real-time
• Detect when the word being read is an unknown
  word
• Vocalization of the unknown word

25
Dual Route Reading Model
Coltheart, M. et al. (2001)
26
Dual Route Reading Model
Each words graphemes are processed in parallel
27
Dual Route Reading Model
Each words graphemes are individually converted
into phonemes based on mapping rules
28
Detecting Unknown Words

• For unknown words, the lexical route fails and
  the slower non-lexical route is used

Hypothesis we can differentiate between known
and unknown words by the duration of the
processing time
29
Processing Time
30
Setting a Threshold Curve
31
Setting the Threshold

• Threshold curve is a function of word length
• Model processing time for known words (length k)
  as a Gaussian random variable
• ?(µk, sk2)
• Estimate µk, sk2 from a short training set for
  each subject
• Each point on threshold curve is given by
• a is the constrained probability of false alarm

32
Experiment Detecting Unknown Words

• Remote point-of-gaze estimation system
• Reading material presented on computer screen
• Head position stabilized using a chinrest
• Four subjects read from 40 passages of text
• 20 passages aloud and 20 passages silently
• Divided into training set to learn µk, sk2 and
  set detection threshold curves
• Set false alarm probability a 0.10
• Evaluate detection performance

33
Experiment Detecting Unknown Words
34
Experiment Natural Setting Reading Assistance

• Natural reading pose
• Unrestricted head movement
• Reading material is hand-held
• Head-mounted eye-tracker
• Identify viewed word in real-time
• Measure per-word processing time
• Detecting unknown words
• Processing time threshold curves established in
  previous experiment
• Assistance
• Detection of unknown word activates vocalization

35
Experiment Natural Setting Reading Assistance

• Results
• Point-of-gaze mapping method accommodated head
  and reading material movement without reducing
  detection performance

Subject Detection Rate False Alarm Rate
M.E. 0.94 0.10
P.L. 0.95 0.09
36
Conclusions

• Developed methods to map point-of-gaze estimates
  to an object coordinate system attached to a
  moving 2D scene object (e.g. reading card)
• Head-mounted system
• Remote system
• Developed method to detect when a reader
  encounters an unknown word
• Demonstrated principle of operation for an
  automated reading assistance system

37
Future Work

• Implement reading assistant using remote-gaze
  estimation methodology
• Validate efficacy of system as a teaching tool
  for unskilled English readers, in collaboration
  with an audiologist
• Evaluate other forms of assistive intervention
• e.g. translation, definition

38
Questions?