Senior Capstone Project Gaze Tracking System what you see is finally what you get

1
Senior Capstone ProjectGaze Tracking Systemwhat
you see is finally what you get

• Team IRALAR
• Breanna Heidenburg -- Michael Lenisa --
  Daniel Wentzel
• Advisor Dr. Malinowski

DISCLAIMER This presentation was used
specifically for the Bradley University Student
Expo on April 4th, 2008. Other versions may
contain a more comprehensive description of the
project.
2
Presentation Outline

• The Project
• Why is it important
• The Goals
• System breakdown
• Image recognition
• Point transformation
• User Interface
• The Results

3
Brief overview

• What is our project?

4
Brief Overview

• Track a users eye and use the information to
  control a computer cursor

Wherever you look on the display is where the
cursor goes
5
Brief Overview

• Enhances Human Computer Interaction
• Speed of use
• Hands-free use

6
System Development

• 3 Part System
• Image Processing Application
• Calibration and Mapping system
• GUI designed for gaze-based interaction
• Systems developed concurrently and independently
• Separate Applications at run-time

7
System Development

• Hardware and Image Processing Application

8
System Development Hardware
User Interface

• Hardware
• Camera
• QuickCam Pro for Notebooks
• Visible Spectrum Camera
• Polarizer
• Tiffen 25mm polarizing filter
• Removes glare from eye reflections
• Lighting
• Diffuse LEDs
• Slightly distracting to the user, but necessary
  to provide light for the camera

9
System Development User Interface

• LitEye LE-500
• High resolution (SVGA)
• Color Display
• Translucent or opaque operation
• Stationary relative to users eye

10
System Development Software Image Processing

• Real time pupil tracking system
• Developed in C using OpenCV image processing
  libraries
• Traditional image processing and blob tracking
• Capabilities
• Locate and determine center of pupil in image
• Low light and high reflection environments
• All eye colors
• Data logging and static test modes
• Packaged into self contained Windows installer
  for easy deployment onto any computer

11
System Development Software Image Processing

• The Process
• Capture Image
• Extract Red Channel
• Smooth image
• Apply binary threshold
• Locate blobs
• Reject false positives
• Determine center of pupil blob
• Adapt threshold
• Repeat

12
System Development Software Image Processing

• Summary
• The Good
• Dynamically adapts to changing lighting
  conditions and eye types
• Maintains performance in low-light and specularly
  noisy conditions
• The Bad
• Still relies on Logitech camera drivers
• Extreme reflections still cause problems

13
System DevelopmentImage Processing in Action
Raw Capture
Pre-Processed Image
Completed Recognition
14
System Development Image Processing

• Examples of performance in poor conditions
• Low Light Difficult False Positive

15
System Development

• Calibration and point mapping

16
System Development Eye Mapping

• System for mapping the location of the center of
  the pupil to a pixel on a computer screen
• Reasons
• Geometry
• The eye is not flat but a screen is not
• User Customization
• All eyes are different
• Everyone wears the HMD differently
• User Training
• Calibration system also acts as a quick tutorial

17
System Development Point Transformation

• 3 dimensional best fit plane
• Currently using a 4th degree best fit
• Xpix A1 XeyeB1 YeyeC1
• Ypix A2 XeyeB2 YeyeC2
• Calibration sub-system determines these
  coefficients

18
System Development Calibration

• How do we solve the problem?
• Multiple Variable Linear Regression Least
  Squares
• Y B0 B1x1 Bkxk
• Uses matrix algebra to obtain a coefficient
  matrix
• B (XX)-1XY

19
System Development Calibration
B (XX)-1XY

• Results
• pixelX 224.9 4.8eyeX 2.9eyeY
• pixelY 1612.5 2.0eyeX 5.6eyeY

20
System Development Clicking

• How do we click?
• Monitor eye movements
• Identify pauses
• When eye position is within a small percentage
  for a certain amount of time
• Currently set at 5 frames (200 mS)
• Generally, it takes 230 mS for a hand to click a
  mouse.

21
System Development

• Communications and Custom GUI interface

22
System Development Communication

• Why is it important?
• Allows multiple processes to communicate
• Allows relay of time sensitive information
• UDP vs. TCP
• UDP -gt Connectionless
• TCP -gt Connection oriented
• Multi-threading
• Necessary for running multiple pieces of code in
  a single process

23
System Development Communication
Multi-threading issues present
24
System Development Communication

• Multi-threading
• Issues with public variable usage
• Solutions we are using
• Critical Section
• Raises thread priority (does not allow for
  interrupts)

Receive request for data (over UDP)
Raise Thread priority
Lower Thread Priority
Reply to request
Read Variable
25
System Development User Interface

• Custom GUI for Gaze Tracking Applications
• Why?
• Gaze tracking accuracy limited by inherent
  properties of human vision system
• Traditional GUI too small and intrusive for use
  with transparent HMD
• Demonstrate applications of gaze tracking

26
System Development User Interface

• Achieved via a combination of Hardware and
  Software
• Software
• WPF C
• XAML
• (eXtensible Application Markup Language)
• Very similar to XML
• Uses code-behind in a similar style to
  javascript
• code-behind is in C

27
System Development software User
Interface

• Multiple applications have been created within
  the interface
• Multiple screens for functionality testing
• even games
• Additional ability to minimize interface and
  interact with external applications is present

28
System Development User Interface
Main Screen
Calibration Screen
29
Results

• What did our test results show?

30
ResultsCalibration
31
ResultsCalibration
32
ResultsTiming
33
ResultsSchedule
34
Thanks

• Who helped us out?

35
Thanks

• Northrop Grumman
• LitEye
• Dr. Malinowski and the EE faculty
• Mr. Mattus Mr. Schmidt

36
Questions