Pupil Detection and Tracking System

1
Pupil Detection and Tracking System
Lior Zimet Sean KaoEE 249 Project Mentors Dr.
Arnon Amir Yoshi Watanabe
2
Outline

• Introduction and Goals
• Design Methodology
• Model of Computation
• Mapping and Implementation
• Verification
• Conclusions

3
Motivation

• Human-computer interfaces are becoming more
  important
• New interfaces may benefit from knowledge of the
  location of the users eyes
• Auto-stereoscopic displays
• Virtual Reality interfaces
• Facial recognition systems
• Eye gaze tracking

4
Goals

• Exercise design process from functional
  specification to implementation and verification
• Develop an embedded system that will find the
  two-dimensional location of a users pupils
• Apply various methodologies we have learned
• Use a heterogeneous collection of various
  components in a real-time environment

5
Background

• Human pupils may be found using two infrared
  light sources
• An on-axis light source will give the red-eye
  effect.

6
Background

• An off-axis light source will give a dark-pupil
  effect
• We can synchronize the two light sources with the
  capturing device

7
Difference of the two Images
8
Design Methodology

• Begin with a definition of the system with
  illustrations
• Make a formal specification using the Unified
  Modeling Language
• Describe the system using a behavioral model
• Explore architectural space
• Map functionality into chosen architecture
• Verify implementation

9
System Definition

• Take in an image illuminated with two different
  light sources
• Find the pupils in the image
• Calculate the position of the pupils
• Output the coordinates

10
UML Diagrams

• UML is used for formally describing a system
• Use-case diagram shows functions of the system
  without implying how it is done
• Class diagram shows what functional blocks are
  used
• Sequence diagrams show how the use-cases are
  executed

11
Use Case Diagram
12
Class Diagram
13
Y-Chart of the Project
Implementation of System
First, well describe the system behavior with an
appropriate model of computation
14
Model of Computation

• Processes large amounts of data in a similar way
• Dataflow model is the most appropriate
• Our system is not control-centric
• But some parts of the system are easier to
  describe using sequential algorithms
• Mixed models of computation

15
Simulink versus Ptolemy (Virgil)

• Ptolemy has the ability to mix models of
  computation and has support for synchronous
  dataflow
• But Virgil does not have a simple way to
  integrate sequential algorithms
• Simulink has extensive support for sequential
  algorithms (Matlab)
• But lacks clearly defined semantics, combination
  of dataflow and discrete-event

16
Simulink Model of Computation

• Use one clock to synchronize the system
• No implicit definitions how many tokens are
  generated or used
• We use counters and synchronous signals to
  determine how many tokens are on edges

17
Simulink Model
Image capture model (source)
Properties of objects on a line
Subtraction and threshold
FIFO
Update properties of known objects in a frame
Connect objects in different lines
Calculate (X,Y) coordinates from those properties
Display the coordinates
18
Verifying the Functionality

• Obtained images from IBM Almaden Research Center
• Run Simulink model on half-scale images and
  verify the behavioral model

19
System Architecture on Y-Chart
Implementation of System
Next, well define the system architecture
20
System Architecture

• Image capture device
• Programmable hardware
• Frame buffer FIFO
• Interface controller
• PC

21
Mapping on Y-Chart
Implementation of System
Map the behavior onto the architecture
22
Mapping and HW/SW Partition

• Frame buffer
• RAM vs FIFO
• Subtraction and thresholding of data
• Hardware offers a fast, simple way
• Software requires high memory bandwidth
• Extracting objects from the video data
• Algorithm is non-trivial to implement in
  hardware, a fully parallel implementation is
  costly
• Software implementation is straightforward for
  sequential algorithms, but requires fast data
  transfers to processor

23
Mapping and HW/SW Partition

• Calculating coordinates
• Requires a hardware divider (expensive)
• Needs lots of data to be passed, calculation is
  cheap in software
• Displaying the XY coordinates
• Hardware requires complicated interface to some
  type of display
• Easy to display on monitor, only requires small
  amounts of data to transfer

24
Communication

• Deal with heterogeneous blocks
• Different types of blocks require different types
  of communication

USB
Register Arrays
FPGABlock
PC
Image capture device
PC
Parallel data stream
I2C
25
Design Choices

• Zoran CMOS Sensor
• Zoran Video IP Evaluation Board
• IBM Almaden BlueEyes LED Board
• Altera APEX20K FPGA
• Averlogic Frame Buffer FIFO
• Cypress USB Controller

26
Design Choices
Properties of objects on a line
Image capture model (source)
Subtraction and threshold
FIFO
CMOS Sensor
Frame buffer
Altera FPGA and Zoran Evaluation board
Connect objects in different lines
Update properties of known objects in a frame
PC
USB controller
Calculate (X,Y) coordinates from those properties
Display the coordinates
27
Implementation from Behavioral Model

• No available tool that can effectively apply
  model of computation to various architectural
  implementations yet
• Xilinx SysGen tool cannot handle data stream
• Real-Time Workshop for Simulink
• Does not necessarily generate efficient code
• Various programs that generate Verilog from C
• May not be synthesizable

28
Implementation

• Verilog (synthesized manually) is used for
  hardware blocks
• C/C (synthesized manually) used for software
  blocks

Simulink
ModelSim
Implementation of System
LeonardoSpectrum and Quartus
29
Verification

• ModelSim used to simulate Verilog
• LeonardoSpectrum used to synthesize Verilog into
  FPGA netlist
• Quartus used to map and place route
• Visual C used to compile and simulate software
  components

30
System Demo
31
Conclusions

• Good design methodology allowed us to exercise
  the complete design process in a short time span
  (time-to-market)
• Separating functionality from implementation can
  solve many errors early in the design process
• Communication and interface-based design is vital
  for system integration
• No available programs to automatically synthesize
  the process all the way down
• Tried to apply EE 249 design methodologies, but
  implemented with traditional techniques

32
Acknowledgements

• Dr. Arnon Amir (IBM Almaden Research Center)
• Zoran Video IP team
• Zoran CMOS Sensor team
• Yoshi Watanabe
• Prof. Sangiovanni-Vincentelli
• Rong Chen