Noninvasive Techniques for Human Fatigue Monitoring

1
Non-invasive Techniques for Human Fatigue
Monitoring
Qiang Ji Dept. of Electrical, Computer, and
Systems Engineering Rensselaer Polytechnic
Institute qji_at_ecse.rpi.edu http//www.ecse.rpi.edu
/homepages/qji Funded by AFOSR and Honda
2
(No Transcript)
3
Visual Behaviors

• Visual behaviors that typically reflect a
• person's level of fatigue include
• Eyelid movement
• Head movement
• Gaze
• Facial expressions

4
Eye Detection and Tracking
5
Eye Detection
6
Eye Tracking

• Develop an eye tracking technique based on
  combining mean-shift and Kalman filtering
  tracking.
• It can robustly track eyes under different face
  orientations, illuminations, and large head
  movements.

7
(No Transcript)
8
Eyelid Movements Characterization

• Eyelid movement parameters
• Percentage of Eye Closure (PERCLOS)
• Average Eye Closure/Open Speed (AECS)

9
(No Transcript)
10
Gaze (Pupil Movements)

• Real time gaze tracking
• Develop a real time gaze tracking technqiue.
• No calibration is needed and allows natural head
  movements !.

11
Gaze Estimation

• Gaze is determined by
• Pupil location (local gaze)
• Local gaze is characterized by relative positions
  between glint and pupil.
• Head orientation (global gaze)
• Head orientation is estimated by pupil shape,
  pupil position, pupil orientation, and pupil size.

12
(No Transcript)
13
Gaze Parameters

• Gaze spatial distribution over time
• PERSAC-percentage of saccade eye movement over
  time

14
Gaze distribution over time while alert
15
Gaze distribution over time while fatigue
16
Gaze distribution over time for inattentive
driving
17
Plot of PERSAC parameter over 30 seconds.
18
(No Transcript)
19
Head Movement

• Real time head pose tracking
• Perform 3D face pose estimation from a single
  uncalibrated camera.
• Head movement parameters
• Head tilt frequency over time (TiltFreq)

20
The flowchart of face pose tracking
21
Examples Face Model Acquisition
22
(No Transcript)
23
Head pitches (tilts) monitoring over time
(seconds)
24
(No Transcript)
25
Facial Expressions

• Tracking facial features
• Recognize certain facial expressions related to
  fatigue like yawning and compute its frequency
  (YawnFreq)
• Building a database of fatigue expressions for
  training

26
(No Transcript)
27
The plot of the openness of the mouth over time
28
Facial expression demo
29
Fatigue Modeling

• Observations of fatigue is uncertain, incomplete,
  dynamic, and from different from perspectives
• Fatigue represents the affective state of an
  individual, is not observable, and can only be
  inferred.

30
Overview of Our Approach

• Propose a probabilistic framework based on the
  Dynamic Bayesian Networks (DBN) to
• systematically represent and integrate various
  sources of information related to fatigue over
  time.
• infer and predict fatigue from the available
  observations and the relevant contextual
  information.

31
Bayesian Networks Construction

• A DBN model consists of target hypothesis
  variables (hidden nodes) and information
  variables (information nodes).
• Fatigue is the target hypothesis variable that we
  intend to infer.
• Other contextual factors and visual cues are the
  information nodes.

32
Causes for Fatigue

• Major factors to cause fatigue include
• Sleep quality.
• Circadian rhythm (time of day).
• Physical conditions.
• Working environment.

33
Bayesian Fatigue Model
34
Dynamic Fatigue Modeling
35
Bayesian Fatigue Model Demo
36
Interface with Vision Module

• An interface has been developed to connect the
  output of the computer vision system with the
  information fusion engine.
• The interface instantiates the evidences of the
  fatigue network, which then performs fatigue
  inference and displays the fatigue index in real
  time.

37
(No Transcript)
38
Conclusions

• Developed non-intrusive real-time computer vision
  techniques to extract multiple fatigue parameters
  related to eyelid movements, gaze, head movement,
  and facial expressions.
• Develop a probabilistic framework based on the
  Dynamic Bayesian networks to model and integrate
  contextual and visual cues information for
  fatigue detection over time.

39
Effective Fatigue Monitoring

• The technology must be non-intrusive and in real
  time.
• It should simultaneously extract multiple
  parameters and systematically combine them over
  time in order to obtain a robust and consistent
  fatigue characterization.
• A fatigue model is needed that can represent
  uncertain and dynamic knowledge associated with
  fatigue and integrate them over time to infer
  and predict human fatigue.