Center for Intelligent Systems Research

1
NDIA 3rd Annual Intelligent Vehicle Systems
Symposium Driving Simulator ExperimentDetectin
g Driver Fatigue by Monitoring Eye and Steering
Activity
Dr. Azim Eskandarian, Riaz Sayed (GWU)

• Center for Intelligent Systems Research
• GW Transportation Research Institute
• The George Washington University, Virginia
  Campus, 20101 Academic Way, Ashburn, VA 20147

2
Research Objective

• Conduct Simulator Experiment and Analyze the
  Data, to search for a system for automatic
  detection of drowsiness based on drivers
  performance

3
Significance of the Problem

• Drowsiness/Fatigue Related Accident Data
• NHTSA Estimates 100,000 drowsiness/fatigue
  related Crashes Annually
• FARS indicates an annual average of 1,544
  fatalities
• Fatigue has been estimated to be involved in
  10-40 of crashes on highways (rural Interstate)
• 15 of single vehicle fatal truck crashes
• Fatigue is the most frequent contributor to
  crashes in which a truck driver was fatally
  injured

4
Significance of the Problem

• A drowsy/sleepy driver is unable to determine
  when he/she will have an uncontrolled sleep onset
• Fall asleep crashes are very serious in terms of
  injury severity
• An accident involving driver drowsiness has a
  high fatality rate because the perception,
  recognition, and vehicle control abilities
  reduces sharply while falling asleep
• Driver drowsiness detection technologies can
  reduce the risk of a catastrophic accident by
  warning the driver of his/her drowsiness

5
Driver Drowsiness Detection Techniques

• 1. Sensing of driver physical and physiological
  phenomenon
• Analyzing changes in brain wave or EEG
• Analyzing changes in eye activity and Facial
  expressions
• Good detection accuracy is achieved by these
  techniques
• Disadvantages
• Electrodes have to be attached to the body of the
  driver for sensing the signals
• Non-contact type sensing is also highly dependant
  on environmental conditions

6
Driver Drowsiness Detection Techniques

• 2. Analyzing changes in performance output of
  the vehicle hardware
• Steering, speed, acceleration, lateral position,
  and braking etc.
• Advantages
• No wires, cameras, monitors or other devices are
  to be attached or aimed at the driver
• Due to the non-obtrusive nature of these methods
  they are more practically applicable

7
Approach for Drowsiness Detection and Driver
Warning
8
Experiment

• Conducted in the Vehicle Simulator Lab of the
  CISR. GWU VA Campus, Ashburn VA.
• Twelve subjects between the ages of 23 and 43
• Test Scenario consisted of a continuous rural
  Interstate highway, with traffic in both
  directions Speed limit of 55 mph.
• Morning session 8 10 am
• Night session 1 3 am

9
CISR Driving Simulator
10
Eye Tracking Equipment
11
Sample Data From Simulator

• RUN ZONETIME SPEEDLIM CRASHB CRASHV
  LANEX BRAKEFOR BRAKETAP
• 1 0 35 0 0 0 0 0
• 1 2.1 35 0 0 0 0 0
• 1 4.2 35 0 0 0 0 0
• 1 6.2 35 0 0 0 0 0
• 1 8.3 35 0 0 0 0 0
• STEERPOS STEERVAR LATPLACE LATPLVAR
  SPEED SPEEDVAR SPEEDDEV
• -0.1 0 -0.09 0 53.71
  0 -4.65
• 0.2 0 -0.22 0 53.71
  0 -4.65
• 0.4 0 -0.31 0 53.71
  0 -4.65
• 0 0 -0.35 0
  53.71 0 -4.65

12
Lateral Position of Vehicle
13
Power Spectrum Density for Vehicle Lateral
Position
14
Steering Anglefilter correction for curves
15
Hypothesis

• The hypothesized relationship between driver
  state of alertness and steering wheel position is
  that under an alert state, drivers make small
  amplitude movements of the steering wheel,
  corresponding to small adjustments in vehicle
  trajectory, but under a drowsy state, these
  movements become less precise and larger in
  amplitude resulting in sharp changes in
  trajectory (Planque et al. 1991).

16
A Hybrid Artificial Neural Network Architecture
Unsupervised Layer Clustering Competitive
Algorithm
Supervised Layer Classification Feedforward
Algorithm
Wj1
2
8 X 8
17
Hybrid Artificial Neural Network Architecture
18
ANN Training for Unsupervised Competitive Layer

• 1. Initialize the weight vector randomly for
  each neuron.
• 2. Present the input vector X(n) .
• 3. Compute the winning neuron using the
  Euclidean distance as a metric.
• Where Wi w1, w2, . w8T is the weight
  vector of neuron i.
• bi is the bias to stop the formation of dead
  neurons.

19
ANN Training Competitive Layer Continued

• N number of time a neuron wins in competitive
  layer
• ? and ? are learning constants and o(n) is the
  outcome of the present competition (1 if neuron
  wins else 0).
• Ci initially set to small random value
• 4. Update the weight vector of the winning neuron
  Wi only.
• 5. Continue with step (2) two until change in the
  weight vectors reaches a minimum value.

20
ANN Training Competitive Layer Continued

• The competitive algorithm moves the weight
  vectors of all the neurons closer to the center
  of the clusters.
• Each neuron (or set of neurons) of the
  competitive layer represents a cluster.
• The Output of the neuron is 1 if it wins the
  competition and 0 if it losses.
• The Output of the Competitive layer is an
• n-dimensional binary vector T(n) t1,
  t2, .., tnT .

21
ANN Training for supervised feed forward layer

• Step 1 Initialize the synaptic weights and the
  thresholds to small random numbers.
• Step 2 Present the network with an epoch of
  training exemplars
• Step 3 Apply Input vector X(n) to the input
  layer and the desired response d(n) to the output
  layer of neurons. The output of each neuron is
  calculated as

22
ANN Training Continued
23
ANN Training Continued

• N No. of training sets in one epoch
• ? Learning rate parameter
• ? Momentum constant
• Step 5 Iterate the computation by presenting
  new epochs of training examples until the mean
  square error (MSE) computed over entire epoch
  achieve a minimum value. MSE is given by

24
ANN Training Parameters

• Hybrid architecture using an unsupervised
  clustering algorithm and a classifier (Back
  propagation learning algorithm in batch mode)
• Tanhyperbolic activation function, with output
  range from 1 to 1
• Variable learning rate and momentum were used
• Cross validation during training

25
Input Discretization of Steering Angle
Algorithm to select r (ranges) for each driver to
compensate performance variability between drivers

Discretized steering angle for one driver
26
Accounting for Individual Driver Behaviors

• Some drivers are more sensitive to vehicle
  lateral position and make very accurate
  corrections to the steering for lane keeping
  while other are less sensitive and make less
  accurate corrections.
• The result is a low amplitude signal (steering
  angle) for more sensitive drivers and
  relatively high amplitude signal for less
  sensitive drivers.
• Larger values for Pk will make the descritization
  ranges wider to accommodate large amplitude while
  small values will make them shorter for small
  amplitudes.
• Therefore, same ANN (8-dimensional
  descritization) can be used

27
Input Discretization of Eye closures

• Eye closure data is recorded at 60 Hz
• Ci No. of zeros in 1 second of data
• Ci is further discretized according to the
  following scheme

28
Input Discretization of Eye closures
Algorithm to select r (ranges) for each driver to
compensate eye closure variability between
drivers P values are representative of
variability of eye closures (blinking) for each
driver
Sample of a few seconds of Discretized Eye
closures for one driver
29
Input Vector

• The two vectors are combined to form a 12 dim
  vector J(T)
• Vector J(T) is summed over 15 sec time interval
  to get the input vector X(n)

30
Input and Desired Output Vector
Each row represents the sum of discretized input
over a selected time interval, e.g., 15 sec.
31
ANN Performance During Training
32
ANN Test Data

• Driving data from 12 subjects available
• 1 subject night session not recorded due to
  equipment error.
• 1 subject morning data not available, software
  error.
• Remaining 10 were used for training ANN and
  testing results,
• NOTE training data and testing of the ANN were
  not the same, Testing data selected randomly from
  the sets not used in the training

33
Results
Actual Totals
Network Output
Actual Totals
Network Output
Wake
Sleep
Wake
Sleep
Wake
193
179
14
Wake
193
179
14
False Alarm
False Alarm
Sleep 207
16
191
Sleep 207
16
191
Mis
-
classified
Mis
-
classified
Crash Prediction
All crashes that occurred due to
driver falling asleep during the experiment were
predicted before the crash occurred.
34
Morning and Night session results
35
Morning and Night session results
36
Morning and Night session results
37
Morning and Night session results
38
Morning and Night session results
39
Time Before Crash When the ANN Generated a first
Warning
40
Conclusions

• A non-intrusive method of drowsiness detection
  using steering data is possible
• A method using ANN is developed and successfully
  predicts drowsiness (91 Success Rate)
• Method is solely based on drivers (Vehicle)
  steering performance
• Same method may be applied to detection of
  fatigue or other related driver performance
• Further refining and validation of the algorithm
  is recommended
• Capturing individual drivers steering while
  drowsy requires additional research

41
Recommended Additional Research

• Additional Simulator Experiments
• Validate the Developed Algorithm
• Additional Road Conditions
• More Diversified Group of Drivers
• Road (Experimental) Tests in an Instrumented
  Vehicle
• Further Refining the Algorithm Based on the Road
  Test Data
• Testing of Other Fatigue Related Scenarios
• Research on Warning Systems Integrated With This
  Detection System