A Framework for a Video Analysis Tool for Suspicious Event Detection

1
Data Mining for Surveillance Applications Suspici
ous Event Detection
Dr. Bhavani Thuraisingham
November 2007

2
Problems Addressed

• Huge amounts of video data available in the
  security domain
• Analysis is being done off-line usually using
  Human Eyes
• Need for tools to aid human analyst ( pointing
  out areas in video where unusual activity occurs)
• Consider corporate security for a fenced section
  of sensitive property
• The guard suspects there may have been a breach
  of the perimeter fence at some point during the
  last 48 hours
• They must
• Manually review 48 hours of tape
• Consider multiple cameras and camera angles
• Distinguish between normal personnel and
  intruders

3
Example

• Using our proposed system
• Greatly Increase video analysis efficiency

4
The Semantic Gap

• The disconnect between the low-level features a
  machine sees when a video is input into it and
  the high-level semantic concepts (or events) a
  human being sees when looking at a video clip
• Low-Level features color, texture, shape
• High-level semantic concepts presentation,
  newscast, boxing match

5
Our Approach

• Event Representation
• Estimate distribution of pixel intensity change
• Event Comparison
• Contrast the event representation of different
  video sequences to determine if they contain
  similar semantic event content.
• Event Detection
• Using manually labeled training video sequences
  to classify unlabeled video sequences

6
Event Representation

• Measures the quantity and type of changes
  occurring within a scene
• A video event is represented as a set of x, y and
  t intensity gradient histograms over several
  temporal scales.
• Histograms are normalized and smoothed

7
Event Comparison

• Determine if the two video sequences contain
  similar high-level semantic concepts (events).
• Produces a number that indicates how close the
  two compared events are to one another.
• The lower this number is the closer the two
  events are.

8
Event Detection

• A robust event detection system should be able to
• Recognize an event with reduced sensitivity to
  actor (e.g. clothing or skin tone) or background
  lighting variation.
• Segment an unlabeled video containing multiple
  events into event specific segments

9
Labeled Video Events

• These events are manually labeled and used to
  classify unknown events
• Walking1 Running1 Waving2

10
Labeled Video Events
11
Experiment 1

• Problem Recognize and classify events
  irrespective of direction (right-to-left,
  left-to-right) and with reduced sensitivity to
  spatial variations (Clothing)
• Disguised Events- Events similar to testing
  data except subject is dressed differently
• Compare Classification to Truth (Manual
  Labeling)

12
Experiment 1
Disguised Walking 1

• Classification Walking

13
Experiment 1
Disguised Walking 2

• Classification Walking

14
Experiment 1
Disguised Running 1

• Classification Running

15
Experiment 1
Disguised Running 2

• Classification Running

16
Classifying Disguised Events
Disguised Running 3

• Classification Running

17
Classifying Disguised Events
Disguised Waving 1

• Classification Waving

18
Classifying Disguised Events
Disguised Waving 2

• Classification Waving

19
Classifying Disguised Events
20
Experiment 1

• This method yielded 100 Precision (i.e. all
  disguised events were classified correctly).
• Not necessarily representative of the general
  event detection problem.
• Future evaluation with more event types, more
  varied data and a larger set of training and
  testing data is needed

21
XML Video Annotation

• Using the event detection scheme we generate a
  video description document detailing the event
  composition of a specific video sequence
• This XML document annotation may be replaced by a
  more robust computer-understandable format (e.g.
  the VEML video event ontology language).

22
Video Analysis Tool

• Takes annotation document as input and organizes
  the corresponding video segment accordingly.
• Functions as an aid to a surveillance analyst
  searching for Suspicious events within a stream
  of video data.
• Activity of interest may be defined dynamically
  by the analyst during the running of the utility
  and flagged for analysis.

23
Summary and Directions

• We have proposed an event representation,
  comparison and detection scheme.
• Working toward bridging the semantic gap and
  enabling more efficient video analysis
• More rigorous experimental testing of concepts
• Refine event classification through use of
  multiple machine learning algorithm (e.g. neural
  networks, decision trees, etc). Experimentally
  determine optimal algorithm.
• Develop a model allowing definition of
  simultaneous events within the same video
  sequence
• Define an access control model that will allow
  access to surveillance video data to be
  restricted based on semantic content of video
  objects
• Biometrics applications
• Privacy preserving surveillance

24
Access Control and Biometrics

• Access Control
• Control access based on content, association,
  time etc.
• Biometrics
• Restrict access based on semantic content of
  video rather then low-level features
• Behavioral type access instead of fingerprint
• Used in combination with other biometric methods

25
Privacy Preserving Surveillance -
Introduction

• A recent survey at Times Square found 500 visible
  surveillance cameras in the area and a total of
  2500 in New York City.
• What this essentially means is that, we have
  scores of surveillance video to be inspected
  manually by security personnel
• We need to carry out surveillance but at the same
  time ensure the privacy of individuals who are
  good citizens

26
System Use
Raw video surveillance data
Face Detection and Face Derecognizing system
Suspicious people found
Faces of trusted people derecognized to preserve
privacy
Suspicious events found
Comprehensive security report listing suspicious
events and people detected
Suspicious Event Detection System
Manual Inspection of video data
Report of security personnel
27
System Architecture
Input Video
Finding location of the face in the image
Breakdown input video into sequence of images
Perform Segmentation
Compare face to trusted and untrusted database
Raise an alarm that a potential intruder was
detected
Potential intruder found
Trusted face found
Derecognize the face in the image
28
Acknowledgements

• Prof. Latifur Khan
• Gal Lavee (Surveillance and access control)
• Ryan Layfield (Consultant to project)
• Sai Chaitanya (Privacy)
• Parveen Pallabi (Biometrics)