The Recognition of Human Movement Using Temporal Templates

1
The Recognition of Human Movement Using Temporal
Templates

• Liat Koren

2
Lecture subjects

• Introduction
• Prior work
• The Temporal Templates
• Usage example

3
Introduction

• Computer vision trends
• Less image or camera motion
• More on labeling of action
• Reasons
• More computational power
• Wireless application
• Interactive environments

4
Introduction cont.

• Recent efforts are in Three Dimensional object
  reconstruction
• Assuming it will have to be used in the
  recognition of human motion.
• This article claims otherwise
• View-based approach
• Direct recognition

5
Motivating Example
6
Motivating Example
7
Motivating Example

• Static pictures
• Hard to recognize.
• Sequence on motion
• Human can recognize without three dimensional
  reconstruction.
• Conclusion
• It is possible to recognize movement using only
  the motion itself.

8
3D Based recognition

• Process
• Recover the pose of the person at each time
  instant using a 3D model.
• The models projected image should be as close as
  possible to the object(e.g. edges of body in the
  image)
• Drawbacks
• Complicated process
• Human interference is usually required
• Special imaging environment

9
2D Based recognition

• Action is a sequence of static poses of object.
• Requires
• Normalization
• Removal of background

10
Wilson and Bobiks approach

• Actions are usually hand gestures
• Representation
• Actual image
• Grayscale
• No background
• Benefits
• Hand appearance is fairly similar over a wide
  range of people
• Problems
• Actions that include the appearance of the whole
  body are not visually consistent across different
  people.

11
Yamatos et al. approach

• Representation
• No background
• Black and white silhouettes
• Matching
• Vector quantize
• Usage of a mathematical method
• Benefits
• Help handling the variability between people
• Problems
• Disappearance of movement inside the silhouette

12
Summery of prior work

• Action is a sequence of static poses.
• Requires individual features or properties that
  can be extracted and tracked from each frame.
• Recognition of movement from a sequence of images
  is a complicated task.
• Usually requires previous recognition and
  segmentation of the person.

13
Motion based recognition

• Attempt to characterize the motion itself without
  reference to the underlying static poses of the
  body.
• Possible approaches
• Blob like representation
• Tracking of predefined regions (e.g., legs, head,
  mouth) using motion.
• Face expression patches
• Whole body patches
• Measure typical patterns of muscle activation

14
Terms

• Movement
• where motion has occurred in image sequence.
• MEI Motion Energy Image
• how the motion is moving.
• MHI Motion History Image

Temporal Templates
15
Temporal Templates

• Representation of movement
• View specific
• Movement is motion in time
• Vector image that can be matched against stored
  representations of movements.
• Assumptions
• Background is static
• Camera movements can be removed
• Motion of irrelevant objects can be eliminated

16
Motion-Energy Images
where did the movement occurred .
17
Motion-Energy Images

• Notice that
• If t is very big, all the differences are
  accumulated
• ? has a vast influence on the temporal
  representation of a movement.

18
Motion-Energy Images

• Smooth change in the viewing angle causes a
  smooth change in the viewed image, thus coarse
  sampling of the viewing circle is enough (30)

19
Motion-History Images

• Intensity of a pixel represents the temporal
  history in that pixel.
• Newer movement is brighter.

20
Motion-History Images
One may wonder, why not use only MHI ?Answers
will be given later

• A time-window of size t is used movement
  older than t is ignored.
• The results of the article uses a simple
  replacement and decay operator

Notice that MEI can be calculated out of MHI by
painting in white any non-black pixel
21
MEI and MHI in a nutshell

• MEI and MHI are two vector images designed to
  encode a variety of motion properties.
• Benefits in this representation is that the
  calculation is recursive, thus only up-to-date
  information need to be stored, making the
  computation both fast and space efficient.

22
Matching Temporal Templates

• Collect training examples of each movement from a
  variety of viewing angles.
• Compute statistical representation of the MHI/MEI
  images (Hu moments)
• Given an input movement
• Calculate a statistical representation
• Use mahalanobis distance to find a stored
  movement, that is the nearest to the input.

23
Mahalanobis Distance Example
24
Reasoning for the algorithm

• Mahanobis distance provides
• Good matching as shown in the results of the
  article.
• Simple calculation which makes real-time
  applications feasible.
• Hu moments allow representation of images, that
  is invariant to scale or translation.
• One problem with Hu moments is that Hu moments
  are difficult to reason about intuitively (the
  authors)

25
Testing the system
18 exercises performed by experienced
aerobic instructor. MEIs are on the bottom rows.
26
Why both MHI and MEI ?
Because MHI and MEI perceive two different
characteristics of the movement (the
where and the how) they look different ,and
thus, both essential.
27
First experiment

• Input 30 left of the subject
• Match against all seven views of all 18 moves
• 12 out of 18 are correctly recognized

28
Analyze the results of 1st exp.
false
correct
input
Move 13 in 30
Move 6 in 0
The correct match
29
Combining multiple views

• Two cameras with orthogonal views
• Minimize the sum of the mahalanobis distance
  between the two input templates and two stored
  views of movement that have 90 between them.
• Hidden assumption we know the angular
  relationship between the cameras.

30
Second Experiment

• Input with two cameras
• 30 left of the subject
• 60 right of the subject
• Match against all seven views of all 18 moves
• 15 out of 18 are correctly recognized

31
Analyze the results of 2nd exp.
false
correct
input
The correct match
Move 16
Move 15
32
Segmentation and Recognition

• Problem speed of performance is different among
  different people.
• Solution Segmentation
• When training the system, calculate tmax and tmin
  for each movement.
• Use algorithm to match over a wide range of t.

33
Problems

• Problems with current system
• One person partially occludes another
• Solution Use several cameras
• More than one person appears in the view point
• Solution use a tracking bounding box

34
More Problems

• Motion of part of the body is not specified
  during a movement
• Possible solutions
• Automatically mask away regions of this type of
  motion
• Always include them
• Camera motion
• Rather easy to eliminate since camera motion is
  limited.
• Person is performing the movement while locomotion

35
The KidsRoom An Application

• room is aware of the children (at most 4)
• The room takes the children to a story.
• The rooms reaction is influenced by the actions
  of the children.
• Current story adventurous tour to monster land
• In the last scene the monsters teach the children
  to dance.
• Then, the monsters follow the children if they
  perform movements they know
• The narration coerces the children to room
  locations where occlusions is not a problem

36
The End