Computing Motion from Images

1
Computing Motion from Images

• Chapter 9 of SS plus otherwork.

2
General topics

• Low level change detection
• Region tracking or matching over time
• Interpretation of motion
• MPEG compression
• Interpretation of scene changes in video
• Understanding human activites

3
Motion important to human vision
4
Whats moving different cases
5
Image subtraction

• Simple method to remove unchanging background
  from moving regions.

6
Change detection for surveillance
7
Change detection by image subtraction
8
What to do with regions of change?

• Discard small regions
• Discard regions of non interesting features
• Keep track of regions with interesting features
• Track in future frames from motion plus component
  features

9
Some effects of camera motion that can cause
problems
10
Motion field
11
FOE and FOC
Will return to use the FOE or FOC or detection of
panning to determine what the camera is doing in
video tapes.
12
Gaming using a camera to recognize the players
motion

• Decathlete game

13
Decathlete game
Cheap camera replaces usual mouse for input
Running speed and jumping of the avatar is
controlled by detected motion of the players
hands.
14
Motion detection input device
Jumping (hands)
Running (hands)
15
Motion analysis controls hurdling event (console)

• Top left shows video frame of player
• Middle left shows motion vectors from multiple
  frames
• Center shows jumping patterns

16
Related work

• Motion sensed by crude cameras
• Person dances/gestures in space
• System maps movement into music
• Creative environment?
• Good exercise room?

17
Computing motion vectors from corresponding
points

• High energy neighborhoods are used to define
  points for matching

18
Match points between frames
Such large motions are unusual. Most systems
track small motions.
19
Requirements for interest points
Match small neighborhood to small neighborhood.
The previous scene contains several highly
textured neighborhoods.
20
Interest minimum directional variance
Used by Hans Moravec in his robot stereo vision
system. Interest points were used for stereo
matching.
21
Detecting interest points in I1
22
Match points from I1 in I2
23
Search for best match of point P1 in nearby
window of I2
For both motion and stereo, we have some
constraints on where to search for a matching
interest point.
24
Motion vectors clustered to show 3 coherent
regions
All motion vectors are clustered into 3 groups of
similar vectors showing motion of 3 independent
objects. (Dina Eldin)
Motion coherence points of same object tend to
move in the same way
25
Two frames of aerial imagery
Video frame N and N1 shows slight movement most
pixels are same, just in different locations.
26
Can code frame Nd with displacments relative to
frame N

• for each 16 x 16 block in the 2nd image
• find a closely matching block in the 1st image
• replace the 16x16 intensities by the location in
  the 1st image (dX, dY)
• 256 bytes replaced by 2 bytes!
• (If blocks differ too much, encode the
  differences to be added.)

27
Frame approximation
Left is original video frame N1. Right is set of
best image blocks taken from frame N. (Work of
Dina Eldin)
28
Best matching blocks between video frames N1 to
N (motion vectors)
The bulk of the vectors show the true motion of
the airplane taking the pictures. The long
vectors are incorrect motion vectors, but they do
work well for compression of image I2!
Best matches from 2nd to first image shown as
vectors overlaid on the 2nd image. (Work by Dina
Eldin.)
29
Motion coherence provides redundancy for
compression

• MPEG motion compensation represents motion of
  16x16 pixels blocks, NOT objects

30
MPEG represents blocks that move by the motion
vector
31
MPEG has I, P, and B frames
32
Computing Image Flow
33
(No Transcript)
34
Assumptions
35
IMAGE FLOW EQUATION 1 of 2
36
Image flow equation 2 of 2
37
Aperture problem
38
Solving flow by propagation of constraints
39
Info at corner constrains the flow along both
edges
Solve constraints using contraint propagation or
differential equation with boundary conditions.
40
Tracking several objects

• Use assumptions of physics to compute multiple
  smooth paths.
• (work of Sethi and R. Jain)

41
(No Transcript)
42
Tracking in images over time
43
General constraints from physics
44
Other possible constraints

• Background statistics stable
• Object color/texture/shape might change slowly
  over frames
• Might have knowledge of objects under survielance
  
• Objects appear/disappear at boundary of the frame

45
(No Transcript)
46
Sethi-Jain algorithm
47
(No Transcript)
48
Total smoothness of m paths
49
Greedy exchange algorithm
50
Example data structure
Total smoothness for trajectories of Figure 9.14
51
Example of domain specific tracking (Vera Bakic)
Tracking eyes and nose of PC user. System
presents menu (top). User moves face to position
cursor to a particular box (choice). System
tracks face movement and moves cursor
accordingly user gets into feedback-control loop.
52
Segmentation of videos/movies

• Segment into scenes, shots, specific actions, etc.

53
Types of changes in videos
54
How do we compute the scene change?
Anchor person scene at left
Scene break
Street scene for news story
From Zhang et al 1993
55
Histograms of frames across the scene change
Histograms at left are from anchor person frames,
while histogram at bottom right is from the
street frame.
56
Heuristics for ignoring zooms
57
American sign language example
58
Example from Yang and Ahuja
59
(No Transcript)
60
(No Transcript)