Detecting Motion Gradients using OpenCV

1
Detecting Motion Gradients using OpenCV

• Part I Foreground object selection and game
  design.
• Eric Shafer

2
Introduction

• This Project was done in conjunction with Anthony
  Do. His materials are presented separately.
• The goal of this project was to determine the
  global motion gradient of a foreground object and
  use that information as the input to a pong
  type game.

3
Definitions

• Motion Gradient- the displacement of a pixel
  between two or more images.
• Global Motion Gradient- an average (possibly
  weighted) of the motion gradients of a specified
  set of pixels.
• OpenCV- a open source computer vision library by
  Intel and optimized for the Intel processors.

4
System Input and Output

• Input Images at a rate of 30 per second from a
  live feed camera.
• Output A similar sequence of images with only
  the foreground object indicated and possibly the
  pong game overlaid on the image.

5
The Algorithm

• Train the background
• Extract the largest foreground object
• Calculate the global motion gradient of the
  foreground object over a period of frames
• Use the gradient as input to a game

6
Train a Background

• The background for this system is arbitrary.
• CONSTRAINT The background must be constant.
• To account for any background combination, we
  train on the background before adding any
  foreground objects.

7
Train a Background

• The simplest training would be a single image.
  Any deviations from that image in successive
  images would be considered a possible foreground
  object pixel.
• Noise and changing illumination conditions will
  adversely effect this simple approach.

8
Train a Background

• An average of the pixel values could be used to
  improve the system. But a simple average weighs
  noise equally with accurate pixel information.

9
Train a Background

• Solution Track an average and a standard
  deviation for the values of the background. This
  is done over multiple frames.
• Any new pixel value can be compared to the
  background image. If the new value is more than
  a set number of standard deviations from the
  background it is likely a foreground object
  pixel.

10
Extract the Foreground Object

• Comparing each new image to the background we can
  create a map of all the pixels that are possibly
  foreground ones.
• This map sets the values of all possible
  foreground pixels to 255, all other pixels are
  set to 0.

11
Extract the Foreground Object

• By examining each pixel in the image we can find
  connected regions.
• Starting at the beginning of the image we scan
  until we find a pixel with value 255 and change
  the value to 100.
• We then scan all the 8-neighbors of that pixel.
  Any neighbor which has a value of 255 we change
  to 100.

12
Extract the Foreground Object

• We then search the neighbors of the neighbors and
  change all 255s to 100s. This is continued
  until all neighbors have either a 100 or 0 value.
• This process is called region-growing
• The OpenCV command which performs this operation
  is cvFloodFill

13
Extract the Foreground Object

• Once the floodfill is complete we then have a
  connected region of possible foreground pixels.
• If this is the largest region we have found so
  far, we mark it and turn all the pixels of the
  old largest region to 0 using cvFloodFill.
• If this new region is smaller than one we have
  found then we zero its pixels out again using
  cvFloodFill.

14
Extract the Foreground Object

• The end of this process yields an image with only
  0s and 100s for pixel values.
• The 100s correspond to the largest connected
  region which is now designated as the foreground
  object.

15
Extract the Foreground Object

• Noise and similarities between the foreground
  object and the background cause holes to appear
  in the foreground object.
• To eliminate these hole we dilate the foreground
  pixels using the Intel library function
  iplDialte
• This has the effect of growing the size of each
  pixel and hopefully filling in the holes.

16
Calculating the Gradient

• This topic is covered by Anthony Dos
  presentation.

17
Pong

• A Data structure is created which contains the
  information about the ball and paddles.
• After each input frame is processed, the systems
  uses the global gradient to update the position
  of the players paddle.

18
Pong

• The computers paddle and the balls location are
  also updated for each input frame.
• On each output frame the location of the paddles
  and the ball are displayed.

19
System Properties

• Frames for Training- The number of input frame
  to use when training the background.
• of Deviations- the degree to which new frames
  pixels must deviate from the background image
  before being considered as potential foreground
  pixels.

20
System Properties

• Dilation Size- The amount to expand each pixel in
  the selected foreground object to fill in the
  holes created by noise.
• Min Region- The minimum area of a foreground
  object to be considered.

21
Results

• Our initial results were poor due to the
  auto-gain feature of our camera.
• Auto-gain is a feature which compensates for
  changing lighting values by adjusting the
  contrast of all the pixels to make the image
  appear more balanced.

22
Results

• The auto-gain feature was thereby changing the
  values of our background pixels and was thus
  violating our constant background constraint.
• Once this was discovered and the auto-gain turned
  off, our results were dramatically improved.

23
Results

• A player is able to control the paddle with some
  degree of accuracy.
• Noise and inaccurate gradient determination still
  make the game somewhat difficult to play. Though
  a player with little experience is still able to
  hit the ball 3 to 4 times in a row.

24
Further Issues

• Improving the output and playablity of the game
  are the primary areas which need attention.
• The OpenCv library provides a substantial savings
  in terms of processor time for all calculations.
  In fact, the 30 frames per second rate would be
  unattainable if this code were to be written in
  standard C.