Mobile Motion Tracking using Onboard Camera

1
Mobile Motion Tracking using Onboard Camera

• Supervisor
• Prof. LYU, Rung Tsong Michael
• Presented by
• Lam Man Kit, Wong Yuk Man

2
Outline
1. Motivation Objective
2. Motion Input
3. Previous Work
4. Improvement New Components
5. Applications
6. Experimental Result
7. Demo
3
Motivation

• Rapid increase in the use of camera-phone.
• Camera-phones processing power is increasing
• Symbian OS makes programming on mobile phone
  possible

4
Objective

• Develop motion input method on Symbian phone
  without requiring additional hardware
• This acts as an innovative input method for
  applications such as
• Camera mouse to control the cursor
• New input method for interactive games
• Gesture input

5
What is Motion Input?
A
6
Why Motion Input?

• Motion input is quick and convenient
• Common way to play game on mobile phone is to use
  joystick or keypad
• Limited freedom of movement
• Lack of maneuverability
• Difficult to press the small-sized keypad
• Motion input doesnt have these problems!

7
Motion Tracking

• Motion tracking is a process to find the motion
  vector of the current frame from the reference
  frame
• We use block matching algorithm to find the
  motion vector

8
Block Matching Algorithm
Comparison of blocks
New Position
The motion vector which corresponds to the best
match
(dx,dy)
9
Block Matching Algorithm

• Evaluate the "goodness" of a match by
• Sum of Absolute Difference
• Select the candidate block with the lowest error

10
Motion Tracking

• Block matching algorithm
• Traditional block matching algorithm is slow
• Our solution Hybrid-type fast block matching
  algorithm
• That is Adaptive Spiral SEA PPNM SSD algorithm
• Adaptive Search Window
• Spiral Scan Method
• Successive Elimination Algorithm
• Progressive Partial Norm Matching
• Sum of Squared Difference

11
Previous Work

• In the previous semester,
• Developed a testing platform in both Window and
  Symbian OS
• Developed our own block matching algorithm and
  feature selection algorithm
• Built a real-time translational motion tracking
  engine
• Developed some applications using the engine

12
Improvement on Feature Selection

• A good feature block
• High variance -gt complex block
• Not in a repeated pattern
• If a good feature block is found, the performance
  of the motion tracking will be improved
• We have made improvements to our existing
  algorithm

13
Old Feature Selection

• Divide the current frame into squares
• Apply Feature Selection Algorithm to each square
• However, the best feature block may appear
  between two squares

Best feature block
14
New Feature Selection

• More blocks are sampled
• Sample 26x26 squares with size 15x15 in a 54x54
  window
• Old feature selection only samples 12x12 squares
• Search in spiral way and stop searching when
  selection criteria are matched
• Prefer to find a feature block in the center
• Prevent out of bound problem (block appear out of
  screen)

15
New Feature Selection

• If a feature block is found on the edge, the
  following will happen
• The tracking algorithm will fail to find the
  exact match
• Solution
• Apply additional constraint

16
New Feature selection

• In order to find a feature block that is not on
  the edge, we apply a checking algorithm
• Important difference in all directions -gt
  interest point

17
New Feature Selection

• Now our newest feature selection algorithm
  becomes
• Find a block with a large variance which
  indicates the complexity of the block
• Check if the block is on the edge or not by
  calculating the SAD between the candidate block
  and its 4 neighbors
• If either one of the SAD is small -gt the block is
  on edge -gt reject
• Else the block is not on edge

18
Rotation Tracking Engine

• Observation and Motivation
• As we rotate the phone, the object can still be
  tracked correctly (center of object always
  roughly equals center of green box)

19
Rotation Tracking Engine

• First approach
• Track one block by translational motion tracking
  engine, rotate the current block and then find
  which rotation of the current block best match
  with the previous block
• Fail if tracking object is the same for different
  angle

One-block approach
20
Rotation Tracking Engine

• Our approach
• If two blocks are tracked simultaneously, angle
  of line connecting the two blocks indicates the
  rotation angle of the phone
• Use only translational motion tracking

Two-block approach
21
Rotation Tracking Engine

• Modification of adaptive search window method to
  suit rotation tracking
• Phones motion can be both linear motion and
  circular motion
• Linear Adaptive method used in translational
  motion tracking is not used in this engine
• Why need adaptive search window method?
• To increase performance based on the fact that
  during block matching, reaching the best
  candidate block earlier increases the elimination
  effect of SEA, PPNM, PDE

22
Rotation Tracking Engine

• What to predict?
• Predict the coordinates of the blocks in the next
  frame
• Given the coordinates of the tracking blocks in
  the current frame and the previous frames
• When to predict?
• After each run of rotation tracking algorithm

Line L1
Line L2
?
?
Line L3
23
Rotation Tracking Engine

• Solution
• Angle between L2 and L3 ?
• ? Tan-1(slope(L2)) Tan-1(slope(L3))
• Coordinates of the next tracking block (xL1, yL1)
  are calculated by multiplying the column matrix
  of coordinates of the previous block with a
  rotation matrix

24
Rotation Tracking Engine

• Solution
• The prediction of the positions of the next
  tracking blocks should also take the
  translational movement into account
• Horizontal displacement Tx
• Tx ( xL21 xL22 xL31 - xL32 )/2
• Vertical displacement Ty
• Ty ( yL21 yL22 yL31 - yL32 )/2
• Coordinates of the next tracking block (xL1, yL1)
  is calculated by

25
Rotation Tracking Engine

• Reducing error by using level
• Increase/decrease one level only when change is
  large
• To give less sensitive but more desirable output
  for game
• e.g. skiing game skier face only to 7 directions
• Reduce difficulty of the game, increase
  reliability of the engine

26
Rotation Tracking Engine

• Some screenshot showing how rotation tracking is
  carried out and how the result is displayed

27
Rotation Tracking Engine
A short video demo
28
Tailor-made background for both engines

• Objective
• Max. performance measurement
• Increase usability
• Condition
• Feature selection can always find a good feature
  point
• Within certain distance
• No repeat pattern
• Very distinct pattern
• Pattern is nearly the same when rotated. E.g.
  Circle is good for rotation detection

29
Virtual Mouse

• An application that make full use of the
  translational motion tracking engine
• Remote control the mouse of PC by mobile phone
  using motion as input
• Advantages
• It allows input in many directions and provides
  high levels of control
• Many buttons left for other purposes
• E.g. Joystick can be used as course adjustment of
  mouse cursor
• E.g. Some buttons can be used as shortcuts of
  applications

30
Virtual Mouse
31
Virtual Mouse

• Server
• Configure Bluetooth device to provide RFComm
  Service and regards the Bluetooth transmission
  port as Comm. Port
• Server receives message from that Comm. Port
• Call function in MouseAction.h to make the mouse
  move and trigger mouse click event

32
Virtual Mouse

• Client
• Search for Bluetooth device nearby
• Connect to the selected Bluetooth device
• Every time motion tracking algorithm finishes,
  results are sent to server (14 times/sec)
• Joystick and keypad can also be used to control
  mouse in PC (Multi-button mouse)

33
Car Racing Game

• A Car Racing Game is developed using the motion
  tracking engine.
• Game lags. It is because
• The engine takes time to find the motion vector
• CPU speed of the Symbian phones is low
• Solutions
• Double Buffering
• Direct Screen Access

34
Skiing Game

• We have developed a skiing game that makes use of
  the rotation tracking engine
• Rotate the phone to control the angle of the
  skier
• After using the rotation tracking engine
• The game becomes more interactive
• The game has more degree of freedom for rotating
  the skier

35
Skiing Game
36
Skiing Game

• The process of using the rotation tracking engine
  is very simple
• Create an instance of the tracking engine
• Call the function of the engine to find the
  motion vector
• Use the motion vector for the game logic

37
Skiing Game
38
Experimental Result on Symbian phone

• Testing Environment
• Platform
• Symbian Phone Nokia 6600
• Algorithm
• Our final (hybrid-type) algorithm (Block matching
  algorithm featured with Adaptive Window, Spiral
  Scan, SEA, PPNM and PDE method)
• Algorithm parameter
• Block size         17 x 17 (pixels)
• Search window size     16 x 16 (pixels)
• Number of block to track in each run of algorithm
  1 block ONLY

39
Result
40
Experimental Result on Symbian phone

• Testing Result (avg time to run)
• Final algorithm
• 7ms
• Partial Spiral algorithm
• 22ms
• Full Exhaustive Search algorithm (the most
  simplest one)
• 55ms

41
Result
42
Conclusion

• We have developed
• The real-time motion tracking algorithm and
  feature selection method
• The translational and rotational motion tracking
  engine on Symbian
• A series of applications/games that make full use
  of our engines

43
Demo

• Virtual Mouse

44
Demo

• Act as an innovative input device for games

45
Q A
46

• Thank You !!

47
Experimental Result1 - Backgrounds
48
Experimental Result2 - Backgrounds
49
Stereo

• If we want to track the movement in 3D, we have
  to
• Use two cameras
• Know the focal length of the cameras
• Know the distance between two cameras
• Depth can be recovered with two images and
  triangulation

50
Stereo

• Therefore, stereo has two steps
• Finding matching points in the images
• Then using them to compute depth.

P
P1
P2
O2
O1
51
Stereo

• Suppose we have found the matching point P. Now
  the step to compute the depth is as follow

Note y-axis is perpendicular to the slide
b distance between two cameras
f focal length of the cameras
52
Stereo

• If b is the distance between the two cameras, and
  f is the focal length of the cameras, we have
• Use similar triangular, we can solve to get

53
Mono

• In our project, only one camera is available in
  the Symbian phone
• If we want to find the z-axis movement, we can
  find it by comparing the size of the capture
  image
• Getting closer -gt larger image
• Getting farther -gt smaller image
• However, we have tried that using this algorithm
  is not accurate enough