Person Following with a Mobile Robot Using Binocular Feature-Based Tracking

1
Person Following with a Mobile RobotUsing
Binocular Feature-Based Tracking

• Zhichao Chen and Stanley T. Birchfield
• Dept. of Electrical and Computer Engineering
• Clemson University
• Clemson, South Carolina USA

2
Motivation

• Goal Enable a mobile robot to follow a person in
  a cluttered indoor environment by vision.
• Previous approaches
• Appearance properties color, edges. Sidenbladh
  et al. 1999, Tarokh and Ferrari 2003, Kwon et
  al. 2005
• Person has different color from background or
  faces camera.
• Lighting changes.
• Optical flow. Piaggio et al 1998, Chivilò et
  al. 2004
• Drift as the person moves with out-of-plane
  rotation
• Dense stereo and odometry. Beymer and Konolige
  2001
• difficult to predict the movement of the robot
  (uneven surfaces, slippage in the wheels).

3
Our approach

• Algorithm Sparse stereo based on Lucas-Kanade
  feature tracking.
• Handles
• Dynamic backgrounds.
• Out-of-plane rotation.
• Similar disparity between the person and
  background.
• Similar color between the person and background.

4
System overview
5
Detect 3D features of the scene

• Kanade-Lucas-Tomasi (KLT) feature tracker
• Automatically selects 2D features using
  eigenvalues of 2x2 gradient covariance matrix
• Automatically matching features by minimizing sum
  of squared differences (SSD) between left and
  right images.
• Augmented with gain and bias to handle lighting
  changes
• Open-source implementation

gradient of image
unknown displacement
gray-level images
http//www.ces.clemson.edu/stb/klt
6
Detect 3D features of the scene ( Cont. )

• Features are selected in the left image IL and
  matched in the right image IR.

Left image
Right image
7
System overview
8
Overview of Removing Background
1) using the known disparity of the person in the
previous image frame.
2) using the estimated motion of the background.
3) using the estimated motion of the person
9
Remove BackgroundStep 1 Using the known
disparity

• Discard features for which
• where is the known disparity of the
  person in the previous frame,
• and is the disparity of a feature at time
  t .

Original features
10
Remove BackgroundStep 2 Using background
motion

• Estimate the motion of the background by
  computing a 4 4 projective transformation
  matrix H between two image frames at times t and
  t 1

• Random sample consensus (RANSAC) algorithm is
• used to yield dominant motion.

11
Remove BackgroundStep 3 Using person motion

• Similar to step 2, the motion model of the person
  is calculated.
• The size of the person group should be the
  biggest.
• The centroid of the person group should be
  proximate to the previous location of the person.

Foreground features after step 2
Foreground features after step 3
12
System overview
13
Detect Face

• The Viola-Jones frontal face detector is applied.
• This detector is used both to initialize the
  system and to enhance robustness when the person
  is facing the camera.

Note The face detector is not necessary in our
system.
14
System overview
15
Experimental Results
16
Video
17
Comparison with a ColorHistogram-Based Algorithm
Color-based (Camshift, Bradski 1998)
Our approach
18
Conclusion

• Approach
• detects and matches feature points between a
  stereo pair of images and between successive
  images.
• RANSAC-based procedure to estimate the motion of
  each region.
• Advantages
• does not require the person to wear a different
  color from the background.
• track a person in an office environment, even
  through doorways, with clutter, and in the
  presence of other moving objects.
• Future work
• fusing the information with additional
  appearance-based information ( template or edges)
  .
• integration with other modules (obstacle
  avoidance)