Department of Information Physics and Computing,

1
Markerless Laser-Based Tracking for Real-Time 3D
Gesture Acquisition
Álvaro Cassinelli, Stéphane Perrin, Masatoshi
Ishikawa
Writing in the air
Department of Information Physics and Computing,
University of Tokyo
Demos Results
Introduction
? The proposed interface enables interaction and
data input by just executing bare-hand gestures
in front of the portable device.

• Angular resolution of about 0.2 (or 3 mm at one
  meter), over an angular viewing field of about
  25.

If a gesture remains relatively flat, projecting
the acquired 3D data over a (real-time estimated)
best fitting plane will suffice to feed a 2D
character recognition algorithm.

• Sub-centimeter depth discrimination up to 30 cm.

Inputting information and interacting with
portable electronic devices is becoming a
challenging task as these become smaller and
smaller.
Most PDAs offer pointing and clicking as well as
alphanumeric data inputting through a touch
sensitive screen and a stylus. The input space is
thus merged with the viewing space, allowing a
substantial saving of physical space.

• Prototype working distance up to 120 cm.

? This is the first step towards a
constraint-free input interface for character
recognition that would enable the user to write
freely without paying attention to the relative
position of hand and acquisition system.
The next logical step would be to remove the need
for any input space (dedicated or shared with the
viewing space), as well as the need of an
additional input device (such as a stylus,
data-gloves, etc).
? No dedicated input space the proposed
human-machine interface can be embodied in a
keyboardless wrist-watch.
(Depth is computed from the intensity of the
backscattered signal.)
/

• Maximum tracking speed of 3 m/s at a distance of
  about one meter, only limited by the A/D
  conversion rate.

We are investigating a laser-based human-machine
interface capable of tracking simultaneously one
or more fingers, as well as acquiring their
respective 3D coordinates in real time.
Multiple target tracking
tracking the free-fall of a ping-pong ball...
Tracking of multiple targets without replicating
any part of the system was successfully
demonstrated (targets are considered
sequentially).
The system is based on an active tracking
mechanism using a laser diode, a pair of steering
micro-mirrors, and a single non-imaging
photodetector.
...corresponding real-time acquired data.
? Coupling this interface with a projective
display will remove the need of any real
interaction area (for viewing or inputting).
? Multiple user interaction on the same virtual
space
? Virtual manipulation (CAD, molecular biology,
3D animation)
(Typical speed of a hand/finger performing
gestures is lt 2.5 m/s)
? Gesture-driven user interface, as imagined in
Spielberg's film Minority Report (but without
special gloves nor markers).
Optical Setup

• Non-imaging photodetector
• Insensitive to harsh or changing lighting
  conditions thanks to wavelength filtering /
  synchronous detection.

Tracking Principle
Sequence shows simultaneous tracking of a pair of
fingertips (on same hand and different hands).
? The system is a smart rangefinder scanner that
restricts its scanning area, on the basis of a
real-time analysis of the backscattered signal,
to a narrow window matching the size of the
target.

• Pair of galvano-mirrors
• Used for both generating the saccade and
  performing the actual tracking.

Conclusion
Future Works
More precisely, a millisecond-long circular laser
saccade is generated around the current target
position.
The proposed active-tracking mechanism has some
appealing features over other conventional
(passive or active) tracking systems

• Hardware Software
• - signal-to-noise improvement
  (synchronous photo-detection, optical pick-up
  configuration)
• - robust tracking using predictive
  filters (e.g. Kalman)
• - tracking distant targets using more
  sophisticated telemetric techniques (i.e.
  time-of-flight measurement)

• Class II laser diode (633 nm, max. power 1 mW)
• Visibility of the source simplifies calibration,
  but is not compulsory.

• This saccade is supposed to remain fully inside
  the object while tracking

• Few constraints on the user side. No special
  input posture no additional input device (active
  transceivers, reflective optical markers,
  gloves).

• User hand.
• Light is isotropically backscattered by the
  skin.
• No gloves/markers needed on the user hand

• For distances less than 10 cm the required
  optical power drops below 30?W.

(b) As the object moves, a small portion of the
saccade may fall outside the object, and the
backscattered signal will momentarily drop. Due
to the synchronous operation of the beam-steering
mirrors and the photodetection, an accurate
recentering vector (blue arrow) is computed
?Applications - 3D gesture recognition
based on HMMs - feasibility of a 3D
graffiti alphabet
? Few constraints on the working environment. It
does not require a special projection area (other
than the hand itself) as an active-illumination
system, it does not imposes stringent
illumination conditions.
? Overall complexity of the setup is equivalent
to that of a portable laser-based barcode reader.

• Setup miniaturization using MEMS micro-mirrors.

? Real-time, full 3D acquisition. Tracking and
three-dimensional data acquisition is achieved
without the need for a stereoscopic camera.
more...
(c) The center of the saccade is updated
accordingly.
? System on-chip integration. The simplicity of
the system is such that, using state-of-the-art
Micro-Opto-Electro-Mechanical-System (MOEMS)
technology, it would be possible to integrate the
whole system on a chip, making it an interesting
input interface for portable computing devices.
www.k2.t.u-tokyo.ac.jp/fusion/LaserActiveTracking

• S. Perrin et al., Gesture Recognition Using
  Laser-based Tracking System, FG 2004, Seoul,
  17-19 May 2004.

The sequence (a)-(b)-(c) is repeated every
millisecond.

• S. Perrin et al., Laser-Based Finger Tracking
  System Suitable for MOEMS Integration, IVCNZ
  2003, New Zealand, 26-28 Nov. 2003.