Hand-Eye Coordination and Vision-based Interaction

1
Hand-Eye Coordination and Vision-based Interaction
Martin Jagersand Collaborators Zach Dodds, Greg
Hager, Andreas Pichler
2
Vision-based Tasks
3
Everyday Physical InteractionsInstalling a
Light Bulb
4
Sensory-Motor Coordination

• Humans interact effortlessly with the world
  through vision and physical contact
• Fundamental Sensory-Motor transforms

5
Different issues

• What tasks can be performed?
• Camera models, geometry, visual encodings
• How to do vision guided movement?
• H-E transform estimation, feedback, feedforward
  motion control
• How to plan, decompose and perform whole tasks?

6
Task and Image Specifications
task space error
task space satisfied
image space error
image space satisfied
7
Task ambiguity

• Will the scissors cut the paper in the middle?

8
Task ambiguity

• Will the scissors cut the paper in the middle? NO!

9
Task Ambiguity

• Is the probe contacting the wire?

10
Task Ambiguity

• Is the probe contacting the wire? NO!

11
Task Ambiguity

• Is the probe contacting the wire?

12
Task Ambiguity

• Is the probe contacting the wire? NO!

13
Camera Models

• Internal calibration Longuet-Higgins 1981
• Weak calibration Faugeras 1992, Hartley 1992
• Affine calibration Horaud et al. 1998, 1999
• Stratification of stereo vision Faugeras
  1995 - characterizes the reconstructive
  certainty of weakly, affinely, and
  internally calibrated stereo rigs

up to a similarity (scaled Euclidean
transformation)
up to an affine transformation of task space
up to a projective transformation of task space
reconstruction up to a bijection of task space
14
Visual Invariance
15
Task Invariance
A task T(f)0 is invariant under a group G of
transformations, iff
x
n
n
? f ? V , g ? G with g(f ) ? V T(f )0
? T(g(f ))0
x
T(f ) must be 0 here, if T is G invariant
sim
If T(f ) 0 here,
T(f ) must be 0 here, if T is G invariant
aff
T(f ) must be 0 here, if T is G invariant
proj
T(f ) must be 0 here, if T is G invariant
inj
16
JacobianEstimation
17
Estimating a Force Frame
18
Composite Task Solving a Puzzle
19
Modulating Basis Vectors
20
Human Arm Animation
Training sequence
Simulated Animation