Consistent Visual Information Processing

1
Consistent Visual Information Processing
Axel Pinz EMT Institute of Electrical
Measurement and Measurement Signal
Processing TU Graz Graz University of
Technology pinz_at_emt.tu-graz.ac.at http//www.emt.
tu-graz.ac.at/pinz
2
Consistency

• Active vision systems / 4D data streams

• Multiple visual information

• Imprecision

• Ambiguity

• Contradiction

3
This Talk Consistency in

• Active vision systems
• Active fusion
• Active object recognition
• Immersive 3D HCI
• Augmented reality
• Tracking in VR/AR

4
AR as Testbed

• Consistent perception
• in 4D
• Space
• Registration
• Tracking
• Time
• Lag-free
• Prediction

5
Agenda

• Active fusion
• Consistency
• Applications
• Active object recognition
• Tracking in VR/AR
• Conclusions

6
Active Fusion
Simple top level decision-action-fusion loop
7
Active Fusion (2)

• Fusion schemes
• Probabilistic
• Possibilistic (fuzzy)
• Evidence theoretic (Dempster Shafer)

8
Probabilistic Active Fusion
N measurements, sensor inputs mi
M hypotheses oj , O o1, , oM
Bayes formula
Use entropy H(O) to measure the quality of P(O)
9
Probabilistic Active Fusion (2)
Flat distribution P(oj )const. ? Hmax
Pronounced distribution P(oc ) 1 P(oj )
0, j ? c ? H 0

• Measurements can be
• difficult,
• expensive,
• N can be prohibitively large,
• ? Find iterative strategy to minimize H(O)

10
Probabilistic Active Fusion (3)
Start with A ? 1 measurements P(ojm1, ,mA),
HA
Iteratively take more measurements mA1, ,mB
Until P(ojm1, ,mB), HB ? Threshold
11
Summary Active Fusion

• Multiple (visual) information, many sensors,
  measurements,
• Selection of information sources
• Maximize information content / quality
• Optimize effort (number / cost of measurements, )

Information gain by entropy reduction
12
Summary Active Fusion (2)

• Active systems (robots, mobile cameras)
• Sensor planning
• Control
• Interaction with the scene
• Passive systems (video, wearable,)
• Filtering
• Selection of sensors / measurements

13
Consistency

• Consistency vs. Ambiguity
• Unimodal subsets Ok
• Representations
• Distance measures

14
Consistent Subsets

• Hypotheses O o1 ,, oM
• Ambiguity P(O) is multimodal
• Consistent unimodal subsets Ok ? O

• Application domains
• Support of hypotheses
• Outlier rejection

Benefits
15
Distance Measures

• Depend on representations, e.g.
• Pixel-level SSD, correlation, rank
• Eigenspace Euclidean
• 3D models Euclidean
• Feature-based Mahalanobis,
• Symbolic Mutual information
• Graphs Subgraph isomorphism

16
Mutual Information
Shannons measure of mutual information O o1
,, oM A ? O, B ? O I(A,B) H(A) H(B)
H(A,B)
17
Applications

• Active object recognition
• Videos
• Details
• Tracking in VR / AR
• Landmark definition / acquisition
• Real-time tracking

18
Active vision laboratory
19
Active Object Recognition
20
Active Object Recognitionin Parametric Eigenspace

• Classifier for a single view
• Pose estimation per view
• Fusion formalism
• View planning formalism
• Estimation of object appearance at unexplored
  viewing positions

21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
Applications

• ? Active object recognition
• Videos
• Details
• ? Control of active vision systems
• Tracking in VR / AR
• Landmark definition / acquisition
• Real-time tracking
• ? Selection, combination, evaluation
• ? Constraining of huge spaces

29
Landmark Definition / Acquisition

• What is a landmark ?

corners
blobs
natural landmarks
30
Automatic Landmark Acquisition

• Capture a dataset of the scene
• calibrated stereo rig
• trajectory (by magnetic tracking)
• n stereo pairs
• Process this dataset
• visually salient landmarks for tracking

31
Automatic Landmark Acquisition

• visually salient landmarks for tracking
• salient points in 2D image
• 3D reconstruction
• clusters in 3D
• compact, many points
• consistent feature descriptions
• cluster centers ? landmarks

32
Processing Scheme
33
Office Scene
34
Office Scene - Reconstruction
35
Office Scene - Reconstruction
36
Unknown Scene
Landmark Acquisition
Real-Time Tracking
37
Real-Time Tracking

• Measure position and orientation of object(s)
• Obtain trajectories of object(s)
• Stationary observer outside-in
• Vision-based
• Moving observer, egomotion inside-out
• Hybrid
• Degrees of Freedom DoF
• 3 DoF (mobile robot)
• 6 DoF (head and device tracking in AR)

38
Outside-in Tracking (1)
stereo-rig IR-illumination

• wireless
• 1 marker/device
• 3 DoF
• 2 markers 5 DoF
• 3 markers 6 DoF

devices
39
Outside-inTracking (2)
40
Consistent Tracking (1)

• Complexity
• Many targets
• Exhaustive search vs. Real-time
• Occlusion
• Redundancy (targets cameras)
• Ambiguity in 3D
• Constraints

41
Consistent Tracking (2)

• Dynamic interpretation tree
• Geometric / spatial consistency
• Local constraints
• Multiple interpretations can happen
• Global consistency is impossible
• Temporal consistency
• Filtering, prediction

42
Consistent Tracking (3)
43
(No Transcript)
44
(No Transcript)
45
Hybrid Inside-Out Tracking (1)
Inertial Tracker

• 3 accelerometers
• 3 gyroscopes
• signal processing
• interface

46
Hybrid Inside-Out Tracking (2)

• complementary sensors
• fusion

47
Summary Consistency in

• Active vision systems
• Active fusion
• Active object recognition
• Immersive 3D HCI
• Augmented reality
• Tracking in VR/AR

48
Conclusion
Consistent processing of visual information can
significantly improve the performance of active
and real-time vision systems
49
Acknowledgement
Thomas Auer, Hermann Borotschnig, Markus
Brandner, Harald Ganster, Peter Lang, Lucas
Paletta, Manfred Prantl, Miguel Ribo, David
Sinclair
Christian Doppler Gesellschaft, FFF, FWF, Kplus
VRVis, EU TMR Virgo