An Integrated System of 3D Motion Tracker and Spatialized Sound Synthesizer

1
Motivation Implementing a vision-based real time
tracking system in a 3D environment and use the
tracking result to control the spatialized sound
synthesizer for online interactive music
composition
Hardware Software Unibrain Fire-i cameras,
Windows XP PC, Apple Powerbook and G5, four
speakers, OpenCV library, OSC, SuperCollider,
Max/MSP/Jitter. All machines communicate via the
network.
Segmentation
Temporal Difference
Thresholded Temporal Difference
Thresholded Background Subtraction
LeftCam
CenterCam
RightCam
Center of Mass
Center of Mass
Center of Mass
Camera calibration
Tracking results
2.09 fps
OSC Messages
World_X

• 3D Position Calculation
• Epipolar Geometry
• Least Square

World_Y
Xcamera MextXworld
Ximage MintXcamera
World_Z
A captured frame from the visualization
Performance Estimation Error (cm)
The Data Manager distributes the tracking results
to the visualization system and the synthesis
servers.
EPP-avg EPP-std LS-avg LS-std
X-direction 3.6 7.0 5.3 8.7
Z-direction 5.9 33.5 5.7 30.0
Music Synthesis
Calibration/Tracking
Object paths create phrase level material in the
mobile form.