Using Mac OS X for RealTime Image Processing

1
Using Mac OS X for Real-Time Image Processing

• Daniel Heckenberg
• UNSW

2
Real-Time Image Processing

• Many promising applications
• video conferencing
• augmented reality
• context aware computing
• video-based interfaces for human-computer
  interaction
• RTIP is significantly different to video editing

3
Real-Time Image Processing

• System diagram
• Hardware selection and software performance are
  both crucial

4
Real-Time Image Processing

• Platform Requirements
• high resolution, high frame rate video input
• low latency video input
• low latency OS scheduling
• high processing performance

camera
ADC
driver
RTIP
display
bus
5
Sampling Resolution

• Most video hardware uses broadcast video derived
  formats
• NTSC 720x480 at 30 fps
• PAL 768x576 at 25 fps
• Spatial resolution and temporal resolution are
  both crucial.

camera
ADC
driver
RTIP
display
bus
6
Low Latency Video Input

• Latency targets
• perceived causality 50ms
• perceived synchronicity 10ms
• Unavoidable latency
• 1 to 2 frames (40 - 80ms for PAL)
• frame accumulation (up to 1 frame period)
• frame transmission (1 frame period)
• Additional latency must be minimized

camera
ADC
driver
RTIP
display
bus
7
Other latency sources

• Software-related
• bus / driver buffering
• driver / application buffering
• processing time
• Hardware-related
• output device buffering
• screen double-buffering
• audio device buffering
• Key factor OS scheduling latency

camera
ADC
driver
RTIP
display
bus
8
High Processing Performance

• Both latency and throughput are important
• PAL video frame 884Kb (YCrCb 422)
• Sustained data rate 22Mb/s
• Memory bandwidth is crucial
• AltiVec is very useful

camera
ADC
driver
RTIP
display
bus
9
Video Capture Hardware

• USB 1.0 devices arent suitable
• Firewire DV devices arent suitable
• latency
• DV decoding is CPU intensive
• PCI hardware
• good match to Quicktime architecture
• few affordable capture cards for Mac OS X

camera
ADC
driver
RTIP
display
bus
10
Video Capture Hardware

• Suitable FireWire devices
• DFG/1394-1
• Analog video input, FireWire output
• Proprietary, uncompressed wire protocol
• IIDC (Instrumentation Industrial Digital
  Cameras)
• Not limited to broadcast video formats
• Host control of camera modes
• Good range of features and costs

camera
ADC
driver
RTIP
display
bus
11
QuickTime Video Capture

• Oriented towards recording and editing in
  compressed formats using hardware-based codecs.
• Throughput favoured over latency
• no dropped frames
• not designed for multitasking or multithreading
• These issues can be overcome.

camera
ADC
driver
RTIP
display
bus
12
QuickTime Components

• Sequence grabber
• High level capture control
• not well suited to low-latency capture
• Video Digitiser - vdig
• Video device representation
• can be configured and controlled directly by an
  application
• can achieve good latency and performance

camera
ADC
driver
RTIP
display
bus
13
High Performance IIDC capture

• IIDC modes

camera
ADC
driver
RTIP
display
bus
14
Third-party IIDC driver

• Apples IIDC driver has partial support
• IOXperts vdig is more flexible
• supports 640x480 YUV capture at 30fps with
  common, cheap IIDC devices including iBot, Fire-I
  iSight
• can use 411 colour spatial sampling in the
  camera and on the wire
• must perform a conversion to QT supported YUV
  422 format in CPU

camera
ADC
driver
RTIP
display
bus
15
Low-Latency Capture

• Capture cycle
• request frame capture from vdig
• perform image processing
• return frame to vdig
• Use asynchronous capture so we dont stall the
  processor
• Many vdig drivers support multiple outstanding
  capture buffers
• Overlap frame capture and frame processing

camera
ADC
driver
RTIP
display
bus
16
Asynch, Overlapped Capture

• Set up capture device
• SetupVDig()
• VDSetDigitizerRect() VDSetCompressionOnOff(vdCom
  p, 1)VDSetFrameRate()         VDSetCompression()
         VDGetDigitizerRect()
• VDGetImageDescription(imageDesc)  VDResetCompre
  ssSequence()
• VDCompressOneFrameAsync()
• StartVDigPolling(myVDigPollFunc, pollPeriod)

camera
ADC
driver
RTIP
display
bus
17
Asynch, Overlapped Capture

• Poll for captured frames
• myVDigPollFunc()
• if (!VDCompressDone(queuedFrames)
  queuedFrames)
• VDCompressOneFrameAysnc()
• myProcessFrame()
• VDReleaseCompressBuffer()

camera
ADC
driver
RTIP
display
bus
18
Display of Image Sequences

• Important for monitoring of RTIP and for system
  output
• Hardware accelerated formats are much more
  efficient - everything else must be converted in
  CPU
• In OS X, openGL texture formats are the only
  hardware accelerated image formats
• monochrome (8bpp) raw GL_LUMINANCE
• RGB (24bpp) raw GL_RGB
• RGBA (32bpp) raw GL_RGBA
• YCbCr (16bpp) 2yuv GL_APPLE_ycbcr_422

camera
ADC
driver
RTIP
display
bus
19
QuickTime Display

• QuickTimes most often used YUV format yuvu
  (kComponentVideoPixelType) is not accelerated in
  Mac OS X
• Image Compression manager functions should be
  used for image sequence display
• initialisation per sequence, not each frame
• implicit conversion of frames into displayable
  formats
• functions are prefixed with DecompressSequence

camera
ADC
driver
RTIP
display
bus
20
OpenGL Image Display

• Best performance for image sequence display in OS
  X is achieved by using openGL directly
• Apple specific extensions allow efficient texture
  transfers for video frames
• GL_APPLE_ycbcr_422
• YCbCr 422 texture format
• GL_APPLE_client_storage
• use an application supplied buffer for the
  texture, rather than making a copy
• GL_APPLE_texture_range
• control over texture caching and memory mapping
  (AGP)
• GL_TEXTURE_RECTANGLE_EXT
• textures with non-power of two dimensions

camera
ADC
driver
RTIP
display
bus
21
Development

• Software profiling is indispensible to target
  performance bottlenecks
• Apples CHUD tools provide low-level analysis of
  the time-performance of software
• Shikari offers excellent profiling support
• including modules for which source is not
  available e.g. Quicktime
• Apples sample code library offers examples of
  some of the techniques presented

camera
ADC
driver
RTIP
display
bus
22
Conclusion

• RTIP involves many aspects of hardware and
  software in order to achieve
• high resolution input
• low latency capture
• high performance processing
• efficient display
• Mac OS X and suitable hardware provides a capable
  platform for RTIP using the techniques presented

camera
ADC
driver
RTIP
display
bus
23
QA