Pipeline for DSP in Interactive Audio

1
Pipeline for DSP in Interactive Audio

• Keith Weiner
• DiamondWare
• keith_at_dw.com

2
What are We Trying to Do?

• General-Purpose
• Realtime
• Interactive
• Low latency
• Interoperability
• Agnostic to h/w vs. s/w
• Agnostic to renderer

3
Overview

• Existing Systems
• Requirements
• Pipeline Operation
• Demonstration
• Conclusions

4
Fixed-Function Accelerator

• Sound Blaster Live
• Detailed parameters for 3D
• No tempo changer or flange
• The Roach Motel Model
• API inertia

5
Proprietary Plugins

• WinAmp
• Great within sys. designers vision
• Support for multiple compression format
• Inflexible and limited
• Plugin provides GUI for parameters
• Proprietary

6
Filter Graph

• DirectShow
• Diverse modules available
• Unlimited connection topology
• Param. changes not under app. Control
• Unbounded latency
• No on the fly buffer length change
• Useful in authoring tools

7
Requirements

• Universal
• Polyphonic
• Control over connection topology
• On the fly changes to any parameter
• Compute efficient
• Easy to code a processing segment

8
Latency

• Impacts user experience
• Buffer mgmt. is most important task
• Small buffer sizes
• But never allow renderer to starve
• Asynchronous (e.g. h/w) possible

9
The Pipeline
10
Pipeline Concept

• Term borrowed from 3D graphics
• Implementation unique to audio
• Segment is a processing unit
• Pipeline has ordered set of segments
• Segments opaque to pipe, each other
• Segment is useful abstraction

11
Pipeline Data Flow

• Sink pulls from source
• Sink is inflexible in
• Sampling rate
• Timing
• Buffer size
• Source is usually flexible
• Notable exception is telephony

12
Segment Assumptions

• Segment is source for next in pipeline
• And sink for previous
• Input length ! output length
• Input and output lengths not constant
• Parameters not constant
• Private data is persistent

13
Execution Flow I

• Renderer (last seg.) needs X samples
• Query made to prev segment
• To output X samples, how many input?
• Then query previous, etc.
• Queries traverse pipe backwards
• Last query gives num. source samples

14
Execution Flow II

• Source called to produce the samples
• Its output sent to next segment, etc.
• Final output sent to renderer
• Hopefully this is the right amount
• It may not be

15
Execution Flow III

• Segment may
• Consume too little input
• Produce too little output
• Produce too much output
• Pipeline must provide smart buffering
• There are tricky special cases
• Segments have minimal glue logic

16
Demonstration
17
Technical Conclusions

• Meets needs of interactive software
• No limitations on apps or processing
• Additional features beneficial
• Add, remove, reorder segments
• Meter large parameter changes
• Sync between different streams
• Deterministic CPU utilization

18
Market Conclusions

• Aligned with vendors
• CPUs
• Algorithms
• DSP h/w
• Fixed-fn. board vendors?
• OS vendors?

19
More Details in Paper

• Limitations of the Filter Graph
• Latency
• Segment messages
• Pipeline operation
• Work to improve system