QOrchestra

1
QOrchestra FlowML
High-level audio synthesis tools on top of SAOL
Bert Schiettecatte, VUB
2
Introduction (1)

• This presentation
• Introduce an audio synthesis diagram format
  (FlowML)
• Present a compiler from FlowML to SAOL
  (FlowML2SAOL)
• Present a high-level authoring tool for SAOL
  (QOrchestra)

3
Introduction (2)

• The SA (Structured Audio) standard of Mpeg-4
  audio consists of 5 elements

• SAOL (SA Orchestra Language) a signal-processing
  language for describing musical instruments
• SASL (SA Score Language) a language for
  describing a performance using the instruments of
  a SAOL program

4
Introduction (3)

• SASBF (SA Sample Bank Format) allows
  transmission of sample banks
• Scheduler description used to translate MIDI or
  SASL to events internally, in a decoder
• Reference to MIDI supported MIDI messages and
  their meaning

5
Available technology (1)

• Command-line encoders, decoders, profilers

• saolc reference implementation of
  encoder/decoder
• sfront SAOL-to-C compiler with realtime I/O and
  some profiling support
• sarun compiles SAOL to a program for a virtual
  machine with a special instruction set, for
  efficient decoding

6
Available technology (2)

• Authoring tools

• QOrchestra high-level synthesis diagram tool
  with SAOL export function, non-realtime,
  supports re-use of diagrams
• CPS low-level software synthesis tool with SAOL
  save function, realtime
• VisOrc Similar to CPS, with integrated sequencer

7
Available technology (3)

• IDEs

• SFX SAOL editing environment with syntax
  highlighting, auditioning, some profiling support

8
QOrchestra (1)

• Goal experimental visual authoring tool for SAOL
  (GPL license), drawing space with primitive
  building blocks, connection tool, library for new
  blocks,
• Main idea tool for people who dont know SAOL,
  or dont want to write SAOL code
• Also a goal case study of a complex diagram
  editor and object-oriented programming technology

9
QOrchestra (2)

• QOrchestra is NOT
• A low-level SAOL editor (e.g. CPS)
• A complete SA authoring environment
• A real-time software synthesis application
• An IDE
• Finished

10
QOrchestra (3)
11
FlowML (1)

• QOrchestra project requirement save, load and
  re-use synthesis diagrams
• Available storage technology XML (eXtensible
  Markup Language), a way to add meta-information
  to information (e.g. ltauthorgtBert
  Schiettecattelt/authorgt)

12
FlowML (2)

• Tags for a specific document class are described
  in an XML DTD (document type definition)
• XML technology is freely available, tested,
  reliable, very easy to use,
• FlowML XML DTDs describing synthesis diagram
  formats on a high level collection of synthesis
  constructs synthesis environment-specific
  extension mechanism ( C pragmas)

13
FlowML (3)

• Solved storage problems in QOrchestra store
  diagrams and libraries as FlowML
• Generating SAOL code from FlowML is relatively
  easy
• Very easy to understand (stored as text file)
• Could be a bridge between various software
  synthesis systems
• Could be the right high-level tool for describing
  big synthesis systems (ltgt SAOL)

14
FlowML (4)

• Web-friendly easy to create browser plugins
• Is not a programming language, but a high-level
  modelling language, in a way similar to UML
  (methodology for structuring OO designs)
• FlowML is open and free
• Extensions for future versions can be proposed
  and will be examined prior to becoming official
  (same license as HTML)

15
FlowML (5)

• FlowML diagram rules (informal)
• Multiple outputs per block
• Multiple connections per output
• Multiple inputs per block
• Only one connection per input
• All signals are mono

16
FlowML (6)

• FlowML diagram rules (informal)
• Every component instance has a unique ID used for
  referring
• Every definition contains a description and a
  unique name used for instantiating
• 2 types of signals audio and control
• Diagram/component I/O is done through special
  primitives defined in the standard

17
FlowML (7)
Audio synthesis diagram example Reverb
18
FlowML (8)
FlowML code for the reverb diagram

• lt?xml version'1.0'?gt
• ltGraphPrototype name"revbfilt"gt
• ltDescriptiongt
• Reverb (filtered delay).
• lt/Descriptiongt
• ltImplementationgt
• ltInstance id"audio_in" type"audio_in"
  x"0" y"0"gt
• ltConnection to"in1" from"out"
  targetid"mixer"/gt
• ltConnection to"in1" from"out"
  targetid"xfade"/gt
• lt/Instancegt
• ltInstance id"fback_in" type"control_in"
  x"0" y"0"gt
• ltConnection to"lvl2" from"out"
  targetid"mixer"/gt
• lt/Instancegt
• ltInstance id"level" type"control_in"
  x"0" y"0"gt
• ltConnection to"lvl1" from"out"
  targetid"mixer"/gt
• lt/Instancegt
• ltInstance id"mixer" type"mixer" x"0"
  y"0"gt
• ltConnection to"in" from"out"
  targetid"dly"/gt
• lt/Instancegt

19
FlowML (9)
FlowML code for the reverb diagram (contd)

• ltInstance id"dly" type"delay" x"0"
  y"0"gt
• ltConnection to"in2" from"out"
  targetid"xfade"/gt
• ltConnection to"in2" from"out"
  targetid"mixer"/gt
• lt/Instancegt
• ltInstance id"xfade" type"xfade" x"0"
  y"0"gt
• ltConnection to"in" from"out"
  targetid"audio_out"/gt
• lt/Instancegt
• ltInstance id"drywet" type"control_in"
  x"0" y"0"gt
• ltConnection to"x" from"out"
  targetid"xfade"/gt
• lt/Instancegt
• ltInstance id"audio_out" type"audio_out"
  x"0" y"0"/gt
• lt/Implementationgt
• lt/GraphPrototypegt

20
SAOL Generator (1)

• Originally intended to be a part of QOrchestra,
  but now a separate program
• Reads FlowML synthesis diagrams and generates
  SAOL code with the behaviour of the diagram
• Uses libraries of SAOL opcodes to simplify
  generated code every block in the diagram
  translates to an opcode call

21
SAOL Generator (2)

• Will support libraries containing FlowML diagrams
  and SAOL code (currently only FlowML)
• Generated code will require preprocessing (e.g.
  include)
• Problem separation authoring tool (including
  synthesizer) code generator can cause
  differences in resulting sound

22
SAOL Generator (3)

• Problem Integrating software synthesizer/code
  generator might cause a problem what about
  intellectual property?

23
SAOL Generator (4)
Code generation opcode declaration

• generateOpcDecl(Prototype, Program, DeclsDone)
• VarDecls new List
• Code new List
• InstsDone new Set
• Params getInputNames(P)
• Outputs getOutputNames(P)
• foreach O in Outputs
• add(Results, generateOpcResult(Prototype, O,
  Null, Program,
• VarDecls, Code, DeclsDone,
• InstsDone))
• add(Code, makeReturn(Results))
• return makeOpcDecl(getRate(getElement(Outputs)),
  
• getName(P), Params, VarDecls, Code)

24
SAOL Generator (5)
Code generation opcode result (1/3)

• generateOpcResult(Prototype, N, C, Program,
  VarDecls, Code,
• DeclsDone, InstsDone)
• if isOutput(Prototype, N)
• X getConnectionTo(N)
• return generateOpcResult(Prototype, getFrom(X),
  
• getFromConnector(X), Program,
  VarDecls,
• Code, DeclsDone, InstsDone)
• else if isInput(Prototype, N)
• return makeVarRef(N)
• else if contains(InstsDone, N)
• if hasMultipleOutputs(N)
• return makeArrayLookup(last(N), getNumber(N,
  C))
• else
• return makeVarRef(last(N))

Output instance
Input instance
Instance already calculated
25
SAOL Generator (6)
Code generation opcode result (2/3)

• else
• add(InstsDone, N)
• if isGraphPrototype(N) and not
  contains(DeclsDone, N)
• add(DeclsDone, N)
• add(Program, generateOpcDecl(N, Program,
  DeclsDone))
• Params new List
• ConnectionsTo getConnectionsTo(N)
• foreach X in ConnectionsTo
• add(Params, generateOpcResult(Prototype,
  getFrom(X),
• getFromConnector(X), Program,
• VarDecls, Code, DeclsDone,
• InstsDone))
• A makeOpcCall(N, Params)
• R getRate(N, C)
• S getNrOutputs(N)

Instance in middle of diagram
Satisfy instance inputs
26
SAOL Generator (7)
Code generation opcode result (3/3)

• if hasMultipleOutputs(N) or hasMultipleConnectio
  nsPerOutput(N)
• add(Code, makeAssignment(last(N), next(N)))
• add(Code, makeAssignment(next(N), A)
• if hasMultipleOutputs(N)
• add(Vars, makeArrayDecl(last(N), R, S))
• add(Vars, makeArrayDecl(next(N), R, S))
• return makeArrayLookup(last(N), getNumber(N,
  C))
• else
• if hasMultipleConnectionsPerOutput(N)
• add(Vars, makeVarDecl(last(N), R))
• add(Vars, makeVarDecl(next(N), R))
• return makeVarRef(last(N))
• else
• return A

27
SAOL Generator (8)
Generated SAOL code (reverb diagram)

• aopcode revbfilt(asig audio_in, ksig drywet, ksig
  fback_in,
• ksig level, ksig time_in)
• asig dly_next, dly_last
• dly_last dly_next
• dly_next delay(mixer(audio_in, level,
  dly_last, fback_in), time_in)
• return xfade(drywet, audio_in, dly_last)

28
The future

• FlowML needs input to mature and become really
  interesting
• Free software (e.g. FlowML parser, SAOL and
  CSound code generators, ) needs to be developed
• Serious authoring tools are needed to make Mpeg-4
  popular
• Artists need to make a difficult transition

29
Conclusion (1)

• FlowML might be interesting as a bridge between
  synthesis systems
• FlowML hides much of the engineering side of
  audio synthesis and provides a high-level
  description format for musicians
• FlowML allows to prototype a synthesizer quickly
  before optimizing it (e.g. compile to SAOL and
  hand-optimize)

30
Conclusion (2)

• The popularity of SAOL will depend on its support
  in commercially available software synthesis
  applications

31
References and contact info

• FlowML and QOrchestra
• http//www.bertschiettecatte.com/
• Mpeg-4 SA at the MLG, MIT Media Lab
• http//sound.media.mit.edu/mpeg4/
• Mpeg-4 SA book at Berkeley
• http//www.cs.berkeley.edu/lazzaro/