GDC 2005

1
New MIDI-Based Game Audio Techniques
Chris Grigg Beatnik Inc. Chairman, MMA Tech
Board chrisg_at_beatnik.com Tom Savell Creative
Labs Advanced Technology Center MMA Tech
Board tcs_at_atc.creative.com
2
Session Overview

• Whats MIDI?
• The Technology
• The Standards Orgs
• The New MIDI Standards
• iXMF Interactive XMF
• 3D MIDI Controllers
• High Definition Protocolfor MIDI Devices
• Questions

3
Whats MIDI?
Whats MIDI?
4
MIDI The Technology

• MIDI 1.0 over 5-pin DIN
• The Underlying Models
• The Recommended PracticesGM, Show Control, MTC,
  MMC, GM2, SP-MIDI, etc.
• The File FormatsSMF, Extended RMID, XMF, Mobile
  XMF
• Support in Computer OSsMac OS, Windows, BeOS,
  Linux etc.
• Support in Other Media StandardsMPEG-4 etc.
• Support in Audio Plug-In APIsVST, DX, AU etc.
• Support in Mobile Phones3GPP etc.

5
MIDI The Standards Orgs

• MMA MIDI Manufacturers Association (world
  except Japan)
• Tech Board
• Working Groups
• Mobile WG XMF WG
• DLS WG Transport Layer WG
• High-Def WG
• IASIG Interactive Audio SIG
• AMEI Assoc. of Musical Electronic Industry
  (Japan only)

6
The New MIDI Standards
The New MIDIStandards
7
iXMF Interactive XMF
1iXMFInteractiveXMF
8
iXMF Overview

• First open, non-proprietary standard for a file
  format for interactive audio content
• Uses MMAs XMF container technology
• Allows game developers to more easily create
  cross-platform titles
• Makes it easier for audio professionals to
  develop interactive content
• Developed by the iXMF-WG inside IASIG
• V1.0 spec about to be published
• Prominent early adopter

9
iXMF Motivation (1)
There IsNoStandard
Whats theIndustryStandard?
BuildProprietaryTool
Do ItRight NextTime
LimitedTime for Tool
Break theDepressing Circle!
LimitedToolQuality
-(
LimitedContentQuality
Harderto MakeContent
10
iXMF Motivation (1)
There IsNoStandard
Whats theIndustryStandard?
BuildProprietaryTool
Do ItRight NextTime
LimitedTime
LimitedToolQuality
-(
LimitedContentQuality
Harderto MakeContent
11
iXMF Motivation (2)

• Many companies continually re-creating the wheel
  in-house / proprietary
• Much of what they do is the same only the
  format is different
• This is redundant, parallel work gt no synergies
• User base (budget) is very small, so tool
  quality improves slowly (at best)
• Tool quality tends to limit IA content quality
• IA work done on one platform doesnt port to any
  other platform--gt Re-do for every platform--gt
  IA tends to be seen as cost-prohibitive--gt Less
  IA gets used

12
iXMF Process (Standardization)

• A Project BBQ idea
• An IASIG Working Group
• Linda Law is Chair
• Chris Grigg is Technical Lead
• Long Requirements phase
• Draft Specification
• Public Review period ended in May 2008
• Final edits to v1.0 spec under way now
• Release of v1.0 spec expected soon (October?)

13
iXMF Technical Design (1)

• The iXMF File
• Self-Contained IA Content Module
• Includes both Media and Adaptive Behaviors

14
iXMF Technical Design (2)

• Runtime Architecture
• iXMF File feeds IA Engine

Game
Middleware
Platform Provided
15
iXMF Technical Design (3)

• Cues soundtrack element objects
• iXMF file contains zero or more Cues
• Cue one soundtrack element a piece of music,
  a sfx, or a change to music/sfx
• Game requests sound/music element by starting a
  Cue
• by name startCue( mainTitlesMusic )
• by number startCue( kMainTitlesMusic )
• In iXMF file, Cue contains (or references)
• zero or more media Chunks
• some Scripts

16
iXMF Technical Design (4)

• Scripts methods / events of Cues
• Stored in iXMF file in portable binary format
• Variables iXMF Global / Cue class / Cue
  instance
• Expressions all the C operators
• User Subroutines call another user Script
• Built-In Statements
• Generic programming stuff
• Specialized IA primitives, to minimize the need
  for programming
• Extension Statements
• No user-defined Statements! Keep it simple
• Tools can hide Scripts behind GUI

17
iXMF Technical Design (5)

• Automatic Behaviors (Engine Model)
• Script hooks
• IxmfFileLoad, IxmfFileUnload
• CueStart, CueStop, CueCancel
• CommitPoint, ChangeOver
• Markers can call Scripts
• NextChunk chaining
• Mix Transitions crossfade, sync points/rules
• SyncGroups
• MixGroups
• Media LRU tracking
• Scripts can invoke game Callbacks
• Game can read/write iXMF variables

18
iXMF Technical Design (6)

• Custom Extensibility
• Add Script commands
• Opcode Spaces
• Add custom media formats
• Platform-specific audio formats
• Generative music
• Even video, graphics
• Extension Area in every iXMF data structure
• Community Extensibility
• Future Profiles of the v1.0 standard
• Future versions of the iXMF standard

19
iXMF Technical Design (7)

• Portability
• Cues Scripts will work properly on any iXMF
  player IF
• The media formats used are supported AND
• ( No extensions are used by content OR
  Any extensions used are supported OR Any
  extensions used arent critical )
• Profiles can manage portability expectations
• Content is often mastered for specific platforms
  portability not an issue there
• A high-res iXMF source file can give
  authoring-time portability
• Export to platform-specific runtime formats

20
iXMF The Ecosystem

• iXMF is non-proprietary
• iXMF is royalty-free to use or implement
• So Expect multiple runtime implementations,
  platforms, authoring tools --gt Ecosystem
• Commercial --gt Competition --gt Improvement
• Expect interoperability for base features /
  profiles
• Dont expect custom extensions to be portable
• Look for integration with game dev / media mgmt
  databases
• Hope that the IA community can start speaking a
  common language
• Expect vendors IASIG to be responsive
• Expect IA to become the norm, not exotic

21
iXMF The Future

• V1.0 Spec releasing soon. Then
• New Versions
• New Features
• New Media Types
• Profiles for specialization as needed e.g.
• Advanced Games
• Simple/Casual Games
• Mobile Phones/Games
• Musical Instruments
• DJ Applications
• Location-Based Entertainment
• Little-Endian
• Join the iXMF Working Group to participate!

22
3D MIDI Controllers
2 3D MIDIControllers
23
3D MIDI Overview

• Provides set of controllers for independent
  positioning on each MIDI channel
• Uses spherical coordinate system azimuth angle,
  elevation angle, and distance
• Seamless integration with legacy pan control
• Egocentric approach describes head-related 3D
  sources
• Well-defined standardized behavior
• Extensible through MMA standardization process
• V1.0 to be published soon.

24
3D MIDI Position Controllers

• Registered Parameter Number (RPN) Bank 0x3D
• Azimuth and Elevation Angular Resolution of 0.02
  degrees from -180 to 179.98 degrees
• Distance represented as ratio of distance to a
  maximum distance e.g. 0.000061 is 6.1
  centimeters if max. distance is 1km.

25
3D MIDI Pan Spread and Roll Angle

• Pan Spread defines the 3D angular distance of the
  MIDI stereo field represented by the legacy pan
  controller
• Roll Angle defines the angle relative to
  horizontal of the MIDI stereo field represented
  by the legacy pan controller

26
3D MIDI Position Model
27
3D MIDI Gain Parameters

• Gain parameters provide content author control
• Gain
• Independent of CC7/11
• Gain at max distance
• Gain when distance ratio is 1.0
• Reference distance
• No additional gain below this distance ratio

28
3D MIDI Distance Model
29
3D MIDI Ex. Distance of a Fly

• The buzz of a fly would typically not be heard
  beyond 10 meters away
• maximum_distance 10
• Sounds significantly louder a few centimeters
  away from your ear
• reference_distance_ratio 0.01 -- between 0 and
  10 cm, the fly sound is not attenuated

30
3D MIDI Future Enhancements

• Reverberation Controls
• Source / Reverb mix dependency on distance.
• Environmental effects (e.g. occlusion)
• Could be made compatible with IA-SIG I3DL2
  guideline
• Subject to MMA Standardization Process

31
High Definition Protocol for MIDI Devices
3 HD ProtocolforMIDI Devices
32
HD-PMD Introduction

• Whole new generation of the basic MIDI message
  protocol
• Address shortcomings of the MIDI 1.0 protocol
  (1983)
• Grow the worldwide MIDI markets by opening up
  new functionality,new musical expressiveness
• HD work started in 2006 by MMA Tech Board
• Standardization work still under way
• Overview Only! All Technical Details Subject to
  Change!

33
HD-PMD Motivation

• MIDI 1.0 Technology Needs an Update
• MIDI 1.0 first released in 1983
• HW transport slow / Timing not tight
• Complicated msg structure / Parsing
• Many multi-message functions
• Message size varies
• Byte aligned
• Hardware orientation --gt Hard to write SW for
• Advanced features difficult to access
• Expansion room exhausted long ago
• Desired New Functionality Includes
• Better support for non-keyboard instrument models
  (strings, voices, horns, winds etc.)
• Better support for non-western music
• Better support for instruments with continuous
  pitch / very fine pitch nuance

34
HD-PMD Standards Process

• MMA Working Group in US/Europe
• AMEI Working Group in Japan
• Joint Parallel technical work
• Weekly WG teleconferences
• WG email reflector
• MMA/AMEI coordination via email
• MMA/AMEI Joint Face-to-Face meetingsin LA / SF /
  Tokyo 2-3x per year

35
HD-PMD Technical Design

• Transports
• An ongoing discussion
• Traditional wired transports
• Software-only transports
• Network/comm protocols
• Plug-In APIs
• OS MIDI services APIs
• Messages
• Significantly reconfigured
• Simplicity
• Future extensibility
• New musical functionality

36
HD-PMD Messages

• Uniform Packet 64/128 bits (8/16
  bytes)Simplifies transports - Memory
  efficient(MIDI 1.0 has 1, 2, 3, or N-byte
  messages)
• Message 1 or More Packets
• Single-Packet Messages (SPM)
• Multi-Packet Messages (MPM)
• Fixed of Packets
• Variable of Packets (VLM)

37
HD-PMDOpcode SpacesCompared

• MIDI 1.0
• Totally full.
• Every Status Nybbleeither in use, or else
  officially deprecated
• Extensions very cumbersome
• HD is shown at right
• Lots of expansion room
• Especially with sub-commands

38
HD-PMD New Functionality (Proposed Under
Discussion)

• Notes
• More Velocity resolution (16-bit 512x MIDI 1.0
  depth)
• Precise Pitch option (1/256 semitone resolution)
• Trigger Index option (like MIDI 1.0 Note Number)
• Controllers
• More Value resolution (16/32-bit 2-4x MIDI 1.0
  depth)
• More Controller Numbers available (32x MIDI 1.0
  space)
• No more MSB / LSB / Data Entry!
• CC is a unified message, including Bank number
• RPN now just another controller number
• NRPN now a unified message all MIDI 1.0 numbers
  available
• Per-Channel and Per-Note Controllers, RPNs, and
  NRPNs
• Addressing
• More Channels (7 bits - 8x MIDI 1.0 space)
• Groups concept - Addl dimension of addressing
  (5 bits - effectively 256x MIDI 1.0 channel
  space)

39
HD-PMD Whats Next?

• More joint MMA/AMEI Meetings
• Draft v1.0 Specification (English) is in
  progress
• Experimental implementations of candidate
  transports ongoing
• Decide on v1.0 Spec(s) release date(s)
• Decide on possible additional Recommended
  Practices for HD
• After v1.0 Spec Release
• Continuing process to standardize new features
  forever and ever and ever

40
Questions?
Questions?
41
Answers!

• YES you can join the iXMF Working Group
• YES you can get a copy of the v1.0 iXMF Spec
  soon
• at www.iasig.org
• YES you can get a copy of the 3D MIDI Spec soon
• YES if you are commercially involved in designing
  or producing MIDI HW or SW you can join the MMA
  and the MIDI High Def Working Group
• at www.midi.org

42
(You just sat through)New MIDI-Based Game
Audio Techniques
Chris Grigg Beatnik Inc. Chairman, MMA Tech
Board chrisg_at_beatnik.com Tom Savell Creative
Labs Advanced Technology Center MMA Tech
Board tcs_at_atc.creative.com