Improvising with Computers: A Personal Survey 19892001

1
Improvising with Computers A Personal Survey
(1989-2001)

• Sergi Jordà
• Music Technology Group
• Audiovisual Institut
• Pompeu Fabra University
• Barcelona, Spain
• sergi.jorda_at_iua.upf.es
• http//www.iua.upf.es/sergi

2
Summary

• Improvisation Concepts Ideas
• In Search of New Computer Music Improvisation
  Paradigms (3 topics)
• Three Previous Works (89-98)
• FMOL (97-2001)

3
Part I Concepts Three Topics
4

• Improvisation Different meanings, different
  rules (baroque, bebop, flamenco)
• ? Improvisation as Instant Composition
• (no matter how free or constrained)
• Improvising with Computers
• 1960s Interactive systems using
  computer-controlled analog synthesizers
• 1970s Real-time algorithmic composition systems
  (D.Behrman, J.Chadabe, S.Martirano, G.Mumma,
  L.Spiegel)
• 1980s-1990s MIDI standardization MAX
• However, still a burgeoning an underexplored
  area

5

• In search of new models and paradigms for
  interactive music making we cannot ignore
  existing models of (computer and non
  computer-based) improvisation and music making
• Multidisciplinary knowledge
• Controller interfaces design
• Real-time sound synthesis and processing
  techniques
• Cognitive science
• Music theory
• Algorithmic composition techniques
• Traditions
• should preserve good achievements
• but should not slow down desirable changes

6
Three possible (and personal) topics

• Do not always separate controllers generators
• Try to avoid the distinction between sound
  music (micro vs. macro levels)
• Think about Individual vs. Collective Performance
  Dilettante vs. Professional Performers

7
Controllers and Generators (1/2)

• The separation brought by MIDI ?
• creation of many new controllers, but ...
• Traditional instruments are both controllers and
  generators
• It is not possible to develop sophisticated and
  efficient controllers independently from their
  associated sound or music generators

8
Controllers and Generators (2/2)

• Bidirectional mappings (e.g. haptics and other
  possible feedback loops)
• Traditional instruments resonate (i.e. they are
  conscious of the sound they produce)
• In spite of audio analysis or machine listening
  techniques, computer music controllers do not
  usually know their output
• These statements are not invalidating low-cost
  and widely available input devices (mice,
  joysticks), but try to search for smarter
  mappings and wider communications between the 2
  systems

9
Controlling at Macro and/or Micro Levels (1/2)

• Western music always ignored the notes inside
• However many improvisation idioms keep a good
  balance between sound and form (e.g. growls in
  free jazz sax, melismas in flamenco singers)
• Computer music still separates both levels (e.g.
  Music-Ns score orchestra)
• Now that real-time synthesis has become widely
  available and highly customizable (virtual
  synths), this separation can be a burden and an
  anachronism
• Many composers (I.Xenakis, G.Scelsi, J.Tenney,
  C.Roads) have addressed this problem (none are
  improvisers, though)
• UPIC, PulsarGenerator, SuperCollider, KYMA
• L. Spiegels Music Mouse (both an instrument and
  a composing tool)

10

Controlling at Macro and/or Micro Levels (2/2)

• Wouldnt it be possible to design performing
  systems that could deal with both levels as a
  whole ? (i.e. controlling sound and form with the
  same tools and gestures)
• Wouldnt that bring new dimensions to real-time
  computer music creation?

11
Individual vs. Collective Instruments

• Collective (distributed) instruments poorly
  explored in traditional music
• Computer Music networks by the League of
  Automatic Composers or the Hub (Bischoff, Gold
  Orton)
• Interactive Sound Installations ?

12
Dilettante vs. Professional Performers

• Different users, dif. skills, dif. Needs
  (simplicity vs. freedom), but
• Are simplicity and freedom incompatible?
• Is Low entry fee with no ceiling on virtuosity
  (David Wessel, Matthew Wright) an impossible
  dream?

13
Part II Three Previous Works

• PITEL (1989-91)
• QWERTYCaster (1996)
• Afasia (1998)

14

• PITEL (1989-91)
• Played 4 interdependent MIDI voices while
  listening to 2 monophonic inputs
• Zero musical knowledge
• High-level parameters controlled by a
  mouse-conductor
• Lack of serious interface design

15
Based on two-term non-linear feedback
relations xa,i f (xa,i-N, xb,i-k)
16
The Low-Tech QWERTYCaster (1996)

• Cheap and simple design and construction
• Fast output (vs. Pitels slow algorithmic
  approach)
• Like a trad. instrument ? focus only in sound
  control (not macro formal control)

17
Afasia (with Marcel.lí Antúnez) (1998)

• One man show multimedia version of Homers
  Odyssey
• The performer controls with his movements
• A mechanical 4-robots orchestra (70-finger
  electric guitarbass, one-string violin, drumkit,
  three-bagpipe horn section)
• 1 sampler
• 1 CD-Audio
• 3 audio effects racks
• 1 MIDI controlled Yamaha Promix mixer
• Interactive multimedia animations
• 1 DVD
• A DMX light table
• Video projector input-switcher (between SVGA and
  DVD)

18
Afasia Robot Quartet
Each robot has its own virtual MIDI-driver
bridge MIDI messages generated by the interactive
software with the digital output cards that
control the relays and the pneumatic mechanisms
(and each driver has several MIDI channels).
19
(No Transcript)
20
Afasia Interactivity

• Limited semantics of sensors employed (gloves,
  buttons, potentiometers in each of the
  performers articulation and mercury switches in
  his extremities) ? score-driven interactive
  model each island is a state with a particular
  behavior and mapping
• Format 1 Standard MIDI Files expanded with 40
  text meta-events tell the custom sequencer how
  to process the data according to the performers
  inputs

21
Afasia MIDI files structure

• Each part (or island) of the show is associated
  with one Standard MIDI file.
• Each MIDI file is made of any number of blocks
  (an Afasia concept not present in standard MIDI
  files).
• A block is a group of sequencers tracks that
  behaves like a permutable section of a score,
  only one block being active at any time.
• Each block is made of a special control track and
  any number of conventional MIDI tracks.
• Control tracks only contain text meta-events that
  indicate how to interact with the other blocks
  tracks, alter the block structure or jump to
  other blocks.
• Conventional tracks can also contain text
  meta-events that indicate how to interact or
  modify the MIDI data inside the track.
• As conventional sequencer tracks, each track
  (except for control tracks) is directed to a
  specific port or device, and to a specific
  channel within that device.
• Six MIDI ports are used in Afasia (a custom port
  for each of the four robots, the standard
  internal MIDI port for controlling the soundcard
  sampler, and the standard external MIDI port for
  controlling the audio mixer and the three effects
  processors).
• Each device can have several MIDI channels (e.g.
  the electric guitarbass robot has one channel
  for each of its strings, while the bagpipe
  section MIDI port uses three channels, one for
  each bagpipe).

22
Afasia Meta-events Interactive MIDI

• COMMAND_NAMEparam1,param2,,paramN ? e.g.
  TRANSPOSE4,3,12
• Some possibilities
• Switch between blocks of tracks
• Mute/unmute tracks
• Transpose tracks
• Loop sequences - modify their length
• Modify current play position
• Modify any MIDI control (by value, by increment,
  by random range)
• Quantize/delay tracks
• Change tempo
• Define chords or scales (for correcting generated
  notes)
• Gestures and solo modes

23
(No Transcript)
24

• These previous works do not follow many of the
  three initial statements
• PITEL controls only form (not sound), lacks
  serious interface design and is thought for only
  one advanced user (myself)
• The QWERTYCaster controls mainly sound, and is
  also designed for myself
• AFASIA uses generic controllers (not specific to
  the instruments they control) and is also
  designed for a trained user
• They are better attained in my next work, FMOL

25
Part III FMOL (F_at_ust Music On Line) 1997-2001

• Project for la Fura dels Baus
• Primary goals
• Collective composition on the Net
• Introduce newcomers into experimental electronic
  music
• Cheap (free) and available (no special hardware)
• Sound over notes priority (micromacro control
  and no General MIDI !)
• Attractive to both trained and non-trained
  electronic musicians (i.e. intuitive but
  intricate...)
• ?Mouse driven client software for RT synthesis
  and RT composition with peculiar visual
  feedback interface

26
FMOLs Interface main characteristics

• Development parallel to the sound engine
• Audiovisual feedback
• Music needs time ? visualization needs animations
• Non-intelligent (no indirect mappings)
• Almost all functionalities are visible (e.g.
  D.Norman scissors)

27
FMOLs Engine

• 6 audio channels (independent or not)
• For each channel 1 generator 3 processors
  (selectable from more than 100 algorithms)

• For each generator-processor 4 LFOs
• For each LFO, dynamic control on frequency,
  amplitude shape (sin, square, saw, triangle,
  random)
• LFOs are fundamental for all time evolution (no
  use of prerecorded sequences)

28
(No Transcript)
29
FMOL Configuration window and graphical
interface Synthesis instruments / Sampler
instruments / Filters / Processing Instruments
(2222)
30
FMOL 1.0 main algorithms and their two primary
parameters
31

• FMOL (2.0) keys allow to
• Sustain strings 1-6 (already in 1.0)
• Mute/unmute strings Z-N (already in 1.0)
• Record and retrieve mouse gestures A,S
  (already in 1.0)
• Record D and retrieve up to 8 snapshots F-Ç
• Change LFO wave TAB,CAPS,SHIFT,CTRL
• and more in FMOL 3.0 (private FMOL trio version)
  .

• Other FMOL possibilities Play, record and
  overdub files, Upload and download files, Render
  to audio

32
FMOLs Musical Social Implications

• More than 1,100 pieces by more than 100 authors
  (Jan-April, 1998)
• More than 600 pieces (September 2000)
• Used in one play (F_at_ust 3.0, 1998) and one opera
  (DQ, 2000) by la Fura dels Baus
• Collective CD published in 1998
• Workshop for visual artists (Lisbon, 2000)
• FMOL Trio (1999-) Improvised electronic music
  visuals (2 live CDs released)

33
The FMOL Trio
The only instrument I designed for everybody is
now the only one I personally use live Cristina
Casanova (FMOL), Pelayo Arrizabalaga (bass
clarinet, alto tenor saxes, scratch turntables)
and Sergi Jordà (FMOL) Free-form improvised
music, while 2 projectors connected to each of
the computers provide complementary visual
feedback to the audience ? giving the public a
deeper understanding of the ongoing musical
processes, and adding exciting elements to the
show
34
Collective Composition

• Collective creation and the production of open
  and continuously evolving works are two of the
  most appealing artistic breakthroughs the
  Internet can offer to music composers and
  creators in general.
• FMOL 1 (1998) 2 (2000) no Real-Time
  collaboration
• Possible non RT Collaboratiev options
• Free
• Horizontal (i.e. exquisite-corpses)
• Vertical (multitrack-overdub) ?
• New composers can add new layers, and
  process/distort existing ones
• (they can also start new pieces from scratch)
• ? Tree like database for storing uploaded pieces
  (synthesis engine and GUI are also very tightly
  related to this concept)

35
(No Transcript)
36
(No Transcript)
37
Net-jamming

• Allow real-time multi-user interaction
• Constraints imposed by current internet
  technology
• High latency will cause delays of 100ms to music
  played on a computer and listened on another
• Lack of a global reliable synchronism reference
  for the distributed synthesis engines
• Due to the timbrical nature of the synthesis
  engine, FMOL music has a high amount of
  robustness towards the internet constraints
• (Atau Tanaka Internets latency can be seen as
  cyberspaces acoustic)
• First experimental concerts between
  Barcelona-Dresden and Barcelona-Berlin, October
  2001 (

38
FMOL net-jam session server (2002)

• Asynchronous and multipoint real-time messaging
  server (probably based on Phil Burks Transjam
  protocol)
• Server listens to periodically incoming messages
  from the client
• At every frame (48times/sec) each client sends
  the generated events to the server
• Real-time typical data rates of 60-180
  bytes/second
• Server redistributes all the generated messages
  (except clients own) to all the clients at same
  framerate
• ?Each client listens to a slightly different mix

39
Future Work

• Next Internet version will allow real-time
  collaboration (jamming)
• King-size concert version with sensor control and
  video tracking, playing with the hands (instead
  of mouse) over a 3x2m retro projected screen

40
FMOL Conclusions

• Good example of low entry fee with no ceiling
  for virtuosity instrument
• Good example of controller-generator parallel
  design
• Conceptually macro-micro approach
• although technically restricted by
  resolutions
• internal use of MIDI ? 7-bit
• frame rate ? 20-50 fps
• ? more midi than micro

41
On-line Additional Resources

• Audio and video excerpts for this paper
• http//www.iua.upf.es/sergi/download/chi2001/
• FMOL main page (download and info)
  http//www.iua.upf.es/sergi/FMOL
• FMOL-DQ Internet composition project
• http//teatredigital.fib.upc.es/dq
• FMOL Trio http//www.iua.upf.es/sergi/FMOL/fmolt
  rio/