Music%20Notation,%20Music%20Representation,%20AND%20Intelligence

1
Music Notation,Music Representation,ANDIntellig
ence

• Donald Byrd
• School of Music, Indiana University
• 3 February 2005
• minor rev. 5 February

2
Overview

• Music representations from abstract to concrete
  (notation)
• Try to assume just the right things e.g., no
  knowledge of music or notation
• For motivation, focus on real-world content-based
  music-IR situations
• Organization of the Talk
• I. Motivation Why is this Important and/or
  Interesting?
• II. Representation and Semantics
• III. Music Notation, Representation, and
  Intelligence
• IV. Conclusions

3
You Are Here

• I. Motivation Why is this Important and/or
  Interesting?
• II. Representation and Semantics
• III. Music Notation, Representation, and
  Intelligence
• IV. Conclusions

4
Audio-to-Audio Music Retrieval

• Shazam - just hit 2580 on your mobile phone and
  identify music
• Query
• Match
• Fantastically impressive to many people
• Have they solved all the problems of music IR?
  No, (almost) none!
• Reason intended signal match are identical gt
  no time warping, let alone higher-level problems
  (perception/cognition)

5
Similarity Scale for Content-Based Music IR

• Relationship categories describing whats in
  common between items whose similarity is to be
  evaluated (from closest to most distant)
• For material in notation form, distinctions
  among (1), (2), and (3) dont apply its just
  Same music, arrangement

• 1. Same music, arrangement, performance,
  recording (Shazam)
• 2. Same music, arrangement, performance
  different recording
• 3. Same music, arrangement different
  performance, recording
• 4. Same music, different arrangement or
  different but closely-related music, e.g.,
  simpler variations (Mozart, etc.), minor revs.
  (OMRAS, etc.)
• 5. Different less closely-related music freer
  variations (Schumann, etc.), extensive revisions
  (AI)
• 6. Music in same genre, etc. (AI?)
• 7. Music influenced by other music (AI!)

6
OMRAS Polyphonic Audio Music IR A Task that
Needs Note Representation

• Started with recordings of Bach preludes and
  fugues
• Did polyphonic (several notes at once) music
  recognition
• Polyphonic audio -gt events is an open research
  problem
• Converted results to MIDI, used as queries
  against database of c. 3000 pieces in MIDI form
• One of worst-sounding cases Prelude in G Major
  from Well-Tempered Clavier, Book I
• Outcome the actual piece was ranked 1st!
• Models built from notation database, but note
  data only

• Query (audio -gt MIDI -gt audio)
• Match (original audio recording)

7
Basic Representations of Music Audio
Audio (e.g., CD, MP3) like speech
Time-stamped Events (e.g., MIDI file) like
unformatted text
Music Notation (sheet music) like HTML text
8
Basic Representations of Music Audio

• MIDI Musical Instrument Digital Interface
  simple, very standard low-bandwidth protocol
  (from early 1980s)

• Audio Time-stamped Events Music Notation
• Common examples CD, MP3 file Standard MIDI
  File Sheet music
• Unit Sample Event Note,
  clef, lyric, etc.
• Explicit structure none little (partial voicing
  much (complete
• information) voicing information)
• Converting to form with less explicit structure
  (to left) moderately difficult
• Converting to form with more explicit structure
  (to right) very difficult

9
Music-IR Problems that Needs More Structure

• Joan Publics problem find a song, given some of
  the melody and some lyrics
• Needs notes and text (lyrics)
• Common question for music librarians, esp. in
  public libraries
• Musicologists problem authorship/origin of
  works in manuscripts
• Full symbolic data is important, even
  insignificant details of notation (John Howard)

10
You Are Here

• I. Motivation Why is this Important and/or
  Interesting?
• II. Representation and Semantics
• III. Music Notation, Representation, and
  Intelligence
• IV. Conclusions

11
Representation, from Abstract to Concrete

• Cf. Basic Representations of Music Audio
• Abstract represention semantics only
• Intermediate syntax (mapping rules)?
• Concrete
• for use by computers encoding
• for use by humans if visual, notation (involves
  graphics and/or typography)
• Analogous to knowledge representation vs. data
  structure

12
Semantics in Music

• Denotation (explicit, well-defined)...
• vs. Connotation (implicit, ill-defined)
• In text
• Two definitions of pig
• 1. Ugh! Dirty, evil-smelling creatures, wallowing
  in filthy sties! (Hayakawa)
• 2. Mammal with short legs, cloven hoofs, bristly
  hair, and a cartilaginous snout used for digging
  (Amer. Heritage)
• Prose is mostly denotation
• Poetry is art gt connotation much more important
• Music is always art, only connotation!
• Major issue for content-based music IR

13
From Representation to Notation

• Choosing a representation inevitably introduces
  bias
• Given a representation, choosing notation
  inevitably introduces more bias
• Important to consider the purpose (R. Davis et
  al Wiggins et al)
• For huge body of important music, we have no
  choice notation is CMN (Conventional Music
  Notation)!
• Really CWMN (W Western)
• Alternative for some music tablature (guitar,
  lute, etc.)
• CMN is among the most successful notations
  ever...
• but enormously complex and subtle

14
Notation Says Much about Representation

• CMN standard for Western music after c.1650
• Evolved for classical music, but heavily used
  for very wide range (pop, jazz, folk, etc.)
• Composers/arrangers/transcribers have pushed it
  hard gt reveals things about music representation
  in general
• Will concentrate on notation (CMN)

15
You Are Here

• I. Motivation Why is this Important and/or
  Interesting?
• II. Representation and Semantics
• III. Music Notation, Representation, and
  Intelligence
• IV. Conclusions

16
How to Read Music (CMN) Without Really Trying
The Basics

• Four basic parameters of a musical note
• 1. Pitch how high or low sound is
• 2. Duration how long the note lasts
• 3. Loudness perceptual analog of amplitude
• 4. Timbre or tone quality
• Above in decreasing order of importance for most
  Western music
• Principles of CMN ( e 1)
• 1. Pitch on vertical axis clef gives offset
  (zero)
• 2a. Duration indicated by note/rest shapes
• 2b. Start times (sum of durations in the voice)
  on horizontal axis
• 3. Loudness indicated by signs like p , mf , etc.
• 4. Timbre indicated with words like violin,
  horn, pizzicato

17
Why is Musical Information Hard to Handle?

• 1. Units of meaning not clear anything in music
  is analogous to words (all representations)
• 2. Polyphony parallel independent voices,
  something like characters in a play (all
  representations)
• 3. Recognizing notes (audio only)
• 4. Other reasons

18
Units of Meaning (Problem 1)

• Not clear anything in music is analogous to words
  
• No explicit delimiters (like Chinese)
• Experts dont agree on word boundaries (unlike
  Chinese)
• Are notes like words?
• No. Relative, not absolute, pitch is important
• Are pitch intervals like words?
• No. Theyre too low level more like characters
• Are pitch-interval sequences like words?
• In some ways, but
• Ignores note durations
• Ignores relationships between voices (harmony)
• Probably little correlation with semantics

19
Independent Voices in Music (Problem 2) ( e 2)
J.S. Bach St. Anne Fugue, beginning
20
Independent Voices in Text

• MARLENE. What I fancy is a rare steak. Gret?
• ISABELLA. I am of course a member of the / Church
  of England.
• GRET. Potatoes.
• MARLENE. I havent been to church for years. / I
  like Christmas carols.
• ISABELLA. Good works matter more than church
  attendance.
• --Caryl Churchill Top Girls (1982), Act 1,
  Scene 1

Performance (time goes from left to right)
M What I fancy is a rare steak. Gret? I
havent been... I I am of course a member of
the Church of England. G Potatoes.
21
Complex Notation Multiple Voices ( e 3)

• Multiple voices on a staff rapidly gets worse
  with more than 2 (Telemann Liebe, Liebe)
• 2 voices in mm. 5-6 not bad stem direction is
  enough
• 3 voices in m. 7 notes must move sideways
• 4 voices in m. 8 almost unreadablewithout
  color!
• Still acceptable because specific voice is
  rarely important

22
Problems Example 1 (superficial but interesting)

• Ravel work has slur with 7 inflection points
• Impressive, but complexity is purely graphical
• No big deal in terms of representation
• but influence of performance on notation is
  revealing

23
Duration and Higher-Level Concepts of Time

• Schubert Impromptu ( e 4)
• Measures everything between barlines
• Time signature 3/4 3 quarter notes per measure
• Triplets 3 notes in the time normally used by 2
• General concept is tuplets

24
Problems Example 2 (Deep)

• Chopin Nocturne has nasty situation ( e 5)
• One notehead is triplet in one voice, but normal
  duration in another
• Semantics (execution) well-defined, obvious
• Note starts 1/16 before barline
• But also (2/3)(1/16) before barline! How to
  play?
• Reason musical necessity
• Solution for performer rubato
• Solution for music IR program ?

25
Problems Example 3 (Medium)

• Bach time signature change in middle of measure
• ( e 6)
• Semantics well-defined and obvious
• Measure has duration of 18 16ths
• But not until the middle of the measure!
• How does this make sense?
• Triplets express same relationship as equivalent
  simple/compound meter
• Invisible (unmarked) triplets
• Cf. Bach Prelude two time signatures at once (
  e 7)
• Reason avoid clutter

26
Problem 4 (Medium)

• Brahms Capriccio ( e 8)
• Time signature 6/8 gt measure lasts 12 16ths
• A dotted half note always lasts 12 16ths
• but here it clearly lasts only 11 16ths!
• Reason avoid clutter

27
Two Ways to Have Two Clefs at Once

• Clef gives vertical offset to determine pitch
• Debussy ( e 9)
• Bizarrely obvious something odd involving clefs
• Ravel ( e 10)
• Only comparing time signature (3/8) and note
  durations makes it clear both clefs affect whole
  measure
• Reason save space (by avoiding a 3rd staff)

28
Surprise Music Notation has Meta-Principles! (1)

• 1. Maximize readability (intelligibility)
• Avoid clutter Omit Needless Symbols
• Try to assume just the right things for audience
• Audience for CMN is (primarily) performers
• General principle of any communication
• Applies to talks as well as music notation!
• Examples Schubert, Bach, Brahms

29
Surprise Music Notation has Meta-Principles! (2)

• 2. Minimize space used
• Save space gt fewer page turns (helps performer)
  also cheaper to print (helps publisher)
• Squeezing much music into little space is a major
  factor in complexity of CMN
• Especially important for music real-time, hands
  full
• Examples Telemann, Debussy, Ravel

30
The Rules of Music Notation

• Tempting to assume that rules of such an
  elaborate successful system as CMN work
  (self-consistent, reasonably unambiguous, etc.)
  in every case
• But (a) rules evolved, with no established
  authority (b) many of the rules are very
  nebulous
• In common cases, there's no problem
• If you try to make every rule as precise as
  possible, result is certainly not self-consistent
• Trying to save space makes rules interact
  something has to give!

31
Music Notation Software and Intelligence

• Despite odd notation, really nothing strange
  going on in almost all of these examples
• Ravel slur, Debussy Ravel 2 simultaneous clefs,
  Bach Schubert invisible triplets, Brahms
  short dotted-half note, Telemann 4 voices/staff
  are all simple situations
• Chopin Nocturne is complex
• Programmers try to help users by having programs
  do things automatically
• A good idea if software knows enough to do the
  right thing almost all the timebut no program
  does!
• Notation programs convert CMN to performance
  (MIDI) and vice-versa gt requires shallow
  semantics makes things much harder

32
You Are Here

• I. Motivation Why is this Important and/or
  Interesting?
• II. Representation and Semantics
• III. Music Notation, Representation, and
  Intelligence
• IV. Conclusions

33
Conclusions Review (1)

• Representations express Semantics
• Semantics of Music Denotation Connotation
• Principles of CMN
• Meta-Principles of CMN
• 1. Maximize readability Omit Needless Symbols
• Try to assume just the right things for audience
• General principle of any communication
• 2. Minimize space used
• Save space gt fewer page turns, less paper

34
Conclusions Review (2)

• We need CMN or equivalent to solve spectrum of
  music-IR (and other music-IT) problems
• But CMN cant represent everything we want
• Even when it can, may not, at least explicitly
• Need high-level intelligence to interpret
• Solution unknown
• Likely to require major funding -)

35
Conclusions Why is this really Important and/or
Interesting?

• Some problems directly related to other areas of
  informatics
• Example Approximate string matching in
  bioinformatics
• Encourages progress on real semantics
• Connotation is an important part of meaning in
  everything
• Can often ignore, but any semantics in arts
  forces you to deal with connotation
• Music is at least as quantifiable as any art, so
  likely to be more tractable than others!

36
You Are Here

• The End