Jack

1
Jack Jill

• AI composition/listening system in progress

2
Differences between AI composing and

• Algorithmic composing
• Transformation of input to output (could be
  random numbers) that is decoded by a human
  listener as music.
• Xenakis, Gendy
• Outcome/performance algorithm
• Use templates/rules/heuristics to construct/limit
  music (eg, constrain proportion of steps/leaps,
  adapt 18th C. theorists, use statistical models
  of existing music)
• Band in a box
• Genetic algorithm with fitness measure
• Develop/learn interestingness rules
• User response model
• Learns to play like you. All the time,
  continuously.
• Machine learning
• Pachet, Continuator
• Process model

3
Theorizing what music sounds like

• In principle, necessary to theorize the affective
  feel of music
• Category problem Does it sound like music?
• Expectation model does it violate expectations
  we have about how it should/should not go?
• Boredom/anxiety model optimal complexity
  theory of Berlyne --- too much or too little?
• Shape/pattern model detailed analysis of
  patterned Gestalts ( wholes) can answer many
  of these questions

4
The shape of music

• Shape Gestalt ( Prägnanz )
• Intuitive concept informally used by many music
  theorists (L. B. Meyer, C. Rosen) without
  explication
• I recognized the shape of your voice
  something specific and particular, yet similar to
  many other things
• Easily recognized in many different contexts
• Figure-ground separation as a condition of
  objecthood (Kubovy)
• Much research on grouping, but no real
  shape-psychology of music

5
Gestalt theory approach
Major idea given a variety of possible
groupings, humans prefer the simplest (e.g.,
Lehrdahl Jackendoff).

• The perceived perceptual description is B.
• Aesthetic problem does that make C more
  interesting?

6
Patterns of shape

• Intuitively, shapes are those things in music
  that we easily take in as a whole, and which we
  almost cant fail to notice
• The complexity of perceived shape is probably
  limited
• The simplest shapes are the simplest patterns
• Basic idea everything in music is under the
  control of underlying simple patterns
• In order to theorize patterns, we need a theory
  of pattern simplicity.
• Structural information theory offers ideas.

7
Structural Information Theory (Leeuwenberg, 1971)

• Computational theory of how to recognize pattern
  simplicity supported by empirical studies
• Takes 1D strings as input, but these can be used
  to represent higher-dimensional objects
• Three kinds of codes Iteration, Symmetry,
  Alternation (recursively applicable)
• Patterns are expressed as codes (eg, aaaa -gt
  Iterate (a, 4) )
• One pattern generates many codes (eg, aaaa -gt
  Symmetric (aa) )
• Minimum principle The simplest code reflects the
  preferred perceptual organization
• Straightforward complexity metric The number of
  primitive elements in the code (parameter load)
  is a measure of simplicity
• SIT can consider context.

8
Structural Information Theory, cont.
(Leeuwenberg, 1971)

• Example. Iterate (a, 4) has one primitive element
  (a)
• Symmetric (aa) has 2
• So Iterate is simpler.
• Problem with this ahead
• Tractable solutions exist. Finding the simplest
  code seems to mean generating all possible codes
  and applying a complexity metric a
  combinatorial explosion. However, Helm
  Leeuwenberg (1985) provide an algorithm that
  solves for simplest code using graph theory.
• Further, an deterministic algorithm to solve
  analogies using SIT has been developed (Dastani,
  2003).
• SIT therefore provides a powerful
  mathematically-anchored basis for reasoning about
  music computationally.

9
Structural Information Theory is not a theory of
music

• Unfortunately the proposed coding language is
  without doubt inappropriate for music. This can
  be made clear by looking at the structures of
  some famous melodies from the viewpoint of the
  proposed language no aesthetic principles
  emerge. (Leeuwenberg, 1971)
• Musical problem human propensity to (e.g.) group
  things in pairs does not recommend straight SIT.
• But this can be addressed.

10
Organization of this talk

• Shape
• Shape analysis
• An example of a shape schema
• Pattern
• Constructing variations of folksongs
• Constructing simple canonical variations

11
Shape analysis

• the way things go up and down
• the way things parallel one another relative to
  up and down in whole or part -- patterns
• the way in which a rhythm moves motion
• The way an object moves against a previous
  context variation
• the rhythm of old new -- grow
• the rhythm of excitation refraction
• the patterns of patterns of (recursive
  application)
• Recursively the sense of an underlying shape
• ( etc.)
• Shapes are therefore about motion, pattern,
  growth, variation and other things. Shapes give
  us a sense of wholeness and of particularity.
• Those things we hear at a glance.

12
Shapes are schemas

• Bartlett (1932), Piaget, Axelrod (1973), Drescher
  (1991), Bregman
• A preexisting assumption about the way the world
  organized (Axelrod)
• Context, action, result (Drescher)
• Schemas are predictive if a context is
  satisfied, and the schemas action is taken, then
  a certain result is expected.
• s??µa form, figure, appearance (Liddel Scott)

13
An example Buckle schema

• One, two, buckle my shoe
• A sequence of 4 things where the penultimate
  thing is more energetic than the 2 preceding
  things, and which stops at the 4th thing. (In
  Jj-speak slams)
• Buckle is about the shape of excitation and
  refraction
• Some simple examples

14
Buckle is also a form
15
Buckles make nice themes

• Bach, violin concerto in a mi.

16
Buckles can be used recursively
17
Buckles can be specialized
18
Shapes can be algebraically typed

• Obviously, there is a limit to the number of
  shapes we want to explicitly name
• An algebraic description system is needed.

19
Buckles are important for analysis

• Buckles other high-level rhythmic shapes are
  part of an overall strategy to discover metrical
  (submetrical/hypermetrical) chains.
• No current method exists for detecting metrical
  changes
• Without top-down schemas, metrical analysis in
  complex music is hopeless.
• Without metrical analysis, it is impossible to
  make valid inferences about what kind of
  structure a listener may be aware of

20
Buckles are interesting

• internal use of figure-ground effect helps
  provides a certain kind of Prägnanz (
  goodness)
• suggests our experience of the physical world.
  Its like a batter winding up and letting go a
  burst of energy that connects with the ball,
  making a grand slam.
• seems more rooted in brain adaptation to physical
  reality (Shepherd) than to styles probably can
  be found in nearly all music, including
  avant-garde music.
• connects to all situations where penultimate
  object maximizes energy/excitation is absorbed
  in a refractory object where the music comes to a
  halt buckle is a kinematic shape.

21
Kinematic Schemas

• Patterns of motion that tell us something about
  the discharge of energy
• Describes situation that can be occur at many
  different levels
• Energy schemas are needed if we want to calculate
  how one shape influences the energy of the next.

22
How Jack uses this

• Jack constructs variations that are based on an
  analysis of shape
• Kinds of variation specific to buckleslam make
  the buckle more or less intense discover what
  has already been done in parallel situations
  create a new pattern
• The new pattern could be derived from an old
  pattern

23
Constructing simple variations of folksongs using
pattern analysis

• Vary a few notes in a way that is sensible
• Vary the rhythm in a way that intensifies the
  piece
• Maintain the overall shape of the piece
• Construct far-ranging variations that are
  interestingly related to the original

24
Simple patterns are everywhere in music

• Pattern of orientation from 3 to 1
• Up, Down, Up, Down, Up, Down, Down, Down
• ABABAAAB
• The pattern is very close to a canonical pattern
  -- alternation.
• It can be viewed as buckle.
• Patterns of this kind are childrens games.
• There are many other patterns in this piece.
• Music packs high-dimensional childrens games
  into an explosive space of illusory motion.

25
The pattern of rhythm

• Doing exact comparison, measure for measure, gets
  us somewhere with very simple music, but
  generally this gets us nowhere.
• Rhythms can be similar to some degree Rosenkranz
  can be simplified, revealing ABABABAC, where B
  is a simple variation of B.

26
Rhythms can be expressed as patterns of ordered
sets

• A contains B, C and D.
• B contains D.
• This containment relation makes it possible to
  express one rhythm in terms of another
• We can reduce the piece to just 2 different
  rhythms slam.
• We can easily force very many folksongs into
  patterns of just 2 rhythms.
• This is called binary decomposition.

27
Binary decomposition of rhythm

• Idea represent piece as variations of at most
  two different rhythms, which may or may not be
  subsets of one another.
• Algorithm
• take intersection of all rhythms. This is
  rhythm_0 at level 0.
• For this level and all subsequent levels, find
  the variant that describes a maximal subset of
  rhythm_0. This is rhythm_1.
• Produce a pattern of 0s and 1s, where 0 is
  rhythm_0 and 1 is rhythm_1
• Construct a tree with this pattern at top.
  Construct nodes at level n by recursing, using
  rhythm_0 rhythm_1 from previous level.

28
The decomposition tree shows childrens game
patterns

• tree
• pat 1 0 1 0 1 1 1
• rhythm_0 1440 480
• rhythm_1 240 240 480 480 480
• sub_0 None
• sub_1
• Some
• pat 1 1 1 0 1
• rhythm_0 240 240 480 480 480
• rhythm_1 240 240 480 240 240 480
  
• sub_0 None
• sub_1 None

29
Using the tree, the piece can be rewritten in a
simpler way

• the rhythm of the entire piece can be viewed as a
  variation of a dotted half a quarter. By adding
  in rhythm_1 we can see the entire piece as an
  alternation of just 2 rhythms. Below that, sub_1
  pattern shows in what way rhythm_1 is varied on
  the next level.

30
Interesting result

• a huge percentage (gt 80) of German folksongs
  decompose into binary patterns of a cycling
  period generally of two measures. That is, there
  is a recurrent subrhythm that runs through the
  entire composition.
• Subjectively, subrhythm alone is musically
  meaningful.
• Subjectively, the addition of rhythm_1 results in
  an excellent melodic approximation of a large
  percentage of all songs examined.
• This would suggest that binary decomposition is
  related to the process of music perception as
  it were, accounted for in the composition of
  music.

31
Decomposition helps structure a variation

• Algorithm was used to rewrite melodies generated
  by Melisma (Temperley) to good effect
• Eg, by removing unique nodes with no children we
  force a rhythm to conform to a rhythm that exists
  somewhere else in the tree. The piece
  automatically takes on more structure than before.

32
Generalizing binary decomposition

• Reconsider the orientation analysis of Happy
  Birthday

It is much more complex to look at 2 patterns
simultaneously than at one. Method collect
patterns of features For some set of features,
attempt to unify by forcing everything into the
best pattern.
33
Implied objects a feature decomposition algorithm

• Rhythm has an inclusion relation.
• So does pitch Pitch-gtInterval-gtOrientation.
• Can generalize Buckles are included in the
  class of rhythms that can be divided into 4 (ie
  R4).
• That is, included features can be schemas.
• Main idea some patterns include other patterns.
• Construct chains of binary patterns, such that
  each chain has a 1 whenever the preceding chain
  has a 1 in that position.

These represent patterns of features that have
been extracted from some music. The implies chain
means that it is possible to construct an object
that has nested constancies which lie in some
pattern.
34
Implied objects in Happy Birthday-- one of 8
decompositions

• Node
• (Several
• (Interval,
• (T4 1 0 1 0 0 0 0 0 0 1,
• 0 2.01.000 D5/62 480 0 0
• 1 2.02.000 C5/60 480 0 0))
• (Pitch,
• (X6 1 0 1 0 0 0 0 0 0 1,
• 0 2.01.000 D5/62 480 0 0
• 1 2.02.000 C5/60 480 0 0)),
• Node
• (One (Interval,
• (X2 1 0 1 0 0 1 0 0 0,
• 0 2.01.000 D5/62 480 0 0)),
• Node
• (One (Ori, (Nb orientation extends to
  next note, not seen here.)
• (T0 1 0 1 0 0 1 1 0 0
  1,
• 0 2.01.000 D5/62 480 0 0
• 1 2.02.000 C5/60 480 0
  0)),

35
Patterns of included features line up
hierarchically
36
Simplifying (?) Happy Birthday

• Just one step away from being a duple
• Preserves a set of features instepdown,
  contour.
• The tune is still recognizable
• Conscionable as an underlying canonical melody
• We can evaluate the kinds of variations that
  transform this back to its original form.
• The tree reveals patterns of variations there is
  a complex metalevel to be considered.
• (In general, pattern analysis always creates a
  new pattern)
• The results can be used recursively. (eg, the
  high b-flat creates a new pattern in register it
  could be registrally simplified etc.)
• Provides an important compositional methodology.
• Proposes that listening somehow or other implies
  recomposition.

37
In sum

• Using ideas like schemas, patterned shapes,
  binary reduction, implied objects Jill provides
  Jack with information about what is happening in
  the music.
• Jack Jill use many other theorizations and
  algorithms not discussed
• System is more like computer vision than like a
  grammar parser