Functional Data Analysis of Continuous Judgments in Music Cognition

1
Functional Data Analysis of Continuous Judgments
in Music Cognition
2
Gesture in Musical Performance

• What role do a musicians gestures play in a
  performance?
• Do they convey emotion?
• Do gestures convey the same things as does the
  music?
• Popular study in field of music psychology
• Data on listeners emotions are collected in
  real-time
• With a real-time Optotrak slider, which
• Measures location of the slider 10 times per
  second.

Continuous Judgments of Music
3
The Tension Gesture Experiment

• Musical performance recorded on video
• Stravinskys 2nd Piece for Solo Clarinet
• 30 musically-trained participants either
• Watched Listened to the recording (Natural)
• Watched the silent recording (Video Only), or
• Listened to the recording (Audio Only).
• And reported their continuous level of tension
  emotion.

4
The Data

• Vectors of length 800
• 10 Audio Only, 10 Video Only, 10 Audio
  Video
• Scaled to 0,1 interval

5
The Functional Objects

• 150 order 6 B-splines, using FDA software in
  Matlab.
• Then smoothed.

6
Functional Principal Components Analysis

• 25 s 65 s crucial separation
• Effect of amplification / attenuation how
  strong are the changes of emotion?

7
Functional Linear Model
Emotion(t) µ(t) ß0(t)AudRemoved
ß1(t)VidRemoved e(t)
8
Functional Linear Model
Emotion(t) µ(t) ß0(t)AudRemoved
ß1(t)VidRemoved e(t)
9
Derivatives describe music dynamics Tension

• These judgments are really measures of musical
  emotion.
• Tension/Resolution is rate of change of emotion
  (velocity).
• When emotion is rapidly increasing, music has
  strong tension.
• When emotion is rapidly decreasing, music has
  strong resolution.

10
Derivatives describe music dynamics Force/Release

• Force/Release is rate of change of tension
  (acceleration).
• When tension is rapidly increasing, we feel a
  musical force.
• When tension is rapidly decreasing, we feel a
  musical release.
• When both derivatives are near zero, music is
  inert (the new age/massage music effect).

11
Phase-Plane Plots

• Can examine dynamics with plot of acceleration vs
  velocity.
• Purely harmonic behavior gives a circle.
• The larger the radius, the more musical energy
  transfer.

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15

• 25 33 s
• AUDIO high volume, note density, and pitch end
  of musical phrase
• tension then strong resolution big energy
  transfer
• VIDEO routine gestures then dramatic flourish
• low tension and small resolution moderate pull
  with flourish small energy transfer
• 33 65 s
• AUDIO decrease in volume (mezzo forte to
  pianissimo), note density, and pitch
• continued resolution push from new phrase back
  to inertness
• VIDEO eyebrow and body movements
• push from new phrase, moderate tension from
  movements

16
What have we learned?

• New applications
• PCA as exploration
• Derivatives have physical (and scientific)
  meaning
• Phase-Plane plots highlight relationships