PowerPoint Presentation - Goal:

1
(No Transcript)
2
How remarkable would it be if one could
experience and express the spectrum of emotions
embodied in music originating from oneself,
without the crutch of a composers
intercessionCan the touch that lies behind music
be tapped? Manfred Clynes
3

• Goal
• Direct music using affective cues
• Issues
• Devise mapping scheme of music parameters
• Correlate affective signals with music
  parameters
• Disambiguate data collected during music
  listening
• Develop algorithm to navigate music map by
  affect

4

• Presentation Outline
• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

5

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

6

• Existing Research
• Berlyne (1974)
• liking is greatest for stimuli of moderate
  arousal
• arousal and liking should be dependent
  familiarity and complexity affect arousal level
• Russell et al., (1981) Circumplex model of
  emotion
• puts emotions in a circle around grid of
  pleasant-unpleasant and arousing-sleepy
• North and Hargreaves (1997)
• correlated circumplex model to valence/arousal
  dimensions
• replaced pleasant-unpleasant axis with
  like-dislike axis in circumplex
• coordinates of liking-arousal was found to be a
  reliable predictor of emotional reaction
• Schubert (1996)
• people often enjoy music that is unpleasant
• Ritossa and Rickard (2004)
• tested dimensions of pleasantness and liking in
  circumplex
• one of eight emotions reliably predicted using
  arousal, familiarity pleasantness
• pleasantness was more useful predictor of
  emotions than liking

7

• Existing Research
• Marrin and Picard (1998) Conductors Jacket
• records a variety of affective signals during
  conducting
• Healey, Picard and Dabek (1998) Affective DJ
• uses a mixture of physiological and subjective
  data
• compares SCR from first 30 seconds of current
  song to last 30 seconds of previous song
• algorithm selects songs to change from current
  affective state to desired state
• Kim and André (2004) Composing Affective Music
• uses ECG, EMG, SCR and RESP data
• composes algorithmically
• SCR is useful indicator for unsettling-relaxing
• EMG useful indicator for positive-negative

8

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

9

• Project Overview
• Small-scale listening experiment
• Change music parameters, and observe
  physiological and self-report data.
• Music generator
• Develop real-time algorithm that modifies music
  parameters based on affect.

10
Music Parameter Mapping
instrument layering frequency-domain
density complexity time-domain density
layering
complexity
user study
Music Affect Mapping
Music Parameter Mapping
music box
11
Music Affect Mapping
arousal reaction level to music valence
subjects like/dislike of music
arousal
valence
user study
Music Affect Mapping
Music Parameter Mapping
music box
12
Music Affect Mapping
high arousal
engaging
annoying
like
dislike
soothing
boring
low arousal
13
The Challenge
high arousal
Current State
Goal State
engaging
annoying
like
dislike
soothing
boring
low arousal
14
The Challenge
Current State
Goal State
engaging
annoying
engaging
boring
engaging
soothing
soothing
annoying
soothing
boring
15
Changing Music Affect
Given set changes in parameters, what are the
trends for motion in affective space? How do
those trends change depending on the initial
affective state?
Music Parameters
Probability
Start State
End State
engaging
annoying
f-1, t-1 f-1, t1 f1, t-1 f1, t1
engaging
boring
?
engaging
soothing
soothing
annoying
soothing
boring
16

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

17

• Listening Experiment
• Preparation of Music Clips
• Five pieces with constant tempo were composed,
  ranging from jazz, rock to electronic music.
• For each piece, looping audio segments were
  produced to reflect the layer/complexity map.
• Segments were arranged into a 4x4 matrix for
  each piece.
• For each piece, four clips were assembled by
  traversing the matrix as follows
• Increasing complexity
• Increasing instrument layering
• Decrease complexity
• Decrease instrument layering
• ((Listen))

18
Listening Experiment Preparation for Data
Collection Physiological Data Galvactivator
set up to measure GSR microphone connected to
measure presence of foot-tapping Self-report
Data dual 7-point scales were set up for
subjects affective response self-report two
sets were prepared for subjects to report their
initial and final reactions
19
Listening Experiment Conducting the
Experiment 8 participants 6 male, 2 female
20 audio clips (25-45 seconds each) were played
for each subject experiment lasted
approximately 25 minutes Physiological Data
GSR was sampled every second throughout entire
listening, BPM and velocity of foot-tapping
was sampled every second, and presence of
foot-tapping was sampled every 2
seconds Self-report Data subject self-reported
affective response twice during each clip
(initial final)
20

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

21
Data Analysis
Total Physiological Response - Subject 3
Total Physiological Response - Subject 4
22
Data Analysis
MUSIC CLIP 9 Increase Complexity
layering
Subject 2 physiological data
annoying engaging
complexity
like
dislike
boring soothing
Subject 2 self-report data
23
Data Analysis
UNUSUAL EXAMPLE
MUSIC CLIP 9 Increase Complexity
layering
Subject 6 physiological data
annoying engaging
complexity
like
dislike
boring soothing
Subject 6 self-report data
24
Data Analysis
engaging
annoying
GSR
presence of foot-tapping
soothing
boring
25
Data Analysis
GSR falling
GSR rising
S1 engaging S2 soothing S3 boring S4
annoying
26
Data Analysis
S1
S2
S1
S2
GSR up
down
S3
S4
S3
S4
GSR up
down
S1 engaging S2 soothing S3 boring S4
annoying
27

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

28
Algorithm Overview A pair of state transition
models was constructed based on physiological and
self-report data The first model tries to
detect the listeners current affective
state. The second model chooses the direction
most likely to induce the goal state.
29
Probability Table for Detecting Affective State
S1 engaging S2 soothing S3 boring S4
annoying
30
Affective State Transition Model
ACTION Increase Complexity
layering
complexity
S1 engaging S2 soothing S3 boring S4 annoying
31
Affective State Transition Model
ACTION Increase Layering
layering
complexity
S1 engaging S2 soothing S3 boring S4 annoying
32
Affective State Transition Model
ACTION Decrease Complexity
layering
complexity
S1 engaging S2 soothing S3 boring S4 annoying
33
Affective State Transition Model
ACTION Decrease Layering
layering
complexity
S1 engaging S2 soothing S3 boring S4 annoying
34
Summary of State Transitions
S1-engaging
S4-annoying
S1 S2 S3 S4
S1 S2 S3 S4
S3-boring
S2-soothing
S1 S2 S3 S4
S1 S2 S3 S4
35
Summary of State Transitions
S1 engaging S2 soothing S3 boring S4 annoying
36
System Diagram
Physiological data input
Control
Affect induced
Affective state detection algorithm
Perception
Goal state
Initial state
Direction choosing algorithm
Action
Music
Change music parameter
37

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

38

• Conclusions
• changes in music parameters can be correlated
  to affective response to music
• Markov chains are a useful tool for
  constructing a predictive listening system
• specific observations about mapping
  layering/complexity to arousal/valence
• engaged listeners tend to stay engaged
• annoyed listeners tend to stay annoyed
• soothed listeners tend to stay soothed, but
  also easily bored or engaged
• bored listeners tend to become interested by
  any change in parameters
• annoyed listeners are more likely to be
  engaged if first induced to boredom

39

• Existing Work
• Project Overview
• Music Parameters
• Affective Signals
• Listening Experiment
• Data analysis
• Algorithm
• Conclusions
• Demonstration

40

• Future Work
• improve accuracy of predictions by
  incorporating more user data
• improve affective state predictions using
  additional affective signals
• apply affect-parameter mapping to algorithmic
  composition
• use machine learning to customize predictions
  to individual subject

41

• References
• Berlyne, D.E. (1974) Studies in New Experimental
  Steps Towards an Objective Psychology of
  Aesthetic Appreciation. Halstead Press.
• Clynes, M. (1977) Sentics. Anchor Press.
• Healey, J., Picard, R. and Dabek, F. (1998) A
  new affect-perceiving interface and its
  application to personalized music selection,
  Proc. of the Perceptual User Interfaces Workshop,
  4-6.
• Kim, S. and André, E. (2004) Composing affective
  music with a generate and sense approach, Proc.
  of Flairs 2004. American Association for
  Artificial Intelligence.
• Marrin, T. (2000) Inside the Conductors Jacket
  Analysis, Interpretation and Musical Synthesis of
  Expressive Gesture. PhD thesis, MIT Media Lab,
  Cambridge, MA.
• Meyer, L. (1956) Emotion and Meaning in Music.
  University of Chicago Press.
• North, A.C. and Hargreaves, D.J. (1997) Liking,
  arousal potential and the emotions expressed by
  music, Pscyhomusicology 1477-93.
• Ritossa, D. and Rickard, N. (2004) The relative
  utility of pleasantness and liking dimensions
  in predicting the emotions expressed by music.
  Psychology of Music. v.32(1)5-22
• Russell, J.A. (1980) A Circumplex Model of
  Affect, Journal of Personality and Social
  Psychology 39 1161-78.
• Rutherford, J. and Wiggins, G.A. (2002) An
  experiment in the automatic creation of music
  which has specific emotional content, Proc. for
  the 7th International Conference on music
  Perception and Cognition, Sydney, Australia.
• Schubert, E. (1996) Enjoyment of negative
  emotions in music an associative network
  explanation, Pscyhology of Music 24 18-28.
• Sloboda, J.A. and Juslin P.N. (2001) Music and
  Emotion Theory and Research. Oxford University
  Press.