LAM: Musical Audio Similarity

1
LAM Musical Audio Similarity

• Michael Casey
• Centre for Cognition, Computation and Culture
• Department of Computing
• Goldsmiths College, University of London

2
Overview

• Machine Music Understanding
• Features / Classes / Clusters
• Real-Time Audio Matching
• Feature Extraction
• Feature Similarity (Indexing / Retrieval)
• PD/MSP Tools
• Music Similarity Applications
• Sound object matching
• Texture matching

3
Sound Understanding
Signal Processing
Sound Understanding
4
Feature Extraction
5
Feature Extraction
6
Feature Extraction
7
Feature Extraction
8
Feature Extraction
9
Feature Extraction
10
Statistical Learningfor Decision Making
Partitioning of feature space
p( ) P( )
P( )
p( )
Decision boundary
Music
Speech
11
MPEG-7 Audio Tools
Audio
12
MPEG-7 Audio Tools
Log Frequency Spectrogram
Audio
AudioSpectrumEnvelopeD
13
MPEG-7 Audio Tools
Decorrelating Transform / Dimension Reduction
Log Frequency Spectrogram
Log Amplitude
Audio
AudioSpectrumEnvelopeD
AudioSpectrumProjectionD
14
SoundModelStatePathD
Use estimated state sequence as a feature
State Path
15
MPEG-7 Audio Tools
Decorrelating Transform / Dimension Reduction
Log Frequency Spectrogram
Hidden Markov Model
Log Amplitude
Audio
AudioSpectrumEnvelopeD
SoundModelDS
AudioSpectrumProjectionD
16
MPEG-7 Audio StringsAcoustic Lexicons
Decorrelating Transform / Dimension Reduction
Log Frequency Spectrogram
Hidden Markov Model
Log Amplitude
Audio
AudioSpectrumEnvelopeD
SoundModelDS
State Path
AudioSpectrumProjectionD
SoundModelStatePathD
? 7 1 V 7 1 0 1 ...
SYMBOL STRING
17
(No Transcript)
18
State Symbol Sequence (40 State Model)
?71V7101 ...
19
State Symbol Sequence (40 State Model)
?71V7101 ...
20
State Symbol Sequence (40 State Model)
?71V7101 ...
21
State Symbol Sequence (40 State Model)
?71V7101 ...
22
SoundModelStateHistogramD
state index
0.01s Frames
state index
seconds
23
Self-Similarity Matrix
24
Self-Similarity Matrix
25
Self-Similarity Matrix
26
Self-Similarity Matrix
a
27
Self-Similarity Matrix
a
b
28
Self-Similarity Matrix
a
b
29
Self-Similarity Matrix
30
S-Matrix
31
Efficient Storage / Retrieval

• Real-Time Access
• Large Databases
• Distributed Databases

32
PostgreSQL Database Representation of State Path
Strings and Histograms
33
Similarity

• Compute distance between feature pairs
• Features SoundModelStateHistogramD
• Similarity Metric
• dist(a,b) gt 0
• dist(a,b) 0 iff ab
• dist(a,b) dist(b,c) gt dist(a,c)
• Vector Dot Product

34
Similarity of Feature Trajectories
35
Dynamic Time Warping
36
Acousticon Strings

• Distance Metric
• String Edit Distance (Levenschtein)
• Scalable to Large Databases
• PostgreSQL Implementation
• Can use built-in Index Structures
• Scalable to Real-Time Implementation
• matching and audio streaming (lt 20ms )

37
Information Retrievalfor Creativity

• Utilize sound extant database for new material
• Take the structure of a music clip but replace
  the content.
• New interfaces for music creativity.

38
Audio Information Retrieval
MPEG-7 Database
A pre-indexed Collection of Sounds
39
Audio Information Retrieval
MPEG-7 Database
Extract
Segment
Match
Audio Query
A Sound or Scene or List of Sounds
Result List
40
Audio Information Retrieval
MPEG-7 Database
Extract
Segment
Match
Audio Query
Feature extraction from audio.
Result List
41
Audio Information Retrieval
MPEG-7 Database
Extract
Segment
Match
Audio Query
Partitioning of audio into chunks.
Result List
42
Audio Information Retrieval
MPEG-7 Database
Extract
Segment
Match
Audio Query
Result List
Find similar chunks of Audio
43
Real-Time Matching
44
Musaics
Real-Time Matching
45
Musaics
Real-Time Matching
Real-Time Matching
46
Musaics
Real-Time Matching
47
Musaics
Real-Time Matching
48
Musaics
Real-Time Matching
49
Musaics
Real-Time Matching
50
Musaics
Real-Time Matching
51
Musaics
Real-Time Matching
52
Musaics
Real-Time Matching
53
Musaics
Real-Time Matching
54
(No Transcript)