Music image processing

1
Music image processing

• Tim Bell
• Department of Computer Science and Software
  Engineering
• University of Canterbury, Christchurch, New
  Zealand

2
With

• David Bainbridge (Waikato)
• Richard Lobb
• Dave Blizzard (Portland, OR)
• Richard Green
• John McPherson
• Karen Lin
• Annabel Church
• Simon Glass

3
Overview

• OMR
• Digital music stand
• Page turning and image size
• Fast capture of music
• Music classification

4
The vision...

• All music available on the web
• as score, recording and MIDI
• search by name, composer, lyrics, phrase,
  similarity, genre, novelty
• remunerate those responsible
• culturally sensitive access

5
Barlow and Morgenstern 1949
6
Barlow and Morgenstern 1949
7
Barlow and Morgenstern 1949
8
Activities with music

• Composing
• Arranging
• Performance
• Teaching
• Musicology
• Recording
• Accompanying
• Transcribing...

9
Digital music problems

• Cost of hardware and software
• Viewing music on small screen
• Loss of inspiration and creativity
• Loss of efficiency
• Learning curve
• Software compatibility

10
Problems with paper

• Pen or pencil?
• Indexed retrieval

11
Visual display
Digital image
Human memory
Digital semantic
Audio
12
Visual display
Digital image
memorised oral tradition original ideas
Human memory
Digital semantic
Audio
13
Visual display
Digital image
Human memory
live performance mp3, wav, cd video?
Digital semantic
Audio
14
Visual display
Digital image
sheet music screen
Human memory
Digital semantic
Audio
15
Visual display
Digital image
sheet music screen
Human memory
Digital semantic
Audio
16
Visual display
Digital image
sheet music screen
Human memory
Digital semantic
Audio
17
Visual display
Digital image
sheet music screen
Human memory
Digital semantic
Audio
18
Visual display
Digital image
sheet music screen
Human memory
Digital semantic
Audio
19
Visual display
Digital image
D (lick 1) Dmin5 Look at me
now, will I ever learn? D (lick
2) Dmin5 G I dont know how but I
suddenly lose control.
sheet music screen
Human memory
Digital semantic
Audio
20
Visual display
Digital image
Live performance
Human memory
Digital semantic
Audio
21
Visual display
Digital image
read
Live performance
Human memory
Digital semantic
Audio
22
Visual display
Digital image
Live performance
Human memory
play (interpret)
Digital semantic
Audio
23
Visual display
Digital image
Transcription
Human memory
Digital semantic
Audio
24
Visual display
Digital image
Transcription
Human memory
listen
Digital semantic
Audio
25
Visual display
Digital image
write
Transcription
Human memory
Digital semantic
Audio
26
Visual display
Digital image
BMP, GIF, JPEG
Human memory
Digital semantic
Audio
27
Visual display
Digital image
scanner, camera
Human memory
Digital semantic
Audio
28
Visual display
Digital image
print, display
Human memory
Digital semantic
Audio
29
Visual display
Digital image
Human memory
MIDI NIFF MUSICXML GUIDO
Digital semantic
Audio
30
Visual display
Digital image
Render (Sibelius, Lime, Guido, Tex etc.)
Human memory
Digital semantic
Audio
31
Visual display
Digital image
OMR
Human memory
Digital semantic
Audio
32
Visual display
Digital image
Human memory
Synthesis (audio rendering)
Digital semantic
Audio
33
Visual display
Digital image
Human memory
Audio analysis (monophonic, polyphonic)
Digital semantic
Audio
34
Visual display
Data entry
Digital image
Human memory
Digital semantic
Audio
35
Visual display
Digital image
Weak links
Human memory
Digital semantic
Audio
36
Visual display
Digital image
Labour intensive links
Human memory
Digital semantic
Audio
37
Visual display
Digital image
Human memory
Digital semantic
Audio
38
Visual display
Digital image
QBH
Human memory
Digital semantic
Audio
39
Operations on music
40
Visual display
Digital image
Compose Arrange/orchestrate Rehearse React
Human memory
Digital semantic
Audio
41
Visual display
Digital image
Library (personal, shared) Music stand (rehearse,
perform)
Human memory
Digital semantic
Audio
42
Visual display
Digital image
Intermediate form Archive
Human memory
Digital semantic
Audio
43
Visual display
Suitable for transposition part
splitting reduction searching theme
detection accompaniment performance following
Digital image
Human memory
Digital semantic
Audio
44
Visual display
Digital image
listening recording studio analysis thumbnail back
ing
Human memory
Digital semantic
Audio
45
Optical Music Recognition
46
(No Transcript)
47
(No Transcript)
48
Wabot-2

• 1980-1984
• Read simple score
• Heavy processing requirements

49
(No Transcript)
50
Staff line removal/identification

• Horizontal projection
• Vertical slices
• Wobble/track
• Chords
• Template

51
(No Transcript)
52
Horizontal projection
53
Piece at an angle

• Rotate until correct

54
Vertical slices
55
Vertical slices
56
Wobble/track
57
Object location

• Fragmentation
• Superimposed
• Touching objects

58
Identifying objects

• Flood fill
• Template matching
• Hough transform

59
Constructing musical features

• Grammars
• Decision tree
• Rules

60
Musical semantics

• Treble clef determines pitches
• Accidentals change pitch
• Time signature changes note lengths

61
Commercial systems

• Sharpeye
• Vivaldi
• Neuratron PhotoScore
• and more

62
Is 96 recognition good enough?

• One mistake in 24 notes
• No interpretation if playing music
• Time to set up, train and correct greater than
  typing?

63
Not just notes
64
(No Transcript)
65
Visual display
Digital image
Optical music recognition in practice!
Human memory
Digital semantic
Audio
66
The gulf of interpretation

• Classical tempo and dynamics
• Jazz improvisation
• Rock style (e.g. syncopation, articulation)
• Figured bass
• Cadenzas
• MIDI vs. Orchestra

67
Interpretation
68
Interpretation
69
Visual display
Digital image
Pen-based music data entry
Human memory
Digital semantic
Audio
70
(No Transcript)
71
(No Transcript)
72
Coloured staveline removal

• Scanned as RGB
• Convert to HSV and CMYK
• V indicates colour
• K indicates pencil, black pen

73
(No Transcript)
74
(No Transcript)
75
Mis-classified images
76
Discussion?