Title: Semantic Web Application: Music Retrieval
1Semantic Web Application Music Retrieval
2- http//www.elec.qmul.ac.uk/easaier/
Enabling Access to Sound Archives through
Integration, Enrichment and Retrieval
3The EASAIER Project
- EASAIER - Enabling Access to Sound Archives
through Integration, Enrichment and Retrieval - EU funded project, 30month duration (started May
2006)? - Partners
4EASAIER - Goals
- Overcome problems for many digital sound archives
concerning online access - sound materials and related media often separate
- searching audio content limited
- EASAIER Framework
- Integration of Sound Archives
- Low level audio feature extraction
(speech/music)? - Intelligent User Interface
- Enhanced Access Tools
- looping, marking of audio
- sound source separation
- time and pitch scale modification
- Semantic Search
- Evaluation
5Semantics in EASAIER
- Description of metadata using an ontology
- High-level metadata
- e.g. title, author of an audio asset
- sources are databases, files in e.g. DC, MARC
- Low-level metadata
- e.g. speech event occurs at timestamp xyz
- feature extractor tools
- Semantic Search
- Search across variety of metadata
- Search across multiple archives
- Similarity Search
- Related content acquisition from the Web
6The EASAIER System
7Music Ontology
- Overview
- Merging existing related ontologies
- Developed by QMUL
- Cover the major requirements
- Widely-adopted
- Four core MO components
- FRBR
- FOAF
- Event
- Timeline
http//musicontology.com/
8The Music Ontology Timeline Ontology
- Expressing temporal information, e.g.
- This performance happened the 9th of March, 1984
- This beat is occurring around sample 32480
- The second verse is just before the second chorus
9The Music Ontology Event Ontology
- Event An arbitrary classification of a
space/time region - This performance involved Glenn Gould playing the
piano - This signal was recorded using a XXX microphone
located at that particular place - This beat is occurring around sample 32480
10The Music Ontology FRBR FOAF
- FRBR Functional Requirements for Bibliographic
Records - Work e.g. Franz Schubert's Trout Quintet
- Manifestation e.g. the "Nevermind" album
- Item e.g. my "Nevermind" copy
- FOAF Friend of a Friend
- Person
- Group
- Organization
11The Music Ontology Music Production Concepts
- On top of FRBR
- MusicalWork, MusicalManifestation (Record, Track,
Playlist, etc.), MusicalItem (Stream, AudioFile,
Vinyl, etc.)?
- On top of FOAF
- MusicArtist, MusicGroup, Arranger, Engineer,
Performer, Composer, etc. all these are defined
classes every person involved in a performance
is a a performer...
- On top of the Event Ontology
- Composition, Arrangement, Performance, Recording
- Others
- Signal, Score, Genre, Instrument, ReleaseStatus,
Lyrics, Libretto, etc.
12The Music Ontology Music Production Workflow
13Metadata in RDF
- Low-level metadata is output in RDF using Music
Ontology - Audio Feature extractor
- Speech recognition service
- Emotion detection service
- High-level metadata import
- DB Schema Mapping
- e.g. D2R, Virtuoso RDF Views
- Standardized Metadata import
- DC, MARC, METS, ...
- Linked Data ?
- DBPedia, Geonames, ...
14Use Case Archive Publication - HOTBED
Publishing
Extending
Hotbed Database
Music Ontology
Instruments Taxonomy
Querying
Query Interface
the Semantic Archivist
Sound Accesstools
FeaturesExtraction,Visualization,...
Hotbed RDF
151) editing the ontology
- using WSMT editor to extend the ontology
Graphical Edit
Music Ontology
Text Edit
162) performing tests on the new extension
- What are the instruments in my taxonomy ?
- Did i forget any kind of pipe ?
173)mapping Scottish Instruments to a general
Instruments taxonomy
184) relating and publishing Hotbed
- Relate tables from hotbed to concepts from the MO
- Publish on the semantic web via the D2R tool
Mapping
Music Ontology
Hotbed Database
RDF Publicationvia D2R tool
- The server offers a SPARQL end-point for external
apps
19Mapping Metadata to the Music Ontologies
music a moSignal dctitle "File 2"
dcauthor "Oliver Iredale Searle"
music-performance a moPerformance
morecorded_as music mocomposer
OliverIredaleSearle moinstrument moflute
moperformer KatiePunter mobpm 50
mometer "4/4" mokey BFlatMajor. KatiePunt
er a foafPerson . ss1 a afPersonPlaying afpe
rson KatiePunter eventtime tlonTimeLine
tl1234 tlbeginsAt "PT0S"
tlduration "PT16S" .
Title File 2 Author Oliver Iredale
Searle Perfomers Katie Punter Source Type
Audio Source File 2 Instrument
Flute Instrument occurrence timings 0"-16" Time
Signature 4/4 Beats per minute 50 Tonality Bb
major
Searle Testbed
20Mapping Metadata to the Music Ontologies
ALL web service output
ltxml version"1.0" encoding"UTF-8"gt ltspeech_retri
eveResultgt ltspeech_descriptor word"power"
audio_material"c/hotbed/performance/1004.wav"
position_sec"10" duration_sec"5"
confidence"89" /gt lt/speech_retrieveResultgt lt/xml
gt
lthttp//www.myarchive.org/signal/1234/event/powerP
T10Sgt a afText aftext "power" afconfidence
"89" eventtime a timetimeInterval tlon
Timeline lthttp//www.myarchive.org/signal-timeline
/1234gt tlbeginsAtDuration "PT10S" tldurati
onXSD "PT5S" .
21Mapping Metadata to the Music Ontologies
Vamp Output
ltmetadata type"audio"gt ltcategory name"vamp"gt
ltfeature name"beats" type"variablerate"
description"Detected Beats" unit"N/A"gt
ltdatagt ltevent idx"0" timestamp"
0.0928" duration"0" label"224.69 bpm"/gt
lt/datagt lt/featuregt lt/categorygt lt/metadatagt
eventtime a timeInstant tlonTimeLine
tl898 tlat "PT0.0928S"
mobpm "224.69"
22RDF Storage and Retrieval Component
- Built on top of OpenRDF Sesame 2.0
- Query interfaces
- Web Service (Servlet)?
- HTTP SPARQL Endpoint
- Web Service provides predefined SPARQL query
templates - Themes
- Music, Speech, Timeline, Related media,
Similarity - Dynamic FILTER constructs
- Results in SPARQL Query ? Results XML Format
- Interface for RDF metadata import using the
Archiver application
23Enhanced Client
24Web client
25Related media
26Related media on the web (1)?
27Related media on the web (2)?
28Demo
- http//www.elec.qmul.ac.uk/easaier/index-3.html
- http//easaier.deri.at/demo/
29Demo
- Time and Pitch Scale Modification (demo)
- Sound source separation (demixing/remixing, Noice
reduction, etc.) (demo) - Video time stretching (to slow down or speed up
images while retaining optimal sound) (demo)
30Scenario 1 Artist Search
- Aggregation of music artist information from
multiple web sources - Ontology based search
- MusicBrainz data mapped to the MusicOntology
- MusicBrainz Web Service
- allows to retrieve artist URI by literal based
search - MusicBrainz RDF Dump
- retrieve RDF
- use SPARQL to perform queries (e.g. resolve
relationships) - Web2.0 Mashups
- Retrieve data (videos, images) from external
sources - utilize RSS Feeds, APIs etc. from Youtube,
LyricWiki, Google - more accurate results using references from
MusicBrainz RDF data
31Scenario 1 Artist Search
WS Interface
Beatles
ltURIgt
ltURIgt
process data...
RDF Dump
32Scenario 1 Artist Search
33Scenario 1 Artist Search
34Scenario 2 Instrument Reasoning
- Reasoning over HOTBED instrument scheme
- Ontologize data from HOTBED (Scottish Music
Archive) - Usage of D2R to lift data from legacy DBs to RDF
- Ontologies
- MusicOntology
- Instrument Ontology (domain related taxonomy)
- Subsumption reasoning
- Retrieve instrument tree
- Search for persons that play an instrument
- Subclass relations resolve persons playing more
specific instruments - Example Wind-Instrument lt WoodWind lt Flute
35Scenario 2 Instrument Reasoning
- Example
- Search for people playing instrument of type
Woodwind
36Demo 3 Rules
- Infer new knowledge with rules
- Domain Rule
- Sophisticated Query
- Albums based on certain Band/Artist/Instrument
- UseCase The Velvet Underground discography
- Available information
- Membership durations
- Album release dates
- Founders of the band ?
- exist _artist, lt_band, hasMember, _artistgt,
lt_artist, onDuration, _durationgt - forall ?x, lt_band, hasMember, ?xgt, lt ?x,
onDuration, ?timegt - lt?time, notBefore, _durationgt
- ? lt_band, founder, _artistgt
- Albums corresponding members
37Demo 3 Rules
Basic Information
Band Founder
Band Duration (Members Albums)
Album Tracks