1. Semantic Web Application: Music Retrieval Ying Ding SLIS, IU

2. Enabling Access to Sound Archives through Integration, Enrichment and Retrieval http://www.elec.qmul.ac.uk/easaier/

3. The EASAIER Project EASAIER - Enabling Access to Sound Archives through Integration, Enrichment and Retrieval EU funded project, 30month duration (started May 2006)‏ Partners:

4. EASAIER - Goals Overcome problems for many digital sound archives concerning online access sound materials and related media often separate searching audio content limited EASAIER Framework Integration of Sound Archives Low level audio feature extraction (speech/music)‏ Intelligent User Interface Enhanced Access Tools looping, marking of audio sound source separation time and pitch scale modification Semantic Search Evaluation

5. Semantics in EASAIER Description of metadata using an ontology High-level metadata e.g. title, author of an audio asset sources are databases, files in e.g. DC, MARC Low-level metadata e.g. speech event occurs at timestamp xyz feature extractor tools Semantic Search Search across variety of metadata Search across multiple archives Similarity Search Related content acquisition from the Web

6. The EASAIER System

7. Music Ontology Overview Merging existing related ontologies Developed by QMUL Cover the major requirements Widely-adopted Four core MO components FRBR FOAF Event Timeline http://musicontology.com/

8. The Music Ontology: Timeline Ontology Expressing temporal information, e.g. This performance happened the 9th of March, 1984 This beat is occurring around sample 32480 The second verse is just before the second chorus

9. The Music Ontology: Event Ontology Event — An arbitrary classification of a space/time region This performance involved Glenn Gould playing the piano This signal was recorded using a XXX microphone located at that particular place This beat is occurring around sample 32480

10. The Music Ontology: FRBR & FOAF FRBR – Functional Requirements for Bibliographic Records Work — e.g. Franz Schubert's Trout Quintet Manifestation — e.g. the "Nevermind" album Item — e.g. my "Nevermind" copy  FOAF – Friend of a Friend  Person  Group  Organization

11. The Music Ontology – Music Production Concepts On top of FRBR: MusicalWork, MusicalManifestation (Record, Track, Playlist, etc.), MusicalItem (Stream, AudioFile, Vinyl, etc.)‏  On top of FOAF:  MusicArtist, MusicGroup, Arranger, Engineer, Performer, Composer, etc. — all these are defined classes: every person involved in a performance is a a performer...  On top of the Event Ontology:  Composition, Arrangement, Performance, Recording  Others :  Signal, Score, Genre, Instrument, ReleaseStatus, Lyrics, Libretto, etc.

12. The Music Ontology – Music Production Workflow

13. Low-level metadata is output in RDF using Music Ontology Audio Feature extractor Speech recognition service Emotion detection service High-level metadata import DB Schema Mapping e.g. D2R, Virtuoso RDF Views Standardized Metadata import DC, MARC, METS,... Linked Data ? DBPedia, Geonames,... Metadata in RDF

14. Hotbed Database Music Ontology Querying Publishing Extending Instruments Taxonomy Hotbed RDF the Semantic Archivist Query Interface Sound Access tools Features Extraction, Visualization,... Use Case: Archive Publication - HOTBED

15. 1) editing the ontology using WSMT editor to extend the ontology Music Ontology Graphical Edit Text Edit

16. 2) performing tests on the new extension What are the instruments in my taxonomy ? Did i forget any kind of [pipe] ?

17. 3)mapping Scottish Instruments to a general Instruments taxonomy

18. 4) relating and publishing Hotbed Relate tables from hotbed to concepts from the MO Publish on the semantic web via the D2R tool Mapping  The server offers a SPARQL end-point for external apps RDF Publication via D2R tool Hotbed Database Music Ontology

19. Mapping Metadata to the Music Ontologies Title: File 2 Author: Oliver Iredale Searle Perfomers: Katie Punter Source Type: Audio Source: File 2 Instrument: Flute Instrument occurrence timings: 0"-16" Time Signature: 4/4 Beats per minute: 50 Tonality: Bb major Searle Testbed :music a mo:Signal ; dc:title "File 2" ; dc:author "Oliver Iredale Searle" ; :music-performance a mo:Performance ; mo:recorded_as :music ; mo:composer :OliverIredaleSearle ; mo:instrument mo:flute ; mo:performer :KatiePunter ; mo:bpm 50 ; mo:meter "4/4" ; mo:key #BFlatMajor. :KatiePunter a foaf:Person. :ss1 a af:PersonPlaying; af:person :KatiePunter; event:time [ tl:onTimeLine :tl1234; tl:beginsAt "PT0S"; tl:duration "PT16S"; ].

20. <speech_descriptor word="power" audio_material="c:/hotbed/performance/1004.wav" position_sec="10" duration_sec="5" confidence="89" /> a af:Text; af:text"power"; af:confidence "89"; event:time [ a time:timeInterval; tl:onTimeline ; tl:beginsAtDuration "PT10S"; tl:durationXSD "PT5S"; ]. ALL web service output Mapping Metadata to the Music Ontologies

21. Vamp Output event:time [ a time:Instant ; tl:onTimeLine :tl898; tl:at "PT0.0928S"; ]; mo:bpm "224.69"; Mapping Metadata to the Music Ontologies

22. Built on top of OpenRDF Sesame 2.0 Query interfaces Web Service (Servlet)‏ HTTP SPARQL Endpoint Web Service provides predefined SPARQL query templates Themes Music, Speech, Timeline, Related media, Similarity Dynamic FILTER constructs Results in SPARQL Query  Results XML Format Interface for RDF metadata import using the Archiver application RDF Storage and Retrieval Component

23. Enhanced Client

24. Web client

25. Related media Double- click

26. Related media on the web (1)‏ Result search for author “Coltrane” Track selection Web related media search launched automatically according to the name of the author

27. Related media on the web (2)‏

28. Demo http://www.elec.qmul.ac.uk/easaier/index- 3.html http://www.elec.qmul.ac.uk/easaier/index- 3.html http://easaier.deri.at/demo/

29. Demo Time and Pitch Scale Modification (demo)demo Sound source separation (demixing/remixing, Noice reduction, etc.) (demo)demo Video time stretching (to slow down or speed up images while retaining optimal sound) (demo)demo

30. Scenario 1 – Artist Search Aggregation of music artist information from multiple web sources Ontology based search: MusicBrainz data mapped to the MusicOntology MusicBrainz Web Service: allows to retrieve artist URI by literal based search MusicBrainz RDF Dump: retrieve RDF use SPARQL to perform queries (e.g. resolve relationships) Web2.0 Mashups: Retrieve data (videos, images) from external sources utilize RSS Feeds, APIs etc. from Youtube, LyricWiki, Google more accurate results using references from MusicBrainz RDF data

31. Scenario 1 – Artist Search “Beatles” WS Interface RDF Dump process data...

32. Scenario 1 – Artist Search

34. Scenario 2 – Instrument Reasoning Reasoning over HOTBED instrument scheme Ontologize data from HOTBED (Scottish Music Archive) Usage of D2R to lift data from legacy DBs to RDF Ontologies: MusicOntology Instrument Ontology (domain related taxonomy) Subsumption reasoning: Retrieve instrument tree Search for persons that play an instrument Subclass relations: resolve persons playing more specific instruments Example: Wind-Instrument < WoodWind < Flute

35. Scenario 2 – Instrument Reasoning  Example:  Search for people playing instrument of type Woodwind

36. Demo 3 – Rules Infer new knowledge with rules Domain Rule Sophisticated Query Albums based on certain Band/Artist/Instrument UseCase: The Velvet Underground discography Available information: Membership durations Album release dates „Founders“ of the band ? exist _artist,, forall ?x,,  Albums & corresponding members

37. Demo 3 – Rules Basic Information Band Founder Band Duration (Members & Albums) Album Tracks