2. GCT731 Fall 2014 Topics in Music Technology - Music Information Retrieval Introduction to MIR Course Overview 1

3. Instructor Name: Juhan Nam ( 남주한 ) Biography – 1994-1998: BS in EE in Seoul National University – 2001-2006: Engineer in Young Chang (Kurzweil) – 2006-2012: PhD in Music (also MS in EE), CCRMA, Stanford University – 2012-2014: Research Engineer in Qualcomm 2

4. Outlines Introduction to MIR – Background Brief history of music technology Recent trends and future direction – MIR Music Data and Information Applications Course overview 3

5. Human, Music and Technology

6. History of Music Technology 5 Material Processing Technology – Mold metal and wood in a high-quality form – Improved or new musical instruments: e.g. piano, saxophone (in 1841) – 3D printer: resurrection of material processing technology?

7. History of Music Technology 6 Electro-Mechanical Technology – Microphone and speakers: sound as “continuous-time signal” – Amplifier and effects: loudness and timbre control – Recorder: paradigm shift in music creation and distribution – Player: listening to music anywhere – New musical instruments: electric guitars

8. History of Music Technology Digital Signal Processing + Computer Technology – A/D, D/A converters: sound as “discrete-time signals” – Synthesizers: “design” musical sounds – Digital audio workstation (DAW): music recording, editing and production – Audio programming: “coding” sound and musical events – MP3 players 7

9. Recent Trends Big data – Online music services: 20M+ songs – Youtube: 100h+ video uploaded per minute – Easy sharing of personal music content Intelligence – Interactive music notation: transcription, score following – Auto-accompaniment: playing with computers Connectivity – Music is combined with human data: play history, preference, location, etc. – Music is distributed via social networks 8

10. Future Directions 9 Challenges – How can we find music content in a human-friendly way? – How can we enhance our musical activity, specifically performance, by musical interaction with computers? – How can we benefit from the connection of human data and network with music? Need of understanding meaning in music!  Intelligent Data Processing + Internet Technology

11. Music Information Retrieval (MIR) Area of research that recently emerged in the background – International Society of Music Information Retrieval (ISMIR) since 2000 Aims to infer various types of information from music data – Making computer understand music as human does Provide intelligent solutions to enhance human musical activities 10

12. Information in Music 11 Instrument: Composer: Key: Similar songs (by motif) : Transcription – melody, score Mood: Melancholy, Sad, … - ELO “After all” - Radiohead “Exit Music” Chopin Piano E-minor Factual Information – track, artist, years Musical Information – timbre, melody, notes, beat, rhythm, chords, structure Semantic Information – genre, mood, user preference

13. Music Data Audio Data – wav or mp3 files, audio streaming from microphones Symbolic Data – MIDI files, Music XML Text Data – Lyrics, review, tags, blog Image Data – CD artwork, media, score (as image files) User Data (human data) – Play history, rate/preference 12

14. MIR Tasks Fingerprinting Cover song detection Music Transcription: melody, notes, tempo, chords Segmentation, structure, alignment Similarity retrieval, playlists, recommendation Classification: genre, mood, tags, … Query by humming Source separation: vocal removal Symbolic MIR: score retrieval or harmony analysis Optical Music Recognition (OMR) MIREX: http://www.music-ir.org/mirex/wiki/MIREX_HOMEhttp://www.music-ir.org/mirex/wiki/MIREX_HOME 13

15. Applications of MIR Musical listening – Music search and recommendation Performance – Interactive music performance Education – Instrument learning Entertainment – Singing evaluation, Game Media production and music creation – Sound sample search in sound libraries 14

16. Music Search Query by music – Search a single unique song identified by the query – Audio fingerprint – Applied to movies, TV and ads, too Query by humming – Sing with humming and find closest matches – Melody match 15

17. Music Recommendation Personalized Radio – Generate Playlist – Based on user data, similarity and context 16

18. Music Performance Score-Following – Listen to performance and track the notes – Examples JKU: http://www.youtube.com/watch?v=Yf05nzix3_whttp://www.youtube.com/watch?v=Yf05nzix3_w Tonara: http://www.youtube.com/watch?v=HBXJZKTOcpw http://www.youtube.com/watch?v=HBXJZKTOcpw Automatic accompaniment – Score following + Interactive Performance – Examples IRCMA’s Antefesco: http://www.youtube.com/watch?v=YkMGtpcAA04 http://www.youtube.com/watch?v=YkMGtpcAA04 Sonation’s Cadenza: http://www.youtube.com/watch?v=UuZUNTEvfhM http://www.youtube.com/watch?v=UuZUNTEvfhM 17

19. Music Education and Entertainment Focus on performance evaluation Learning musical instrument – Examples Ovelin’s GuitarBots: http://www.youtube.com/watch?v=-396LJrqYRk http://www.youtube.com/watch?v=-396LJrqYRk MakeMusic’s smartmusic: http://www.youtube.com/watch?v=b-D00OO7KjY http://www.youtube.com/watch?v=b-D00OO7KjY Karaoke – Singing evaluation: pitch, beat, enthusiasm, etc – Examples BMAT’s Skore TJ media V-scanner (Perfect Singer VS) 18

20. Media Production and Music Creation Sound Sample search – Imagine Research’s MediaMind: search sound effect sample for media production (e.g. film, drama) – Izotope’s Breaktweaker: search similar timbre of drum sounds Automatic Song writing – Algorithmic composition – Automatic arrangement MSR’s Songsmith: http://research.microsoft.com/en- us/um/redmond/projects/songsmith/http://research.microsoft.com/en- us/um/redmond/projects/songsmith/ 19

21. MIR Research Disciplines Digital Signal Processing Acoustics Music theory Machine Learning Natural language processing / Computer vision Psychology Human-Computer Interaction 20

22. About This Course Focus on inferring information from audio data Survey topics in MIR – Music and audio representations – Pitch tracking – Timbre analysis – Onset detection and beat tracking – Chord recognition – Source separation and polyphonic pitch – Music classification: genre, mood and tags – Music search and recommendation – Applications in music performance 21

23. Course Information Prerequisite – Basic digital signal processing: filters, FFT, spectral analysis – Programming: MATLAB – Basic music theory: scale, interval, major/minor chords – Strong interest in music Strong plus – Machine Learning: GMM, HMM, SVM, DNN, … No required textbook – Suggested readings or reference book lists are provided in the course web site. 22

24. Course Information Format – Lecture + Project (Proposal + Presentation + Report) Grading – Assignment: 50 % – Class participation: 10 % Discussions, attendance and enthusiasm – Final research project: 40 % Presentation (20%) and Report (20%) 23

25. Course Information Course website: http://juhannam.github.io/gct731mir2014/ http://juhannam.github.io/gct731mir2014/ 24