GCT731 Fall 2014 Topics in Music Technology - Music Information Retrieval Introduction to MIR Course Overview 1

Instructor Name: Juhan Nam ( 남주한 ) Biography – : BS in EE in Seoul National University – : Engineer in Young Chang (Kurzweil) – : PhD in Music (also MS in EE), CCRMA, Stanford University – : Research Engineer in Qualcomm 2

Outlines Introduction to MIR – Background Brief history of music technology Recent trends and future direction – MIR Music Data and Information Applications Course overview 3

Human, Music and Technology

History of Music Technology 5 Material Processing Technology – Mold metal and wood in a high-quality form – Improved or new musical instruments: e.g. piano, saxophone (in 1841) – 3D printer: resurrection of material processing technology?

History of Music Technology 6 Electro-Mechanical Technology – Microphone and speakers: sound as “continuous-time signal” – Amplifier and effects: loudness and timbre control – Recorder: paradigm shift in music creation and distribution – Player: listening to music anywhere – New musical instruments: electric guitars

History of Music Technology Digital Signal Processing + Computer Technology – A/D, D/A converters: sound as “discrete-time signals” – Synthesizers: “design” musical sounds – Digital audio workstation (DAW): music recording, editing and production – Audio programming: “coding” sound and musical events – MP3 players 7

Recent Trends Big data – Online music services: 20M+ songs – Youtube: 100h+ video uploaded per minute – Easy sharing of personal music content Intelligence – Interactive music notation: transcription, score following – Auto-accompaniment: playing with computers Connectivity – Music is combined with human data: play history, preference, location, etc. – Music is distributed via social networks 8

Future Directions 9 Challenges – How can we find music content in a human-friendly way? – How can we enhance our musical activity, specifically performance, by musical interaction with computers? – How can we benefit from the connection of human data and network with music? Need of understanding meaning in music!  Intelligent Data Processing + Internet Technology

Music Information Retrieval (MIR) Area of research that recently emerged in the background – International Society of Music Information Retrieval (ISMIR) since 2000 Aims to infer various types of information from music data – Making computer understand music as human does Provide intelligent solutions to enhance human musical activities 10

Information in Music 11 Instrument: Composer: Key: Similar songs (by motif) : Transcription – melody, score Mood: Melancholy, Sad, … - ELO “After all” - Radiohead “Exit Music” Chopin Piano E-minor Factual Information – track, artist, years Musical Information – timbre, melody, notes, beat, rhythm, chords, structure Semantic Information – genre, mood, user preference

Music Data Audio Data – wav or mp3 files, audio streaming from microphones Symbolic Data – MIDI files, Music XML Text Data – Lyrics, review, tags, blog Image Data – CD artwork, media, score (as image files) User Data (human data) – Play history, rate/preference 12

MIR Tasks Fingerprinting Cover song detection Music Transcription: melody, notes, tempo, chords Segmentation, structure, alignment Similarity retrieval, playlists, recommendation Classification: genre, mood, tags, … Query by humming Source separation: vocal removal Symbolic MIR: score retrieval or harmony analysis Optical Music Recognition (OMR) MIREX: 13

Applications of MIR Musical listening – Music search and recommendation Performance – Interactive music performance Education – Instrument learning Entertainment – Singing evaluation, Game Media production and music creation – Sound sample search in sound libraries 14

Music Search Query by music – Search a single unique song identified by the query – Audio fingerprint – Applied to movies, TV and ads, too Query by humming – Sing with humming and find closest matches – Melody match 15

Music Recommendation Personalized Radio – Generate Playlist – Based on user data, similarity and context 16

Music Performance Score-Following – Listen to performance and track the notes – Examples JKU: Tonara: Automatic accompaniment – Score following + Interactive Performance – Examples IRCMA’s Antefesco: Sonation’s Cadenza:

Music Education and Entertainment Focus on performance evaluation Learning musical instrument – Examples Ovelin’s GuitarBots: MakeMusic’s smartmusic: Karaoke – Singing evaluation: pitch, beat, enthusiasm, etc – Examples BMAT’s Skore TJ media V-scanner (Perfect Singer VS) 18

Media Production and Music Creation Sound Sample search – Imagine Research’s MediaMind: search sound effect sample for media production (e.g. film, drama) – Izotope’s Breaktweaker: search similar timbre of drum sounds Automatic Song writing – Algorithmic composition – Automatic arrangement MSR’s Songsmith: us/um/redmond/projects/songsmith/ us/um/redmond/projects/songsmith/ 19

MIR Research Disciplines Digital Signal Processing Acoustics Music theory Machine Learning Natural language processing / Computer vision Psychology Human-Computer Interaction 20

About This Course Focus on inferring information from audio data Survey topics in MIR – Music and audio representations – Pitch tracking – Timbre analysis – Onset detection and beat tracking – Chord recognition – Source separation and polyphonic pitch – Music classification: genre, mood and tags – Music search and recommendation – Applications in music performance 21

Course Information Prerequisite – Basic digital signal processing: filters, FFT, spectral analysis – Programming: MATLAB – Basic music theory: scale, interval, major/minor chords – Strong interest in music Strong plus – Machine Learning: GMM, HMM, SVM, DNN, … No required textbook – Suggested readings or reference book lists are provided in the course web site. 22

Course Information Format – Lecture + Project (Proposal + Presentation + Report) Grading – Assignment: 50 % – Class participation: 10 % Discussions, attendance and enthusiasm – Final research project: 40 % Presentation (20%) and Report (20%) 23

Course Information Course website: