CS 445/656 Computer & New Media Music CS 445/656 Computer & New Media

Topics for Today Characterizing music Identifying music Music recommendation Music summarization Interactive music Composition support Automatic music composition Music management support

Characterizing Music From the audio signal From metadata Beat Brightness Pitch Starting note Potential key (music scale) From metadata Artist Genre(s) Album name, release date On-line ratings/comments Inclusion in playlists Lyrics

Identifying Music Music transcription Query by humming

Music Recommendation For an individual For a playlist For a group Cluster based on features and recommend music similar to what individual likes For a playlist Similar, but add constraints to fill particular amount of time Can also use song-to-song similarity to determine order of music For a group Find music that is most liked, least disliked, or some combination thereof For an activity Use mood or similar information

Music Summarization Most summaries in commercial sites are either the first phrase or a single selected musical phrase Study of whether 22 second long multi-phrase music summaries would be better previews Three algorithms compared that vary the selection of the components between phrases that are sonically distinct and phrases that are repeated more often A comparative evaluation study showed that: Users preferred multi-phrase previews in 87% of the cases over the preview representing the first 22 seconds of the song 90% of all of the multi-phrase summaries were considered at least as good representation of the song as the song’s starting segment

Interactive Music Music games MobiLenin Guitar Hero, Rock Band, … Enable interaction with music in a public space Not karaoke Voting like in many pub/bar games Audience can affect which version of music and video is shown

MobiLenin Interface and Process

Lessons Gave a focal point for interaction between members of a group Content variety is necessary for continued engagement Lottery for free beer motivated participation Well Duh!

Music Composition Support Lots of software tools that help composers generate new works Rapid editing and visualization are common features Some systems provide new models of composition PaperTonnetz

Automatic Music Composition Generation of a musical phrase by learning from examples The Continuator adds phrases in real time using a Markov model based on the artist’s style Example 1 Example 2 Automatic composition of details given higher-level structure of the desired music RapidComposer generates the details given a chord progression

Managing Personal Music Collections Music management is often based on: explicit attributes (e.g. metadata values like the artist, album, and genre). explicit feedback (e.g. ratings of preference and relevance) Benefits Easy to understand Formal: consistent updating and access Context-free Question How can music be accessed based on harder to express personal connections The feelings or memories it triggers? e.g. Music that sounds happy, makes us feel gloomy or reminds us of a person

Current Practices Common metadata tags usually not sufficient to describe mood, feelings, memories, etc. Effort/benefit trade-off Personal reactions change Explicit feedback and usage statistics helpful in retrieving music of preference How would people organize music if there was a low-effort way of expressing their personalized interpretation of music? Use of additional tags or customization of the existing ones can be tedious Use of additional ratings associated to specific music attributes can be overwhelming for the user

Preliminary Study 12 participants asked to organize songs & create playlists in spatial hypertext Visual attributes & spatial layout are changed to express associations The majority found spatial hypertext helpful for organizing Participants appreciated: expressive power and freedom of the workspace directly accessible metadata information of music music previews for remembering music Participants missed: interactive hierarchical / tree views based on metadata music previews for understanding music

Statistics of Artist Similarity Relatedness Assessment MusicWiz Architecture MusicWiz Interface Music management environment that combines: explicit information implicit information non-verbal expression of personal interpretation Two basic components: interface for interacting with the music collection inference engine for assessing music relatedness Metadata Module Audio Signal Module Lyrics Module Worksp. Express. Module Artist Module Relatedness Table Inference Engine Workspace Status Related Song Titles Music Collection Songs & Metadata Songs Lyrics Statistics of Artist Similarity Internet Relatedness Assessment Sim. Values

MusicWiz Interface Hierarchical Folder Tree View Workspace Playlist Pane Related Songs & Search Results View Folder Tree View: Provides a location-based hierarchical views of the music collection. Related Songs & Search Results View: Displays songs that are similar to the currently selected songs in the system tree view or the results of the search. Songs then can be dragged and dropped from the list into the workspace and the playlist pane to update collections and playlists respectively. Playback Controls The MusicWiz interface

MusicWiz Music Similarity Assessment 5 modules for extracting, processing and comparing artists, metadata, audio content, lyrics, and workspace expression Overall Similarity (S1, S2) = = W1 * Overall Metadata Similarity(S1, S2) + + W2 * Overall Audio Signal Similarity(S1, S2) + + W3 * Overall Lyrics Similarity(S1, S2) + + W4 * Overall Workspace Expression Similarity(S1, S2) where, S1, S2 are the songs under comparison and Wn, n = 1..4 the user adjusted weights of the specialized similarity assessments Each module produces an assessment of relatedness (a normalized value ranging from 0 – songs very dissimilar, to 1 – songs almost identical)

MusicWiz Inference Engine – Artist Module Assesses relatedness in music using online resources: human evaluations of artist similarity from: Similar Artists lists of the All Music Guide website co-occurrence of artists in playlists from: OpenNap file-sharing network Art of the Mix website Its output is used directly by the metadata module when comparing the artist name

MusicWiz Inference Engine – Metadata Module Evaluates the pair wise similarity of the metadata values of all songs String comparison is applied to the title, genre, album-name, and year of the songs as well as the file-system path where they are stored uses a distance metric that combines the Soundex and the Monge-Elkan algorithms The Soundex phonetic algorithm is valuable for identifying similarity between transliterated or misspelled names. It uses the six phonetic classifications of human speech sounds to convert the input into a string that identifies the set of words that are phonetically alike (similar pronunciation). The Monge-Elkan algorithm identifies similarity among expressions where the words are listed in a different order; it is a dynamic programming algorithm that calculates the distance of two strings based on the cost of transformations required to convert the first expression into the second expression.

MusicWiz Inference Engine – Audio Signal Module Uses signal processing techniques to analyze music content Extracts and compares information about the harmonic structure and acoustic attributes of music beat, brightness, pitch, starting note and potential key (music scale) of the song The greater the distance in the beat, brightness and pitch levels, the less likely songs are perceived as being of similar style or mood

MusicWiz Inference Engine – Lyrics Module Textually analyzes the lyrics Lyrics are scraped from a pool of popular websites for: display in music objects comparison Lyrical comparison uses term vector cosine similarity: Overall Lyrics Similarity (S1, S2) = cos(θ) The more words lyrics have in common, the greater the possibility that the songs are motivated by or describe related themes

MusicWiz Similarity Assessment – Workspace Expression Module Music objects can be related visually and spatially Spatial parser identifies relations between the music objects Recognizes three types of spatial structures: lists, stacks and composites Composite List Stack

MusicWiz Functionality Music collection can be explored by filtering: attribute values (i.e. id3 tags, audio signal attributes and lyrics) similarity values (i.e. overall similarity) Playlists can be created: manually: songs can be added from the left-side views & the workspace) automatically: filter - based mode: selection based on the ID3 tags similarity - based mode: selection based on the relatedness of songs on the current playlist

MusicWiz Evaluation 20 participants were asked to: Task 1: organize 50 rock songs into sub-collections according to their preference Task 2: form three, twenty-minute long playlists based on three different moods or occasions of their choice Task 3: form three six-song long playlists, where each of them had to be related to a provided “seed”-song (not from the fifty of the original collection)

MusicWiz Evaluation Configuration No Suggestions Suggestions Importance of the workspace Importance of the music previews Four groups of system use: Group 1 (no workspace / no suggestions) had to complete the three tasks using MusicWiz’s browsing, searching, and playback functionality and Windows Explorer folders to form the collections and playlists Group 2 (no workspace / with suggestions) used the same features as group 1 but also received suggestions from the similarity inference engine Group 3 (with workspace / no suggestions) had to perform the tasks using the features available in group 1 but used the MusicWiz workspace to create the collections and playlists Group 4 (with workspace / with suggestions) had all MusicWiz features Configuration No Suggestions Suggestions No Workspace Group 1 Group 2 Workspace Group 3 Group 4

Task 1 - Organization of Music Statement (1 – “I strongly disagree” to 7 – “I strongly agree”) Group 1 Group 2 Group 3 Group 4 The system support in organizing effortlessly / quickly was enough 4.4 5.4 5.6 6.2 Enjoyed doing task 5.8 6.4 6 Organization will be easily understood by others 4.2 Configuration No Suggestions Suggestions No Workspace Group 1 Group 2 Workspace Group 3 Group 4

Tasks 2 & 3 – Playlist Creation Statement (1 – “I strongly disagree” to 7 – “I strongly agree”) Task Group 1 Group 2 Group 3 Group 4 System support for quick selection was enough Two 4.8 6.2 5.8 Three 4.4 6.8 5.6 finding music 6 5.4 4.6 6.4 Enjoyed doing task 5.2 6.6 Configuration No Suggestions Suggestions No Workspace Group 1 Group 2 Workspace Group 3 Group 4

Music Overview Music processing can support a variety of activities Composition From traditional to interactive Selection & Recommendation Example: iTunes, Pandora, Use for shared spaces Playback Example: MobinLenin Management & Summarization Example: MusicWiz Games Guitar Hero, Rockband, etc.