Melodic Features and Retrieval ISMIR Graduate School, Barcelona 2004 Musicology 3-4 Frans Wiering, ICS, Utrecht University

Outline yesterday’s assignment demo: MIR outside academia (7:20; 44:10) one-dimensional melody retrieval Gestalt view of melody advanced melody retrieval assignment

one-dimensional melody retrieval common assumption is (was?) pitch-only retrieval is sufficient  e.g. CCGGAAGGFFEEDDEC  mechanisms for fuzzy matching variants  interval (distance between 2 pitches)  pitch-contour same/up/down (Parson’s Code) RURURDRDRDRDRUD examples:  (Rainer Typke)  (CCARH)

Results from Musipedia query is ranked 3 other hits are very unlikely  unfortunately no notation/sound available Haydn: evident false positive  why?

Themefinder Several 1-dimensional search options, e.g.  pitch  interval  contour  rhythm wildcards each theme stored as a number of strings matching by regular expressions ca themes  Barlow and Morgenstern (1948)  ESAC encodings  Lincoln, 16 th Century Motet (DARMS project)

results from Themefinder Example from Byrd & Crawford (2001) other hits  not as far-fetched as musipedia’s  different rhythm  different meter  still not very similar is this what people have in mind? Query: +m2 +M2 P1 -M2 -m2 -M2

Nice one we’ve just discovered Pitch-only search of MIDI on the internet many more MIR systems in Rainer Typke’s survey. URL is in your mailbox

Why pitch-only retrieval is unsatisfactory information contribution of other 3 parameters (estimate for Western music; Byrd & Crawford 2001)  pitch: 50%  rhythm: 40%  timbre + dynamics: 10% melodic confounds (Selfridge-Field 1998):  rests  repeated notes  grace notes, ornamentation  Mozart example

Why pitch-only retrieval is unsatisfactory information contribution of other 3 parameters (estimate for Western music; Byrd & Crawford 2001)  pitch: 50%  rhythm: 40%  timbre + dynamics: 10% melodic confounds (Selfridge-Field 1998):  rests  repeated notes  grace notes, ornamentation  Mozart example

Gestalt and melody melody: coherent succession of pitches  from New Harvard Dictionary of Music coherence important for similarity: creates musical meaning  bottom-up (pitches and durations)  top-down: segmenting, Gestalt Gestalt theory of perception  late 19 th /early 20 th century, Germany, later US  perception of wholes rather than parts  explanations: Gestalt principles of grouping  application in visual and musical domain

Low-level Gestalt principles Snyder mentions:  proximity rhythmic intervallic  similarity duration articulation  continuity melodic these produce closure of wholes Example: Beethoven 5 th symphony: beginning 1 st movement  also illustrates high-level principles from Snyder (2001)

Low-level Gestalt principles Snyder mentions:  proximity rhythmic intervallic  similarity duration articulation  continuity melodic these produce closure of wholes Example: Beethoven  also illustrates high-level principles from Snyder (2001)

High-level Gestalt principles parallellism  very strong in Mozart, Ah vous, second half of melody intensification  important organisational principle in variations and improvisations  Mozart’s last variation from Snyder (2001)

Application in analysis and retrieval Gestalt reduces memory overload: we can ignore the details Analytical: Schering (1911)  14 th century Italian songs  basic melodic shape  might be nice for retrieval Problem with Gestalt principles:  many different formulations  overlap; no rules for conflict  intuitive, cannot be successfully formalized from New Grove, Music analysis

The cognitive interpretation: chunking what creates a boundary  interval leap  long duration  tonality (stable chords)  etc Example of quantification: Melucci & Orio (2004)  using local boundary detection (Cambouropoulos 1997) apply weight to intervals and durations boundary after maximum  chunks forther processed for indexing

Organising chunks STM problem: max. 5-7 different elements  very short span solution: hierarchical grouping melody schemas  contours of melody cf. Schering ex.  examples: axial, arch, gap- fill  Mozart begins with gap-fill next level: form  A-B-A from Snyder (2001)

mental model of a song Ah, vous dirai-je maman melody level phrase level chunk level subchunk level A A B analysis synthesis analysis: from ear to LTM  (sub) chunks created by similarity and continuity a lot of parallellism  boundaries by leaps and harmony chunks may have a harmonic aspect too (I, V, V->I) synthesis: from LTM to focus of attention  recollection using general characteristics of phrases and chunks  performance notes are reconstitued through some musical grammar

Problems of melody retrieval People remember high-level concepts, not notes  often confused with poor performance abilities  theme-intensive music (fugues) stimulate formation of such concepts melodic variability and change  transposition  augmentation/diminution  ornamentation  variation  compositional processes: inversion, retrograde other factors  polyphony  harmony

Set-based approaches to melody retrieval in polyphony General idea:  compare note sets: find supersets, calculate distance  usually take rhythm and pitch into account  hopefully more tolerant agains some of the problems of melodic variety Clausen, Engelbrecht, Meyer, Schmidt (2000):  PROMS  matches onset times; wildcards  elegant indexing Lemström, Mäkinen, Ukkonen, Turkia (several articles, )  C-Brahms  algorithms for matching line segments P1: onsets P2: partial match onset times P3: common shared time  attention to time complexity Typke, Veltkamp, Wiering ( )  Orpheus system

Earth Mover’s Distance The Earth Mover’s Distance (EMD) measures similarity by calculating a minimum flow that would match two set of weighted points. One set emits weight, the other one receives weight Y. Rubner (1998); S. Cohen (1999)

Application to music represent notes as weighted point sets in 2-dimensional space (pitch, time) weight represents duration  other possibilities contour/metric position etc other possible application: pitch event + acoustic feature(s)? here, the ‘earth’ is only moved along the temporal axis

Another example interesting properties  tolerant against melodic confounds  suitable for polyphony  continuous  partial matching disadvantage  triangle inequality doesn’t hold  less suitable for indexing: after alignment, the ‘earth’ is moved both along the temporal axis and along the pitch axis

Test on RISM A/II

Matching polyphony with the EMD EMD’s partial matching property is essential MIDI example used as query for RISM database gross errors in playing are ironed out

Proportional Transportation Distance (PTD) Giannopoulos & Veltkamp (2002) EMD, weigths of sets normalised to 1 suitable for indexing  triangle inequality holds no partial matching

Test on RISM A/II only hits with approximately same length need 4 queries to find all known items

False positive (EMD) problems arise when length and/or number of notes differs considerably

Segmenting overlapping segments of 6- 9 consecutive notes not musical units search results are combined better Recall-Precision averages

Example of new search

Concluding remarks about melodic retrieval lots of creativity go into melody; difficult to give rules  not a ‘basic musical structure’ (Temperley 2001) essential to use multiple features  pitch, rhythm  harmony segmentation  finding perceptually relevant chunks is not easy  finding complete melodies may be harder  arbitrary segments may also work indexing strategies for melody melodic change over time several projects have tentative results for polyphony  gut feeling: false positives are big issue  notion of salience (Byrd and Crawford)