1. Modeling Expressive Performances of the Singing Voice Maria-Cristina Marinescu (Universidad Carlos III de Madrid) Rafael Ramirez (Universitat Pompeu Fabra)

2. voice style timbre interpretation musicality resonance technique color fluidity The human singing voice

3. voice style timbre interpretation musicality resonance technique color fluidity Acoustic Features: Pitch Timing Timbre Articulation Spectral energy distribution Verbal Features: Intonational phrasing The human singing voice

4. Our long-term goal Develop models of operatic singers …and generate expressive performances similar in voice quality and interpretation Why opera? –Constrained environment (score, libretto) makes comparison between singers and classification of expressive content easier –Better voice and technique make singers more effective in performing expressively 1.Entertainment tool – generate interpretations of songs never recorded 2.Learning tool – use by professionals to learn different aspects of expressive singing 3.Re-mastering old records 4.Understand evolution of classical singing in terms of impact and inspiration of singers

5. Our long-term goal Develop models of operatic singers …and generate expressive performances similar in voice quality and interpretation

6. In this work Develop models of operatic singers …and generate expressive performances similar in voice quality and interpretation Acoustic Features: Pitch Timing Timbre Articulation Spectral energy distribution

7. Timing models of expressive performance commercial audio recordings high-level descriptors singer-specific models extract train Properties of musical events Musical event = note Inter-note features – context of musical events Learn predictions for note duration based on h-l descriptor patterns. expressive performance evaluate Predict duration of each note in test melody. How close are performed / predicted values?

8. High-level descriptors Characterize the melody based on: –Score –Performance

9. High-level descriptors Characterize the melody based on: –Score: Note properties: pitch, duration, meter strength, note density EH M L H

10. High-level descriptors Characterize the melody based on: –Score: Note properties: pitch, duration, meter strength, note density Context: neighbours’ relative pitch and interval length, Narmour structures

11. High-level descriptors Characterize the melody based on: –Score: Note properties: pitch, duration, meter strength, note density Context: neighbours’ relative pitch and interval length, Narmour structures –Performance: note onset and duration OUT

12. High-level descriptors Characterize the melody based on: –Score: Note properties: pitch, duration, meter strength, note density Context: neighbours’ relative pitch and interval length, Narmour structures –Performance: note onset and duration –Score + performance: actual tempo

13. Timing models of expressive performance – extracting descriptors commercial audio recordings high-level descriptors singer-specific models extract train expressive performance evaluate Score pitch, duration: manually Note onset and duration: sound analysis techniques based on spectral models Meter strength: automatically computed based on note length Note density: manually computed for whole melody Narmour structure: automatically generate I/R analysis based on score Actual tempo: manually computed based on score and melody duration.wav files

14. Timing models of expressive performance – learning and validation commercial audio recordings high-level descriptors singer-specific models extract train expressive performance WEKA – model trees: - Generate decision list for regression problem using separate-and-conquer - Build model tree and make best leaf into rule – test set evaluation: - Predict note durations for test cases evaluate

15. Our data set 6 tenor arias by Verdi – 415 notes 1.Operatic 2.A cappella 3.Consistent composition style 4.Consistent interpretation style 5.Live

16. Our data set 6 tenor arias by Verdi – 415 notes 1.Operatic 2.A cappella 3.Consistent composition style –Verdi’s middle years (1840-1855) –… maybe with exception of Rigoletto 4.Consistent interpretation style 5.Live

17. Our data set 6 tenor arias by Verdi – 415 notes 1.Operatic 2.A cappella 3.Consistent composition style –Verdi’s middle years (1840-1855) –… maybe with exception of Rigoletto 4.Consistent interpretation style –Josep Carreras mid 70s – beginning 80s 5.Live

18. Experimental results – training and testing per aria Predict note duration as a function of h-l descriptors Compare predicted with actual (performed) Training/testing task: 80% percentage split evaluation Aria Correl coeff Ella mi fu rapita! 0.74 Un di, felice, eterea 0.72 De miei bollenti spiriti 0.46 Forse la soglia attinse 0.57 Oh fede negar potessi! 0.29 La pia materna mano 0.20 All arias (4/4) 0.56 no correlation Widely diverse actual tempos and note densities! 110 / 89 / 54 / 76 / 59 4.62 / 5.66 / 3.6 / 3.47 / 4.71

19. 1 2 3 4 5 6 7 8 Note number Relative duration 0 1 2 3 2.5 1.5 0.5 Actual vs. Predicted - Ella mi fu rapita! Actual Predicted What is the extent to which the model chooses the same time transformation for a note as the singer does?

20. Experimental results – validation of the model Validation task: test set evaluation of each 4/4 aria against learned 4/4 model Aria Correl coeff Ella mi fu rapita! 0.62 De miei bollenti spiriti 0.57 Forse la soglia attinse 0.94 Oh fede negar potessi! 0.61 La pia materna mano 0.80 Larger data set strengthens the model! From statistically irrelevant 0.29 / 0.2 Down from 0.74

21. Relative duration Note number Actual Predicted Actual vs. Predicted - Forse la soglia attinse 0 1 2 3 4 5 6

22. Work in progress and future work Add more arias Add more acoustic features to singer-specific model –Currently: energy model –Future: timbre, exagerration, pitch range, accent shape, articulation Add more input parameters –Currently: syllable information (open/closed, stressed/unstressed) –Future: intonational phrasing

23. Questions?