2. Automatic and Data Driven Pitch Contour Manipulation with Functional Data Analysis Michele Gubian, Lou Boves Radboud University Nijmegen Nijmegen, The Netherlands Francesco Cangemi Laboratoire Parole et Langage University of Provence, Aix-en-Provence, France

3. 2 Outline Pitch Contour Manipulation  Context and problem  Sketch of proposed approach Use of Functional Data Analysis (FDA)  Case study  Data preparation  Functional PCA  Functional synthesis and listening Conclusions

4. 3 Context Languages can express oppositions using intonation  Question/Statement opposition in Neapolitan Italian QUESTIONSTATEMENT “Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?) What are the intonation cues that listeners use? Perceptual experiments where listeners judge stimuli whose pitch (F 0 ) contour has been manipulated  STEP 1: extract pitch contours from speech data  STEP 2: modify pitch contours  STEP 3: re-synthesize speech

5. 4 Pitch Contour Manipulation Use of an intonation model  Stylization Manual changes time F0F0 POSSIBLE IMPROVEMENTS Handle dynamic detail  Locally (e.g. concavity/convexity)  Long range correlation Derive useful variation modes directly and automatically from data

6. 5 A data driven approach Functional Data Analysis x

7. 6 Question/Statement opposition in Neapolitan Italian DATA 2 male speakers 3 carrier sentences (read speech)  “Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)  “Valeria viene alle nove (?)” = Valeria arrives at 9 (?)  “Amelia dorme da nonna (?)” = Amelia sleeps at grandma’s (?) 2 modalities = Q / S 5 repetitions 2 x 3 x 2 x 5 - 3 discarded = 57 utterances

8. 7 Data Preparation Sampled F 0 curves have to be turned into functions  A basis of functions (B-splines) expresses each original curve  Decide how much detail to retain (smoothing)

9. 8 Data Preparation (2) Landmark registration  Align points in time that are deemed as having the same meaning across the dataset

10. 9 Classic Principal Component Analysis (PCA) age2565 salary x x x x x x x x x x x x x x x x x xx x x x x x x x x x x x x x x x x x PC1 PC2

11. 10 Functional PCA

12. 11 PC-based signal reconstruction + 1.65 x - 0.46 x mean(t)PC1(t)PC2(t)

13. 12 Manipulated stimuli

14. 13 Conclusions A data driven approach is possible in the exploration of intonation phenomena FDA provides automatic tools to describe variation in a set of pitch contours extracted from real utterances  provided that the relevant landmarks are annotated The same tools allow to construct artificial contours with desired perceptual characteristics  Smooth and global variation are applied  Variations come from a statistical analysis of data  The process is automatic

15. 14