1. 1 Analysis and Synthesis of Pathological Vowels Prospectus Brian C. Gabelman 6/13/2003

2. 2 OVERVIEW OF PRESENTATION I. Background II. Analysis of pathological voices III. Synthesis of pathological voices IV. Summary

3. 3 What is this project? BACKGROUND Methods for modeling, analysis, and synthesis of pathological vowels incorporating novel approaches in: - System identification - Parameterization of non-periodic components (AM, FM, and noise) - Synthesizer designs for realtime and offline use What is a pathological vowel? May be caused by physical or neural problems. Characterized by substantial and complex NONPERIODIC signal components.

4. 4 Why do it? - Create a non-subjective basis to compare pathological voices for: 1. Improved diagnosis 2. Tracking changes in a patient’s voice - Generate voice samples with known levels of variations (noise, roughness, etc.) for: 1. Evaluation of model parameters 2. Evaluation of listener variability 3. Evaluation of importance of levels of pathological features. BACKGROUND What has been done before? - A well-established theory exists for NORMAL voices - Recent studies of pathological voices employ perturbation of normal features plus additive noise. - ES (external source) stimulation of the vocal tract to analyze formants (vowels) since 1942 for NORMAL voices.

5. 5 For the theoretical/analytical aspect of the project, an expression of the hypothesis of the dissertation in one sentence is: “By means of FM and AM demodulation techniques, estimation of nonperiodic features of pathological vowels may be improved.” BACKGROUND

6. 6 GLOTTAL SOURCE WAVEFORM (time domain) VOCAL TRACT FREQ. RESP. (freq domain) RESULTING VOICE SIG. (time domain) g(t) G(s) f(t) F(s) v(t) V(s) g(t) conv. f(t) = v(t) (time domain) G(s) x F(s) = V(s) (freq domain) ANALYSIS SOURCE - FILTER MODEL OF SPEECH

7. 7 Steps for analysis: PERIODIC ANALYSIS: 1. FORMANT DETERMINATION Uses LP (linear prediction) to model vocal tract as cascaded 2nd order digital resonators. External source testing is shown to augment or replace LP for pathological vowels (Inv). 2. SOURCE MODELING Uses inverse filtering and least squares optimization to fit source waveform to a standard model (LF). NONPERIODIC ANALYSIS: 3. ANALYSIS OF PITCH VARIATION Uses high resolution pitch tracking to measure detailed nonperiodic frequency variation. Variations are segmented into low and high frequeny FM with Gaussian form. ANALYSIS

8. 8 4. FM DEMODULATION Pitch variations are removed from original voice to achieve accurate noise estimation (step 7) (Inv.) 5. ANALYSIS OF POWER VARIATION Uses power tracking to measure detailed nonperiodic loudness variations. Variations are segmented into low and high frequency AM with Gaussian form. 6. AM DEMODULATION Power variations are removed from original voice to achieve accurate noise estimation (step 7) (Inv.) 7. ASPIRATION NOISE Frequency domain methods are used to separate aspiration noise component. The noise is spectrally modeled.[Gus de Krom] ANALYSIS

9. 9 COLLECTION OF VOICE SAMPLES VOICE ANALYSIS PARAMETERS (PITCH, NSR, FORMANTS,..) VALIDATION PERCEPTUAL COMPARISONS + - VOICE SYNTHESIS ANALYSIS BY SYNTHESIS

10. 10 ANALYSIS ANALYSIS/SYNTHESIS MODEL OVERVIEW SOURCE WAVEFORM FM MODULATION AM MODULATION SPECTRAL SHAPING GAUSSIAN NOISE VOCAL TRACT OUTPUT VOICE PERIODIC NONPERIODIC + + SYNTHESIS ANALYSIS

11. 11 MIC SIGNAL LPC FORMANT ANALYSIS / MANUAL OPS JITTER ESTIMATE. TREMOR TIME HIST INVERSE FILTERING RAW FLOW DERIVATIVE SRC NOISE SPECTRUM RESAMPLE TO REMOVE TREMOR SYNTHESIZER FORMANTS FITTED LF SOURCE PULSE LEAST SQR LF FIT PITCH TRACKER CEPSTRAL NOISE ANALYSIS NSR ESTIMATE CONST. PITCH VOICE PITCH TIME HIST SHIMMER ESTIMATE VOLUME TIME HIST POWER TRACKER POWER TIME HIST RESAMPLE TO REMOVE TREMOR CONST. POWER VOICE MIKE SIGNAL ANALYSIS OVERVIEW OF PROJECT OPERATONS

12. 12 UNKOWN SYSTEM PREDICTOR u(n)s(n) s’(n) + - e(n) ESTIMATED INVERSE SYSTEM H(z) A(z) G/H(z) LINEAR PREDICTION (SOURCE) (VOCAL TRACT) Estimates the vocal tract as an all-pole filter by minimizing the error between actual and model-predicted signals. (MODEL) (ERROR) PERIODIC ANALYSIS

13. 13 IDEALIZED LP RESULT Requires a priori knowledge of system. More difficult for pathological vowels. PERIODIC ANALYSIS

14. 14 SOURCE TIME SERIES VOCAL TRACT TRANSFER FUNCTION VOICE TIME SERIESVOICE SPECTRUM SOURCE TIME SERIES VOCAL TRACT TRANSFER FUNCTION VOICE TIME SERIESVOICE SPECTRUM SOURCE TIME SERIES VOCAL TRACT TRANSFER FUNCTION VOICE TIME SERIESVOICE SPECTRUM CASE 0: NORMAL SOURCE AND NORMAL VOCAL TRACT CASE 1: “BREATHY” SOURCE AND NORMAL VOCAL TRACT CASE 2: NORMAL SOURCE AND ABNORMAL VOCAL TRACT SOURCE-FILTER AMBIGUITY Source & filter are mixed in final voice. Unique LP solution may be difficult. PERIODIC ANALYSIS

15. 15 00.0020.0040.0060.0080.010.012 -150 -100 -50 0 50 100 150 T (SECONDS) FLOW U(t) & DERIVATIVE E(t) E1 (FIRST SEG.)E2 (SECOND SEG.) U(t) 30*E(t) tp (te,Ee) tc (t2,Ee/2) (tpk,Epk) FITTING RAW SOURCE TO LF Having established the inverse filtered source, it is fit to the LF model [Qi] PERIODIC ANALYSIS

16. 16 HIGH RESOLUTION PITCH TRACKING Nonperiodic analysis begins with interpolating pitch tracking. NON-PERIODIC ANALYSIS 00.20.40.60.81 255 260 265 270 275 FREQ (HZ) TIME (SEC) VOICE SIGNAL PITCH TRACK

17. 17 FM DEVIATION SEGREATED TO LOW AND HIGH FREQUENCY The pitch track is low/hi pass filtered to yield tremor and HFPV (High Frequency Pitch Variation). NON-PERIODIC ANALYSIS

18. 18 -1.5-0.500.511.5 0 5 10 15 20 25 30 35 40 45 DELT FREQ (%) TOTSKIP=0 FCUT=10 TOTMAN=0 #OCCURRANCES GAUSSIAN HFPV Successful pitch tracking yields a Gaussian distribution in HPFV. The standard deviation is a convenient measure of HFPV. NON-PERIODIC ANALYSIS

19. 19 FM DEMODULATION The pitch track may be used to demodulate the original voice to obtain a version with almost no pitch variation; re-tracking verifies constant pitch (<0.1%). NON-PERIODIC ANALYSIS 00.20.40.60.81 -0.1 -0.05 0 0.05 0.1 DELTA FREQ PERCENT TIME (SEC)

20. 20 00.511.522.533.54 x 10 4 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SAMPLE # POWER POWER TRACKING Analogously to pitch tracking, voice power is tracked. NON-PERIODIC ANALYSIS POWER TRACK

21. 21 POWER SEGREGATED TO LOW AND HIGH FREQUENCY The power track is low/hi pass filtered to yield low frequency power variations and high frequency shimmer. NON-PERIODIC ANALYSIS

22. 22 -20-15-10-5051015 0 5 10 15 20 25 30 35 40 DELT PWR (%) #OCCURRANCES SHIM% HIST. STAND DEV = 4.823% GAUSSIAN POWER VARIATIONS Shimmer also displays Gaussian variations. NON-PERIODIC ANALYSIS

23. 23 AM DEMODULATION The power track may be used to demodulate the original voice to obtain a version with almost no variation in strength; re-tracking verifies constant power. NON-PERIODIC ANALYSIS

24. 24 ASPIRATION NOISE ANALYSIS Aspiration noise is segregated via spectral techniques. Peaks in the FFT of the log of the FFT (cepstrum) represent periodic energy, and are filtered out with a comb filter (lifter). Results are used to calculate noise-to-signal ratio (NSR). NON-PERIODIC ANALYSIS

25. 25 FM DEMODULATION IMPROVES NSR ACCURACY Using FM demodulation improves resolution of spectral peaks of periodic components, thus allowing longer FFT windows and more accurate NSR determination. NON-PERIODIC ANALYSIS 0100200300400500600700800 0 1 2 3 4 5 6 7 FREQUENCY (HZ) LOG10 MAGNITUDE

26. 26 CHANGES IN NSR AFTER FM AND AM DEMODULATION FM demodulation reduces NSR measures by up to 20 dB, yielding results closer to perceived levels. AM demodulation has much less effect. NON-PERIODIC ANALYSIS ORIG ALL FM REMOVED TREMOR REMOVED

27. 27 PC #1: STIMULUS GEN. STIMULUS SIGNAL D/A AMPLIFIER XDUCER /a/ ACOUSTIC CONDUIT PC #2: DAQ SIGNAL CONDITIONER A/D MEM. MICROPHONE CH 1 CH 2 SIGNAL CONDITIONER VOCAL TRACT TUBE OF LENGTH L WITH CLOSED END. FORMANTS AT F = C/4L, 3C/4L, 5C/4L,... EXTERNAL (ES) SOURCE ANALYSIS Source-filter ambiguity may be resolved by augmenting the glottal source with an known external stimulus. [Epps]. PERIODIC ANALYSIS

28. 28 ES VERIFICATION A simple plastic tube model verified the ES experimental setup. Resonances occur at expected frequencies. PERIODIC ANALYSIS

29. 29 ES: NORMAL /a/ LP & FFT analysis show consistent results with ES analysis for a normal vowel. PERIODIC ANALYSIS

30. 30 ES: SIMULATED BREATHY /a/ LP & FFT analysis show poor resolution for F3 and F4 for a breathy /a/, while the ES resolution for F3 and F4 remains good. PERIODIC ANALYSIS

31. 31 SYNTHESIS SYNTHESIS OF PATHOLOGICAL VOWELS Synthesis is a critical step in the study of pathological vowels. It provides evidence of the success of analysis and modeling steps via immediate comparisons of original and synthetic voice. Two synthesizers were implemented: 1. A realtime hardware-based synthesizer capable of providing instant response to changes in model parameters. 2. A software synthesizer implemented in MATLAB with extended features, convenient graphical interface, and ease of modification.

32. 32 LOW PASS FILT. CURRENT REALTIME SYNTHESIZER FUNCTIONAL OVERVIEW D/A AGC 4 ZEROES... 20 POLES GAH ASP NOISE OUTPUT SIGNAL FILE RECORD WAV STORE/RECALL CONTROL PARMS PARAMETER TIME VARIATION SOURCE SELECTION SOURCE CONTROLS JITTERSHIMMERDIPLOPHONIA CLK X86 INTERUPT AMP IMPL KGLOT 88 LF1 LF2 ARB ARBITRARY SOURCE FILE SPKR + SYNTHESIS REALTIME SYNTHESIZER - Implemented in native X86 assembly language - Executes all code within 100us cycle - Overrides PC OS to achieve determinancy - Employs dedicated clock, I/O, and control hardware implemented in a wire-wrap PCB adapter card

33. 33 SYNTHESIS SYNTHESIS VALIDATION The current model, analysis tools, and synthesizers yield a high level of fidelity in generation of synthetic pathological vowels. The system is currently employed at the UCLA Voicelab for NIH funded perceptual studies. In order to objectively validate the analysis/synthesis process, the loop is closed by re-analyzing the synthetic time series to confirm parameter values. Re-analysis also provides opportunity to observe interactions of nonperiodic components.

34. 34 SYNTHESIS SYNTHESIS VALIDATION Re-measured synthetic aspiration noise level agrees with level set in synthesizer.

35. 35 SYNTHESIS SYNTHESIS VALIDATION Aspiration noise adds about %0.2 to HFPV measurements. 00.511.52 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 JITTER LEVEL SET IN SYNTHESIZER (%) JITTER MEASURED IN SYNTHETIC (%) A.N. SET NSR IN SYNTHESIZER (dB) -30-25-20-15-10-50 -30 -25 -20 -15 -10 -5 0 MEASURED NSR IN SYNTH (dB) HFPV adds about 4dB to NSR measurements.

36. 36 SYNTHESIS SYNTHESIS VALIDATION Subjective analysis by synthesis experiments demonstrate the success of AM and FM demodulation in achieving accurate modeling of nonperiodic features. Listeners adjust synthetic aspiration noise to match original. Match improves with demodulation -30-25-20-15-10-50 -30 -25 -20 -15 -10 -5 0 SABS ASPIRATION NOISE (DB) CEPSTRAL NSR (DB) PEARSON = 0.51 ORIGINAL VOICE (NO DEMOD)

37. 37 SYNTHESIS SYNTHESIS VALIDATION

38. 38 SUMMARY CONCLUSION This study has achieved improved automatic, objective analysis and synthesis of speech within the specialization of pathological vowels. Specific accomplishments include: - A unique, symmetric model for nonperiodic components as AM, FM and spectrally-shaped aspiration noise - Improved accuracy of noise analysis via AM & FM demodulation - Application of ES formant identification for pathological vowels. - Implementation of realtime and offline specialized high fidelity vowel synthesizers