1. Correlational and Regressive Analysis of the Relationship between Tongue and Lips Motion - An EMA and Video Study of Selected Polish Speech Sounds
Robert Wielgat, Łukasz Mik, Polytechnic Institute, State Higher Vocational School in Tarnów,
Tarnów, POLAND
Anita Lorenc, Department of Speech Pathology and Applied Linguistics, Maria Curie-Sklodowska University
Lublin, POLAND
Department of Speech and Language Therapy and Voice Production, Warsaw University,
Warsaw, POLAND 1

2. Presentation Plan The presentation is divided into four parts:
About EMA and fundamentals of the acquisition system
Correlational and regressive analysis
Gathered data and their analysis
Conclusion and future research
Introduction
Problem Statement
Results and Discusion
Conclusion

3. Presentation Plan The presentation is divided onto four parts:
About EMA and fundamentals of the acquisition system
Correlation and regressive analysis
Gathered data and their analysis
Conclusion and future research
Introduction
Problem
Statement
Results and Discusion
Conclusion

4. Presentation Plan The presentation is divided onto four parts:
About EMA and fundamentals of the acquisition system
Correlational and regressive analysis
Gathered data and their analysis
Conclusion and future research
Introduction
Problem Statement
Results and Discusion
Conclusion

5. Presentation Plan The presentation is divided onto four parts:
About EMA and fundamentals of the acquisition system
Correlational and regressive analysis
Gathered data and their analysis
Conclusion and future research
Introduction
Problem Statement
Results and Discusion
Conclusion

6. Electromagnetic Articulography (EMA)
Electromagnetic Articoulography (EMA) – 3D imaging of speech articoulator movements (tongue, lips, palate, jaw).
The movement of articulators is visualised due to small sensors (coils) fixed to articoulators e.g. the tongue. Sensors are moving in an electromagnetic field produced by 6 transmitters.
Each transmitter produces an alternating magnetic field at different frequencies. The alternating magnetic field induces an alternating current in the sensors, and allows to obtain the distances of each sensor from six transmitters. It is then possible to calculate in the real time the XYZ coordinates as well as two angles of the sensors.
Introduction
Introduction
Preliminary
Research
Problem Statement
Current and Future Research
Results and Discusion
Summary
Conclusion

7. Block Diagram of the EMA System
Right Front Camcorder
screen
Introduction
Introduction
EMA Sensors
Problem Statement
Preliminary
Research
Central
Front
Cam-
corder
Left Front Camcorder
Computer
Results and Discusion
Current and Future Research
Frame Grabbers
Electromagnetic Articulograph
AG 500
Synchronizer
Summary
Conclusion

8. Articulograph AG 500 For the research purpose the AG-500 EMA was used.
Transmitter coils
12 sensors (small coils)
AG 500 EMA cube
calibrator
Introduction
Introduction
Problem Statement
Positions of the sensors were recorded every 5ms in the x, y, z space
The accuracy of the measurement is 0.5 mm
Results and Discusion
Future
Research
Summary
Conclusion

9. Location of EMA sensors
Introduction
Introduction Z X
Problem Statement
Results and Discusion
Future
Research
Summary
Conclusion
EMA sensor placement on the tongue, lips and mandible.

10. Location of EMA sensors
Introduction
Introduction Z X
Problem Statement
Results and Discusion
Future
Research
Summary
Conclusion
EMA sensor placement on the tongue, lips and mandible.

11. Location of EMA sensors
Introduction
Introduction Z X
Problem Statement
Results and Discusion
Future
Research
Summary
Conclusion
EMA sensor placement on the tongue, lips and mandible.

12. Vision System
Three camcorders recording the pictures simultaneously will be used.
B/W Camera parameters:
model: Point Grey Gazelle
GZL-CL-22C5M-C
Resolution: 2048 x 1088
Frame rate: 280 fps (200 fps)
Interface: Camera Link
Synchronization: via external trigger or software trigger
Good sensitivity at IR range
Introduction
Introduction
Problem Statement
Results and Discusion
Current and Future Research
Summary
Conclusion

13. Face Markers’ Placement
In order to capture the motion of the face, white dots (markers) were placed on the face of the speaker. Each is 4 millimeters in diameter.
7 reference markers - forehead, temples and nasal bridge
9 markers on lower jaw (mandible)
3 markers on beard
9 markers on zygomatic bone and cheeks
8 markers on lips
5 markers on the nose and nearby
1 marker on larynx
Introduction
Introduction
Problem Statement
Preliminary
Research
Results and Discusion
Current and Future Research
Current and Future Research
Summary
Summary
Conclusion

14. Calculation of the Markers’ Coordinates
Calculation of Markers’ Coordinates for the present research purpose was accomplished uisng only front face image. The method of calculation included three steps:
Image thresholding
Opening operation in order to remove small objects not being markers from the image
Finding centers of markers using the Circular Hough Transform (CHT)
Introduction
Introduction
Problem Statement
Preliminary
Research
Results and Discusion
Current and Future Research
Current and Future Research
Summary
Summary
Conclusion

15. EMA Sensor Placement in the Research
Introduction
Introduction
Problem Statement
Results and Discusion
Future
Research
5 sensors on the tongue
2 sensors on lips
1 sensor on the border of lower inscisors and gums
1 sensor for making palate contour
3 control sensors (placed on forehead and bones behind ears)
Summary
Conclusion

16. Pearson Correlation Coefficient
In order to find optimal conditions for determining of correlation between EMA sensor signals and signals representing video markers’ trajectories two video markers have been placed directly on two EMA sensors: for lower lip (sensor LL) and for upper lip (sensor UL). Pearson correlation coefficient was calculated which is defined as:
Introduction
Introduction
Introduction
Problem Statement
Results and Discusion
Current and Future Research
Current and Future Research
Summary
Conclusion
Summary

17. Regression Analysis
Introduction
Introduction
Introduction
Problem Statement
Results and Discusion
Current and Future Research
Current and Future Research
Conclusion
Summary
Summary
Regressive dependency between Z coordinates of sensors placed on the lower lip and tongue tip.

18. Analysed Speech Signal
Introduction
Waveform and spectrogram of the utterance ‘aca’ [at͡sa] extracted from the word ‘macanie’
Problem Statement
Results and Discusion
Conclusion

19. Influence of Delay of Acquisition Time on Correlation
The relationship between Z coordinate of the LL video marker and Z coordinate of the LL EMA sensor without head movement correction for the segment [at͡sa] from the word ‘płacami’.
Introduction
recorded simultaneously
Problem Statement
Results and Discusion
Delayed by 5 ms
Conclusion

20. Correlation between LL video marker and LL EMA sensor - Results
Pearson correlation coefficients between z coordinate of EMA sensor position and z coordinate of video marker position for different measurement conditions
Introduction
sensor TC
correlation coefficients for word No:
mean 89 97
137
261 LL no
-0,961
-0,926
-0,947
-0,964
-0,950
yes
-0,987
-0,962
-0,963
-0,986
-0,974 UL
-0,985
-0,938
-0,853
-0,892
-0,917
-0,990
-0,868
-0,931
-0,934
Problem Statement
Results and Discusion
Conclusion
Abbreviations in the Table: TC – time correction; LL – lower lip; UL – upper lip

21. Jitter of Reference Points
Euclidean distance between 2 reference EMA sensors during recording of single word. Reference points are on the nose bridge and the mastoid processes behing left and right ear.
Introduction
Problem Statement
Video reference markers
Results and Discusion
Conclusion
Euclidean distance between 2 reference video markers during a recording of a single word.

22. Sensor Tilt Z dZ2 dZ1 dX2 dX1 > dX2 dZ1 < dZ2 dX1 Introduction
Marker
dZ1
dX1
dZ2
dX2
dX1 > dX2
dZ1 < dZ2 X Z
Introduction
Problem Statement
Results and Discusion
Conclusion
Changes between EMA sensor and video marker distances in X and Z direction dependently on the sensor tilt

23. Dependency of LLz vs TTz
Introduction
Problem Statement
Results and Discusion
Conclusion
Regressive dependency between Z coordinates of lower lip (LL) and tongue tip (TT) EMA sensors for whole utterance `aca`

24. LLz vs TTz – phonetic segmentation
Introduction
Problem Statement
Regressive relationships between Z coordinates of lower lip (LL) and tongue tip (TT) EMA sensors for phonemes extracted from the utterance `aca` after phonetic segmentation
Results and Discusion
Conclusion

25. LLz vs TTz – articulatory segmentation
Introduction
Problem Statement
Results and Discusion
Conclusion
Regressive relationships between Z coordinates of lower lip (LL) and tongue tip (TT) EMA sensors for segments extracted from the utterance `aca` after articulatory segmentation

26. Conclusion Conclusion
delay time between EMA sensor signals and video recordings should be properly adjusted
strong correlational or regressive relationships exist between coordinates of visible and invisible EMA sensors
Future research
finding transformation of video marker position coordinates into the reference system related with skull bones
more comprehensive study on correlational and regressive relationships between inner and external sensors (markers).
Finding efficient methods of the articulatory segmentation.
Introduction
Introduction
Preliminary
Research
Problem Statement
Current and Future Research
Results and Discusion
Summary
Conclusion

27. Acknowledgment
The research was supported by The Polish National Science Centre grant Nr 2012/05/E/HS2/03770 titled „Polish Language Pronunciation. Analysis Using 3-dimensional Articulography”

28. Thank you for your attention