1. Development of protocols WP4 – T4.2 TRT Nancy review July 6 th & 7 th 2006

2. 2 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Planning What was decided in Torino What has been already done What remains to do

3. 3 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Calendar m18m24 m27 Dec.05 June06Sept.06 200720052006 M4.1D4.2 Specification of evaluation protocols T4.2 : Development of protocols  TRT (leader, 3 m*m)  Loquendo (2), TUC (2)  UGR (1), Loria (1), THAV (1) T4.3&4 Evaluation on the fixed and mobile platforms m29 Nov.06 M3.2 Functional integration on both platforms completed  

4. 4 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Torino’s meeting Mobile platform Environment Grammar Extended version Panel/ the users Colleagues 10 to 20 Location An equipped room, noise diffusion Factory noise  hangar noise (ask Airbus…) Different levels (from clean to ? dB, at the microphone capsule level) A test scenario The maintenance of aircrafts Criteria Objective measures Response time SAC TCR Subjective measures Easiness to use Naturalness of interaction Results analysis Comparison with text input / pen input system Fixed platform Environment Grammar Thav grammar (provided at the end of April) Speech input Colleagues ~20 non native speakers (bad>good accent) Location The THAV cockpit simulator Multi-speaker noise diffusion system MM array A test scenario Depends on the grammar Criteria Objective measures SAC (avg and statistics through speakers) Response time Subjective measures … no pilot Comparison with the Hiwire baseline Results analysis statistics through speakers

5. 5 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Evaluation protocol Test Environment Panel Test space Equipment loudspeakers, sound card, PC, connection, sound data, HSP=PDA or Cockpit simulator display Software gathering results, spectral and volume calibration Grammar Evaluation criteria Objective measures Subjective measures Result analysis Test scenario Preparation Test training Test After test

6. 6 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Evaluation protocol Test Environment Panel Test space Equipment loudspeakers, sound card, PC, connection, sound data, HSP=PDA or Cockpit simulator display Software gathering results, spectral and volume calibration Grammar Evaluation criteria Objective measures Subjective measures Result analysis Test scenario Preparation Test training Test (depends on the grammar) After test

7. 7 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Panel 20 non-native English speakers M/F ratio = 50% Well-balanced origin ratio (France, Greece/Creete, Italy, Spain) Various English levels Names?

8. 8 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Test space (fixed pltf.) D WLS W L = 12m W = 7m H = 3m D WLS >1.5m D LS >1.5m Command, control and storage area + HSR LS-left Cockpit Simulator Multi-mic device controller + HSR D LS L Multi-mic array LS-right PPT Cockpit Simulator Display Unit PC KCCU  Problem:  we do not create a realistic sound field for the multi-mic array  solution: we will record noise in a real cockpit and test different multi LS diffusion (4, 6, 12, ?)  an acoustic distance is necessary to be computed to find the closest LS placing

9. 9 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Test space (mobile pltf.) LS-left LS-right LS+stand test area tester+micro+PDA test area Command, control and storage area lectern W H upper view side view L = W = H = D WLS >1.5m D LS >1.5m H LS = 1.75m lectern D WLS D LS H LS L  Where?  A real hangar would be the best place to have a realistic evaluation but it is difficult to control the noise level in such a place

10. 10 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Equipment Loudspeakers sufficient power for a 300m 3 room  power >100 Watts wide frequency range flat frequency response  YAMAHA HS80M (x2) 39 cm  = 11 kg

11. 11 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Equipment Soundcard mini 20bits / 48kHz SNR > 100dB balanced outputs digital PC link  Behringer F-CONTROL AUDIO FCA202  24bits / 96kHz  SNR = 100dB  Fire Wire PC link Connection PC to soundcard FireWire cable (x1) LS to soundcard Balanced XLR-Jack cable (x2)  The sound card can be replaced by an ITC’s.

12. 12 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Equipment Sound data Sources Cockpit noise Hangar noise (Aircraft tests, workshop, …)  possible recordings in Le Bourget Format WAV PCM 20bits / 48kHz Computer A control PC (laptop) to launch sounds to gather results

13. 13 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Software Objective results gathering Plug-in inside the HSR PC for the fixed pltf. For the mobile pltf. the plug-in is on the control PC, a link between PC and PDA is needed. Subjective results gathering On the control PC (excel sheet) for the mobile pltf. LS spectral compensation and volume calibration

14. 14 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Fixed platform grammar

15. 15 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Mobile platform grammar Aircraft maintenance large place with high sound level differences general observations around the plane before first flight (wheels, windows, oil, gaz, …motor tests)  noisy environment BUT simple grammar (Yes/no answers) Repairs in workshop little quiet workshops Specific unplugged aircraft pieces  large grammar BUT not noisy environment The grammar must present a sufficient variability so it will be the second one.

16. 16 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Mobile platform grammar  Exemple

17. 17 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Criteria Objective measures Response time, RT (s) Time between the end of speech and the system response It must be the lower as possible Sentence Accuracy, SAC (%) Ratio between the number of correct sentences and the total of pronounced sentences It must ideally be 100% Word accuracy, WAC (%) Content word accuracy, CWA (%) Only the words in connection with the topic (non link words) Task Completion Index, TCI (#) Average number of repetitions before the sentence is understood It must be near zero meaning that the sentence is immediately understood (no repetition needed) Time Before Completed Task, TBCT (s) Time between the end of the speech first try and the system good response Lower is better

18. 18 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Filled form Objective measures of one user test (example for mobile pltf.)

19. 19 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Criteria Subjective measures Easiness of use, EASE [1-5] Describes how the tester found the easiness of entering information from “not easy” (1) to “very easy” (5) Can be correlated with the user cognitive workload, the number of actions before accessing to the information, the clarity of the visual interface and the synthetic voice. Naturalness of interaction, NAT [1-5] Gives an indication about way it is natural to use the system from “not natural” (1) to “very natural” (5) Can be correlated with intuitiveness: the closer to human-human interaction it is, the more natural it is Slightly differs from EASE: a system can be easy to use (because everything is clearly presented) but not natural because of a bulky device or because the interface (voice or pen) is not adapted. Completed task performance perception [1-5] Efficiency [1-5] Overall assessment [1-5]

20. 20 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Scenarii Mobile platform Two session modalities: vocal input (VI) and pen input (PI). Two SNR: “clean” (no noise), noisy : the background noise level is adjusted at the same level as in the hangar Some maintenance reports to fill (5 + 1 for training) Visual feedback on the PDA Fixed platform Two kinds of session: with close-talking microphone (Plantronics or AT?) and with the multi-microphone array. Four SNR: “clean” (no noise), noisy taxi and takeoff Comparison with baseline. A list of commands to pronounce Visual feedback

21. 21 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Result analysis Global result table (mobile plft.)

22. 22 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Result analysis Global result table (fixed plft.)

23. To-do list

24. 24 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 To-do Redaction Write a short paragraph for the IP Protection of the evaluation protocol Put presentation on Twiki Justify ICASSP attending Recording into a hangar Do not need to have a multi-mic Stereo recording / diffusion is enough Send the aircraft part list to LOQ

25. 25 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 To-do Recording into a cockpit 6 + 1 Shure microphones (left, right, back, forward, up, down) At least 2 microphones must be at the same distance as in the multi-mic array Need to access to a cockpit and stick the 7 microphones Recording during taxiing, take-off, en route and landing Need a self-alimentation for the devices (octamic + soundcard + laptop)

26. 26 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 To-do Measuring evaluation results Fixed platform: In the HSR PC Mobile platform: In the monitoring PC It is better to keep the scenarii in the monitoring PC (easy to modify, …) Need a link between the PDA and the PC

27. 27 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 To-do Global evaluation Propose common measures Dominique prefers WER and SER  ask all the partners Push for using the Hiwire database Need to add cockpit noise artificially (UGR) Different noise levels artificial noise