1. Development of protocols WP4 – T4.2 Torino, March 9 th -10 th 2006

2. 2 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Presentation plan Planning and partners Definition Test material: what is needed for the evaluation test Evaluation criteria To do : define the protocols for both platforms

3. 3 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Calendar m18m24 m27 Dec.05 June06Sept.06 200720052006 M4.1D4.2 Specification of evaluation protocols T4.2 : Development of protocols  TRT (leader, 3 m*m)  Loquendo (2), TUC (2)  UGR (1), Loria (1), THAV (1) T4.3&4 Evaluation on the fixed and mobile platforms m29 Nov.06 M3.2 Functional integration on both platforms completed

4. 4 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 whathow Definition Evaluation protocol Defines precisely what must be evaluated, in which environment, what criteria are used and how to proceed. ex: wine tasting protocols The performance of the Hiwire recognition systems The integration quality on the fixed and mobile platforms >>> “Define the measures that will be applied during experiments in order to assess the performances of the vocal interaction system as well on a quantitative basis or on a more context dependent, qualitative basis.”

5. 5 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Test material (1/2) Test grammar One for each platform Vocabulary Number of commands Speech input Live speakers Who? (professional pilots, mechanics) Type of microphone (close-talking / multi-mic array) Real conditions simulation (added hangar noise through LPs) Recorded speech Hiwire database Sampling rate / quantification Mixed cockpit noise

6. 6 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Test material (2/2) Location A simulation room PDA Microphone + PtoT A cockpit simulator Graphical interface Microphone + VAD Panel Professional pilots, mechanics, … (both platforms) Hiwire database (fixed platform) Scenario A list of commands. Definition of the interaction (synthetic voice, vocal feedback)

7. 7 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Evaluation criteria (1/3) Objective measures WAC[0-100] % SAC, sentence accuracy[0-100] % CAC, command accuracy[0-100] % Response time# s Time between the end of speech and the system response Task completion rate TCR (+timeout)% of completed tasks  Plugged analyzer inside the system

8. 8 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Evaluation criteria (2/3) Subjective measures Usability Learning time*s Memorisation effort*[1-5] Easiness of use*[1-5] Workload Number of added tasks correctly achieved# Naturalness of the interaction[1-5] Acceptance level[1-5]  A form to fill at the end of the test session, subjective scales  Sensors heart pulsation EEG eyes movement

9. 9 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Evaluation criteria (3/3) Results Analysis Gathering objective data Transforming subjective data into a numerical form  Subjective scales  Comparison with WoOz  Comparison with non vocal text input Statistical features Average, standard deviation Classification

10. 10 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Summary: List of the protocol definition features Mobile platform Material Grammar Extended version Panel/ the users Colleagues 10 to 20 Location An equipped room, noise diffusion Factory noise  hangar noise (ask Airbus…) Different levels (from clean to ? dB, at the microphone capsule level) A test scenario The maintenance of aircrafts Fixed platform Material Grammar Thav grammar (provided at the end of April) Speech input Colleagues ~20 non native speakers (bad>good accent) Location The THAV cockpit simulator Multi-speaker noise diffusion system MM array A test scenario Depends on the grammar

11. 11 This document and any data included are the property of Thales. They cannot be reproduced, disclosed or used without Thales' prior written approval. ©THALES 2005. Template trtcoen version 1.0.2 Summary: List of the protocol definition features Mobile platform Criteria Objective measures Response time SAC TCR Subjective measures Easiness to use Naturalness of interaction Results analysis Comparison with text input / pen input system Fixed platform Criteria Objective measures SAC (avg and statistics through speakers) Response time Subjective measures … no pilot Comparison with the hiwire baseline Results analysis statistics through speakers