Jared Bernstein Elizabeth Rosenfeld Pearson KT Tasso Partners 161st ASA Meeting Seattle, Washington Paper 5pSC6 Measuring Linguistic Performance of a Hand-held Speech-to-Speech Interpreter Jared Bernstein Elizabeth Rosenfeld Pearson KT Tasso Partners Palo Alto, California

The Problem In 2011, voice-in/voice-out interpretation is available for mobile phones – sometimes running remotely on a server, but we evaluate a software system that runs inside the iPhone4. Current systems work surprisingly quickly, but we do not have standard comparison measures for component accuracy. If portable devices can support bidirectional communication that works in spoken exchanges for commercial and travel purposes, how will that impact the future of second language education?

Approach The study reports the accuracy of the ASR and text translation of one such hand-held system operating in quiet environments (offices and private houses). The device itself allows examination of its intermediate results, and we analyze its performance to estimate the level of communicative function it can support. We make one comparison using the Interagency Language Roundtable (ILR) scale to equilibrate the hand-held device to a typical investment by a human learner (~600 hours) to reach a similar level of L2 function. We selected materials consistent with ILR level 2 (≃ CEFR level B2).

Language Learning Cost Hours of instruction typically needed for a gov’t learner to reach various functional language levels (ILR 0+ to 3+). Parameter is L1-L2 difficulty class.

Component Accuracies Assume100% Assume100% nn% nn% ASR Built-in Translator Speech Synthesizer We assumed that all experimental utterances are intelligible and that the Spanish and English synthetic speech is also intelligible. No tests involve repair, accommodation, interviews, long turns, role plays, or ratings of text in context.

Participants & Procedure English-to-Spanish: Six educated adult speakers, 4 male & 2 female, quiet environments, after <10 minutes practice, recite 150 practical sentences selected at random from a travelers’ phrase book. Approx. 900 words/speaker. Retries discouraged. Spanish-to-English: Five adult speakers, 1 male & 4 female, quiet environments, after <10 minutes practice, recite 150 practical sentences selected at random from a travelers’ phrase book. Approx. 900 words/speaker. Retries were allowed.

English Speech Recognition Word Error Rate (WER) Examples: # utterances per speaker % Utterances @ Zero Error % Utterances < 5% WER Global WER 150 69% 82% 0.03 Intended Utterance Recognized String What's my room number? Cuál es el número de mi habitación? I would like to rent a small car. Me gustaría alquilar un coche pequeño. I want to leave as soon as possible. Quiero salir tan pronto como sea posible. I would like a bottle of red wine. Me gustaría una botella de vino tinto. I would like some fish with boiled potatoes. Me gustaría algo de pescado con patatas hervidas. It can you really have napkin, please. Puede usted realmente tiene servilleta, por favor. Add to your room is sometimes a bit exclusive. Añadir a su habitación es a veces un poco exclusiva. Please bring me another bottle of white wine. Por favor tráigame otra botella de vino blanco. Can you show me a double room, please? Puede usted mostrarme una habitación doble, por favor?

English⇒Spanish Translation Word Error Rate (WER) Examples: # utterances per speaker % Utterances @ Zero Error % Utterances < 5% WER Global WER 150 80% 87% .03 Source String (ASR output) Translation That's my suitcase. Ese es mi maleta. At what time do we arrive? A qué hora llegamos? We serve poultry and came. Se sirven aves y vino. How far is it to filing C? A qué distancia está empaste c? I'll have one of those, please. Tomaré uno de esos, por favor. Do you have anything to declare? Tiene algo que declarar? What time does the plane take off? A qué hora sale el avión? Do you have desserts for diabetics? Tienes postres para diabéticos? Did you want a table for five people? Quiere una mesa para 5 personas? Can we have breakfast in our room? Podemos desayunar en nuestra habitación?

Spanish Speech Recognition Word Error Rate (WER) Examples: (EPR) # utterances per speaker % Utterances @ Zero Error % Utterances < 5% WER Range of Speaker WER 150 NN 0.05 Intended Utterance Recognized String ¿Está incluido el kilometraje? Esta incluido el kilometraje? Tráigame la cuenta, p f. Tráigame la cuenta por favor. ¿Dónde está la parada de autobuses? Dónde está la parada de autobuses? ¿Qué clases de queso tiene? Qué clases de queso tiene? ¿Me podrá llamar usted a las siete? Me podrás tomar usted a las siete. ¿Qué tipo de mariscos tiene usted? Qué tipo de mariscos tiene usted? ¿Dónde queda la agencia de alquiler de coches? Dónde queda la agencia de alquiler de coches? Quisiera un billete de ida sola para Londres. Quisiera un billete de ida sola para londres. ¿Me puede dar un vaso de agua, p f? Me puede dar un vaso de agua por favor? ¿Puede usted decirme qué hora es, p f? Puede usted decirme qué hora es por favor?

Spanish⇒English Translation Word Error Rate (WER) Examples # utterances per speaker % Utterances @ Zero Error % Utterances < 5% WER Global WER 150 55% 61% 0.08 Source String Translation Dónde están las duchas? Where are the showers? Por favor pare en la próxima parada. Please stop at the next stop. Quiero que planchen esta ropa por favor. I want this ironed clothes. Pueda ser conexión con un vuelo de allí cante. Can be connection with a flight from there lady sings. Por favor lleve mi equipaje a la parada de autobús. Please take my baggage to the bus stop. Hay un departamento libre en el coche cama? Is there a free department in the sleeping car? Prepáreme un bocadillo de jamón queso y tomate por favor. Make me a ham sandwich tomato and cheese, please. Quiero marcharme cuanto antes. I want to leave as soon as possible. Creo que ese es mi asiento. I think that is my seat. Qué autobús debo tomar para la universidad? Which bus should I take to the University?

English⇒Spanish ASR & Translation Word Error Rate (WER) Examples: (RAD) # utterances per speaker % Utterances @ Zero Error % Utterances < 5% WER 150 63% 77% Intended Utterance Recognized String Translation Where are my black shoes? Dónde están los zapatos negros? How. long will you be staying? How long will you be staying? Cuánto tiempo se va a quedar? Do you have room for a caravan tent? Tienen habitación para una carpa caravana? The bathroom faucet is dripping. El baño está goteando. Where are the showers and toilets? Dónde están las duchas y servicios? Please let me off at the next stop. Por favor déjeme en la próxima parada. I want these clothes ironed, please. Quiero que se planchen estas ropas, por favor. Can I make a connection to Alicante? Can I make a connection to Madrid? Puedo hacer una conexión a madrid? Please take my luggage to the bus stop. Por favor lleve mi equipaje a la parada de autobús. Are there any free compartments in the sleeping-car? Are there any free compartments in the sleeping car? Hay algún compartimento libre en el coche cama?

Spanish⇒English ASR & Translation Examples: (HEF) # utterances per speaker % Utterances @ Zero Error % Utterances WER < 5% 150 40% 45% Intended Utterance Recognized String Machine Translation ¿Cuál es el número de mi habitación? Cuál es el número de mi habitación? What is the number of my room? Quisiera alquilar un coche pequeño. I would like to rent a small car. Quiero marcharme cuanto antes. I want to leave as soon as possible. Me gustaría una botella de vino tinto. I would like a bottle of red wine. Quisiera pescado con papas hervidas. I would like fish with boiled potatoes. ¿Me puede alcanzar una servilleta, p f? Me puede alcanzar una servilleta por favor? Can you get me a napkin please? Un salón de té es a veces, un poco exclusivo. Un salón de té sabes es un poco exclusivo. A tearoom know it's a little sole. Tráigame otra botella de vino blanco, p f. Traigame otra botella de vino blanco por favor. Please bring me another bottle of white wine, please. ¿Puede usted enseñarme una habitación matrimonial, p f? Puede usted enseñarme una habitación matrimonial p.f.? Can you show me one twin room please?

Results 97% Words Correct 97% Words Correct SPOKEN ENGLISH ASR Built-in Translator Speech Synthesizer SPOKEN SPANISH 95% Words Correct 92% Words Correct SPOKEN SPANISH ASR Built-in Translator Speech Synthesizer SPOKEN ENGLISH Differences in ASR accuracy may be partially due to procedural differences. Some errors do not impede communication in context.

Next Steps Next Steps: Estimate the functional accuracy of a representative sample of the imperfect spoken translations. Example: Traveler at train station: Where is the office luggage? presumed intention:_________________________________. 2. Compare component performance to other systems (e.g. Google Translate) 3. Measure progress over time. 4. Consistent procedures for English and Spanish. 5. Experiment with unscripted interactions in situ.

Summary The observed accuracy (in WER) of speech recognition and of text translation for one hand-held system, Jibbigo, running on the iPhone locally, suggests that for most speakers, reasonably complex requests and queries can usually be communicated into another language successfully in quiet environments. 63% of English-Spanish and 40% of Spanish-English utterances are recognized and translated perfectly.

Conclusions To understand the true usefulness of the current generation of hand-held interpreters, more analysis is needed of this data and more cumbersome experimental data is needed. For example, trying to use Jibbigo in a real-life English-Chinese-English exchange in China. In any event, the authors believe that the observed performance is adequate for many communicative exchanges that would otherwise be impossible, and that within several years such devices will be starting to undermine some practical motives for learning second languages.