1. The NESPOLE Interchange Format (IF) Lori Levin, Emanuele Pianta, Donna Gates, Kay Peterson, Dorcas Wallace, Herve Blanchon, Roldano Cattoni, Jean-Philippe Gibaud, Chad Langley, Alon Lavie, Nadia Mana, Fabio Pianesi

2. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Coverage: how do we measure coverage of the domain –Reliability: so that an analyzer written by one person in Italy can work with a generator written by someone he has never met in Korea. –Scalability: move to broader semantic domains without a constant increase in the amount of work.

3. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Coverage –Reliability –Scalability

4. What is an interlingua? Representation of meaning or speaker intention. Sentences that are equivalent for the translation task have the same interlingua representation. The room costs 100 Euros per night. The room is 100 Euros per night. The price of the room is 100 Euros per night.

5. Mi chiamo Alex WaibelMy name is Alex Waibel. Give-information+personal-data (name=alex_waibel) [ s [ vp accusative_pronoun “chiamare” proper_name]] [ s [ np [possessive_pronoun “name”] ] [ vp “be” proper_name]] Direct Transfer Interlingua Vaquois MT Triangle

6. Other Approaches to Machine Translation Direct: –Very little analysis of the source language. Transfer: –Analysis of the source language. –The structure of the source language input may not be the same as the structure of the target language sentence. –Transfer rules relate source language structures to target language structures.

7. Note Some transfer systems may produce a more detailed meaning representation than some interlingua systems. The difference is whether translation equivalents in the source and target languages are related by a single canonical representation.

8. Multilingual Translation with an Interlingua Japanese Arabic Chinese (input sentence) San1 tian1 qian2, wo3 kai1 shi3 jue2 de2 tong4 English French German Italian Korean Arabic Chinese (paraphrase) wo3 yi3 jin1 tong4 le4 san1 tian1 English (output sentence) The pain started three days ago. French German Italian Japanese Korean Analyzers Generators Spanish Catalan Interlingua give-information+onset+body-state (body-state-spec=pain, time=(interval=3d, relative=before))

9. Multilingual translation with transfer Transfer-rules-1: Arabic-Catalan Transfer-rules-2: Catalan-Arabic Transfer-rules-3: Arabic-Chinese Transfer-rules-4: Chinese-Arabic Transfer-rules-5: Arabic-English Transfer-rules-6: English-Arabic Etc.

10. Advantages of Interlingua Add a new language easily –get all-ways translation to all previous languages by adding one grammar for analysis and one grammar for generation Mono-lingual development teams. Paraphrase –Generate a new source language sentence from the interlingua so that the user can confirm the meaning

11. Disadvantages of Interlingua “Meaning” is arbitrarily deep. –What level of detail do you stop at? If it is too simple, meaning will be lost in translation. If it is too complex, analysis and generation will be too difficult. Should be applicable to all languages. Human development time.

12. Interlingual MT Systems University of Maryland – Lexical Conceptual Structure (Dorr) Carnegie Mellon –Kantoo (Mitamura and Nyberg) –Nespole/C-STAR (Waibel, Levin, Lavie) UNL (Universal Networking Language) Microcosmos (Nirenburg) Verbmobil – Domain actions (Block)

13. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage

14. A Travel Dialogue Translated from Italian A: Albergo Gabbia D’Oro. Good evening. B: My name is Anna Maria DeGasperi. I’m calling from Rome. I wish to book two single rooms. A: Yes. B: From Monday to Friday the 18th, I’m sorry, to Monday the 21st. A: Friday the 18th of June. B: The 18th of July. I’m sorry. A: Friday the 18th of July to, you were saying, Sunday. B: No. Through Monday the 21st.

15. A Travel Dialogue (Continued) B: So with departure on Tuesday the 22nd. A: Then leaving on the 22nd. Yes. We have two singles certainly. B: Yes. A: Would you like breakfast? B: Is it possible to have all meals? A: No. We serve meals only in the evening. B: Ok. If you can do breakfast and dinner. A: Ok. B: Do you need a deposit?

16. A Travel Dialogue (Continued) A: You can give me your credit card number. B: Ok. Just a moment. Ok. My name is Anna Maria DeGaperi. The card is 005792005792. A: Good. B: Expiration 2002. A: 2002. Good. Thank you. We need a confirmation on the 18th of July before 6pm. B: Goodbye. A: Thanks. Goodbye. B: Thanks. Goodbye.

17. A Non-Task-Oriented Dialogue (We can’t translate this.) A: Are you cooking? B: My father is cooking. I’m cleaning. I just finished cleaning the bathroom. A: Look. What do you know about Monica? B: I don’t know anything. Look. I don’t know anything. A: You don’t know anything? I wrote her three weeks ago, but if she hasn’t received the letter, they would have returned it. I hope she received it. B: Because Celia told me that the address that Monica had given us was wrong. She said that if I was going to write to her, well, …. From the Spanish CallHome corpus: unlimited conversation between family members.

18. The Ideal MT System… Fully automatic High quality Domain independent (any topic) …. isn’t within the current state-of-the-art.

19. Design Principles of the Interchange Format Suitable for task oriented dialogue Based on speaker’s intent, not literal meaning –Can you pass the salt is represented only as a request for the hearer to perform an action, not as a question about the hearer’s ability. Abstract away from the peculiarities of any particular language –resolve translation mismatches. Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

20. Translation Mismatches Sentences that are translation-equivalents in two languages do not have the same syntactic structure or predicate-argument structure. (Unitrans; Eurotra) –I like to swim. –I swam across the river. –Sue met with Sam/Sue met Sam.

21. Design Principles (continued) Domain independent framework with domain-specific parts Simple and reliable enough to use: –at multiple research sites with high intercoder agreement. –with widely varying type of parsers and generators. Allow robust language engines –Underspeicification must be possible. –Fragments must be represented.

22. Speech Acts: Speaker intention vs literal meaning Can you pass the salt? Literal meaning: The speaker asks for information about the hearer’s ability. Speaker intention: The speaker requests the hearer to perform an action.

23. Remember this term: Domain Action

24. Domain Actions: Extended, Domain-Specific Speech Acts give-information+existence+body-state It hurts. give-information+onset+body-object The rash started three days ago. request-information+availability+room Are there any rooms available? request-information+personal-data What is your name?

25. Domain Actions: Extended, Domain-Specific Speech Acts In domain. –I sprained my ankle yesterday. –When did the headache start? Out of domain –Yesterday I slipped in the driveway on my way to the garage. –The headache started after my boss noticed that I deleted the file.

26. Formulaic Utterances Good night. tisbaH cala xEr waking up on good Romanization of Arabic from CallHome Egypt

27. Same intention, different syntax rigly bitiwgacny my leg hurts candy wagac fE rigly I have pain in my leg rigly bitiClimny my leg hurts fE wagac fE rigly there is pain in my leg rigly bitinqaH calya my leg bothers on me Romanization of Arabic from CallHome Egypt.

28. Language Neutrality Comes from representing speaker intention rather than literal meaning for formulaic and task-oriented sentences. How about … suggestion Why don’t you… suggestion Could you tell me… request info. I was wondering… request info.

29. Domain Action Interlingua and Lexical Semantic Interlingua  and how will you be paying for this  Domain Action representation:  a:request-information+payment (method=question)  Lexical Semantic representation: predicate: pay time: future agent: hearer product: distance: proximate, type: demonstrative manner: question

30. Complementary Approaches Domain actions – limited to task oriented sentences Lexical Semantics– less appropriate for formulaic speech acts that should not be translated literally

31. Components of the Interchange Format a: speaker a: (agent) give-information speech act give-information +availability+room concept* +availability+room (room-type=(single & double), argument* (room-type=(single & double), time=md12) Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

32. Components of IF as of February 2002 61 speech acts give-information –domain independent, –20 are dialog managing 108 concepts availability, accommodation –mostly domain dependent 304 arguments room-type, time –domain dependent and independent 7,652 values single, double, 12th

33. Examples  no that’s not necessary  c:negate  yes I am  c:affirm  my name is alex waibel  c:give-information+personal-data (person-name=(given- name=alex, family-name=waibel))  and how will you be paying for this  a:request-information+payment (method=question)  I have a mastercard  c:give-information+payment (method=mastercard) Instructions: Delete sample document icon and replace with working document icons as follows: Create document in Word. Return to PowerPoint. From Insert Menu, select Object… Click “Create from File” Locate File name in “File” box Make sure “Display as Icon” is checked. Click OK Select icon From Slide Show Menu, Select Action Settings. Click “Object Action” and select “Edit” Click OK

34. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage –Scalability

35. Conventional Speec Acts thank you. c:thank can I help you ? a:offer+help (who=i, to-whom=you) my name is Chad c:give-information+personal-data (person-name=(given-name=chad))

36. Fragments: ellipsis and in a restaurant. a:give-information+concept (conjunction=discourse, location=(restaurant, identifiability=no)) which town? c:request-information+concept (concept-spec=(town, identifiability=question))

37. Fragments: abandoned You should a: suggest+concept (who=you) What should I c: request-suggestion+concept (who=I)

38. Coordination of Sentences I want to go to France and I would prefer to leave today. c:give-information+disposition+trip (destination=(object-name=france), disposition=(who=i, desire)) c:give-information+disposition+departure (conjunction=discourse, time=(relative- time=today), disposition=(who=i, preference))

39. Coordination of sentences, reduced I want to leave Pittsburgh at 2 and return from Rome at 5. c:give-information+disposition+departure (conjunction=discourse, origin=(object- name=pittsburgh), disposition=(who=i, desire), time=(clock=(hours=2))) c:give-information+trip (conjunction=discourse, factuality=unspecified, trip- spec=return, origin=(object-name=rome), time=(clock=(hours=5)))

40. Conjunctive Set I like festivals and plays. c:give-information+disposition+event (... event-spec=(operator=conjunct, [(festival, quantity=plural), (play, quantity=plural)]))

41. Conjunction of modifiers I prefer red and blue cars. c:give- information+disposition+vehicle (... vehicle-spec=(car, quantity=plural, color=(operator=conjunct, [red, blue])))

42. Disjunctive Sets I prefer hotels or cabins. c:give- information+disposition+accommodation (... accommodation- spec=(operator=disjunct, [(hotel, quantity=plural), (cabin, quantity=plural)]))

43. Contrastive Set I like hotels but not cabins. c:give- information+disposition+accommodat ion (... accommodation- spec=(operator=contrast, [(hotel, quantity=plural), (polarity=negative, cabin, quantity=plural)]))

44. Attitudes: often a source of mismatches Disposition Eventuality Evidentiality Feasibility Knowledge Obligation Main verbs in English that occur in other languages as affixes, adverbs, or other construtions that are not clearly bi-clausal.

45. Disposition and I would like to arrive around September ninth. c:give-information+disposition+arrival (disposition=(who=i, desire), /* attitude */ conjunction=discourse, /* rhetorical information */ time=(exactness=approximate, month=9, md=9)) /* time */

46. Disposition I would like to stay in a hotel. –Disposition=desire I hate mushroom picking. –Disposition=dislike I am waiting to see the circle. –Disposition=expectation But wouldn’t matter. –Disposition=indifferent When do you plan on arriving in Pittsburgh? –Disposition=intention

47. Eventuality It is possible I may be arriving earlier. give-information+eventuality+arrival (eventuality=possible) I’m sure that they will arrive tomorrow. Maybe there is something beautiful to see. It is not impossible.

48. Evidentiality: Source of information Apparently there are many castles. Give-information+evidentiality+attraction I heard there are many castles. I noticed there is a winter package available. I’ve been told I must leave before ten.

49. Feasibility You can rent skis at the resort. Give-information+feasibility+rent+equipment (feasibility=feasible….)

50. Knowledge I didn’t know that Trento has lakes. Give-information+negation+knowledge+contain+attraction (knowledge=(who=I, polarity=negative), contain=(lake, quantity=plural), attraction-spec=name-trento) I know the location of the hotel.

51. Obligation You must make a reservation. Give-information+obligation+reservation (obligation=required….) You may cancel at any time. We require that you cancel 24 hours in advance.

52. Negation: with limited facilities for representing scope Of conventional speech act: I didn’t hear. Negate-dialogue-hear Of main predication: I did not make a reservation. Give-information+negation+reserve+accommodation (polarity=negative…) Of attitude: I don’t know if it’s all right. Give-information+negation+knowledge+feature+object (knowledge=(who=I, polarity=negative), object-spec=pronoun, feature=(modifier=acceptable))

53. Negation I didn’t promise I would not come. –Negate-promise+negation+action Of a concept: There is no downhill skiiing? Request-information+existence+activity (activity-spec=(polarity=negative, downhill_skiing)

54. Relative Clauses Broken into two IF’s. I want the hotel that you suggested. Give-information+disposition+accommodation (disposition=(desire, who=I), accommodation-spec=(hotel, identifiability=yes)) Give-information+recommendation+object (object-spec=relative, who=you, e-time=previous) Sentence internal relative clauses (e.g., modifying the subject) are not handled very well. –The only hotel that I can show you is a four star hotel. No long-distance gaps. –They are rare anyway.

55. Some simple relative clauses aren’t broken The hotel that is in Cavalese give-information+concept (accommodation-spec=(hotel, identifiability=yes, location=name- cavalese))

56. Yes-No Questions Conventional speech act: Do you hear me? Dialog-request-hear Does the flight leave at 2:00? Tell me if the flight leaves at 2:00. request-information+departure (transportation-spec=(flight, identifiability=yes), time=(clock=(hours=2)))

57. Wh-questions Who is traveling? request-information+trip (who=question) When are you traveling? What date are you traveling? How quiet is the hotel? Where are you traveling to? How are you traveling? What are you doing? No long distance gaps.

58. Rhetorical Relations Therefore I arrived late. give-information+arrival (cause=discourse, who=I…) I arrived late because of the snow give-information+arrival (who=I, cause=snow, e-time=previous, time=late)

59. Rhetorical Relations because I was tired. give-information+feature+person (rhetorical=cause, …) Other relations: after, before, besides, co- occurrence, concessive, condition, contrastive, dependency, purpose, related- to, restrictive-result, result, while

60. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage –Scalability

61. The Interchange Format Database 61.2.3 olang I lang I Prv IRST “telefono per prenotare delle stanze per quattro colleghi” 61.2.3 olang I lang E Prv IRST “I’m calling to book some rooms for four colleagues” 61.2.3 IF Prv IRST c:request-action+reservation +features+room (for-whom=(associate, quantity=4)) 61.2.3 comments: dial-oo5-spkB-roca0-02-3 d.u.sdu olang X lang Y Prv Z sdu in language Y on one line d.u.sdu olang X lang Z Prv Z sdu in language Z on one line d.u.sdu IF Prv Z IF on-one-line d.u. sdu comments: your comments d.u. sdu comments: go here

62. NESPOLE! Database Annotated turns (end 2001): –English: 815 (235 distinct DAs) –German: 2,873 (367) –Italian: 1,286 (233) –French: 234 (94) Total distinct DAs: 610 Annotated turns (end 2002): 30/40 % more

63. Tools and Resources IF specifications (available on the web) http://www.is.cs.cmu.edu/nespole/db/index.html IF discussion board http://peace.is.cs.cmu.edu/ISL/get/if.html C-STAR and NESPOLE! Data Bases http://www.is.cs.cmu.edu/nespole/db/index.html IF Checker (web interface) http://tcc.itc.it/projects/xig/xig-on-line.html IF test suite http://tcc.itc.it/projects/xig/xig-ts.html IF emacs mode

64. The C-STAR Interchange Format Database English Dialogues English Sentences Korean Dialogues Korean Sentences Italian Dialogues Italian Sentences Japanese Dialogues Japanese Utterances Distinct Dialogue Acts 36 2466 70 1142 5 233 124 5887 554 (310 agent, 244 client)

65. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage –Scalability

66. Comparison of two interlinguas I would like to make a reservation for the fourth through the seventh of July. IF-1 (C-STAR II, 1997-1999) c:request-action+reservation+temporal+hotel (time=(start-time=md4, end-time=(md7,july))) IF-2 (NESPOLE, 2000-2002) c:give-information+disposition+reservation +accommodation (disposition=(who=I, desire), reservation-spec=(reservation, identifiability=no), accommodation-spec=hotel, object-time=(start-time=(md=4), end-time=(md=7, month=7, incl-excl=inclusive)))

67. Comparison of four databases (travel domain, role playing, spontaneous speech) DB-1: C-STAR II English database tagged with IF-1 –2278 sentences DB-2: C-STAR II English database tagged with IF-2 – 2564 sentences DB-3: NESPOLE English database tagged with IF-2 – 1446 sentences –Only about 50% of the vocabulary overlaps with the C-STAR database. DB-4: Combined database tagged with IF-2 –4010 sentences Same data, different interlingua Significantly larger domain

68. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage –Scalability

69. Measuring Coverage No-tag rate: –Can a human expert assign an interlingua representation to each sentence? –C-STAR II no-tag rate: 7.3% –NESPOLE no-tag rate: 2.4% 300 more sentences were covered in the C-STAR English database End-to-end translation performance: Measures recognizer, analyzer, and generator performance in combination with interlingua coverage.

70. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage –Scalability

71. Example of failure of reliability Input: 3:00, right? Interlingua: verify (time=3:00) Poor choice of speech act name: does it mean that the speaker is confirming the time or requesting verification from the user? Output: 3:00 is right.

72. Measuring Reliability: Cross-site evaluations Compare performance of: –Analyzer  interlingua  generator –Where the analyzer and generator are built at the same site (or by the same person) –Where the analyzer and generator are built at different sites (or by different people who may not know each other) C-STAR II interlingua: comparable end-to-end performance within sites and across sites. –around 60% acceptable translations from speech recognizer output. NESPOLE interlingua: cross-site end-to-end performance is lower (but not clearly because of the IF).

73. Intercoder agreement: average of percent agreeent pairwise Speech actDomain ActionArguments IF-1: Site 1 and Site2 (exp.) 82%66%86% IF-2: Site 1 and Site 2 (4 experts) 92%75%87 % IF-2: Within Site 1 (3 experts) 94%88%90% IF-2: Site 1 vs Site 2 (3 experts and 1 experts) 89%62%83% IF-2: Site 1 and Site 2 (experts and novices) 88%63%86% IF-2: Within Site 2 (expert and novices) 89 %64 %87% IF-2: Within Site 2 (novices) 91%61%83%

74. Workshop on InterlinguaReliability SIG-IL Association for Machine Translation in the Americas October 8, 2002 Tiburon, California Intent to participate in coding experiment (dependency representation) (email to lsl@cs.cmu.edu)

75. Outline Approaches to MT: Interlingua, Transfer, Direct. The NESPOLE! Interlingua. –Overview and motivation –Linguistic coverage Tools and resources. Evaluating an interlingua –Reliability –Coverage –Scalability

76. Comparison of four databases (travel domain, role playing, spontaneous speech) DB-1: C-STAR II English database tagged with IF-1 –2278 sentences DB-2: C-STAR II English database tagged with IF-2 – 2564 sentences DB-3: NESPOLE English database tagged with IF-2 – 1446 sentences –Only about 50% of the vocabulary overlaps with the C-STAR database. DB-4: Combined database tagged with IF-2 –4010 sentences Same data, different interlingua Significantly larger domain

77. Measuring Scalability: Coverage Rate What percent of the database is covered by the top n most frequent domain actions? Coverage of 50 most frequent domain actions C-STAR client66.7% NESPOLE client66.5% Combined client62.9% C-STAR agent67.3% NESPOLE agent71.4% Combined agent64.0%

78. Measuring Scalability: Number of domain actions as a function of database size Sample size from 100 to 3000 sentences in increments of 25. Average number of unique domain actions over ten random samples for each sample size. Each sample includes a random selection of frequent and infrequent domain actions.

80. Comparison of four databases (travel domain, role playing, spontaneous speech) English database 1 tagged with interlingua 1: 2278 sentences English database 1 tagged with interlingua 2: 2564 sentences English database 2 tagged with interlingua 2: 1446 sentences –Only about 50% of the vocabulary overlaps with the English database 1. Combined databases tagged with interlingua 2: 4010 sentences Same data, different interlingua Significantly larger domain

81. Conclusions An interlingua based on domain actions is suitable for task-oriented dialogue: –Reliable –Good coverage –Scalable without explosion of domain actions It is possible to evaluate an interlingua for –Realiability –Expressivity –Scalability

82. How to have success with an interlingua in a multi-site project Keep it simple. Periodically check for intercoder agreement. Good documentation Discussion board for developers Know your language typology.