Temporal Relations in Visual Semantics of Verbs Minhua Eunice Ma and Paul Mc Kevitt School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee Derry/Londonderry, N. Ireland

AICS 2003, Dublin, Ireland story in natural language CONFUCIUS movie/drama script 3D animation non-speech audio tailored menu for script input speech (dialogue) Storywriter /playwright User /story listener  To interpret natural language stories and to extract conceptual semantics from natural language  To generate 3D animation and virtual worlds automatically from natural language  To integrate 3D animation with speech and non-speech audio for presenting multimodal stories Background: CONFUCIUS (intelligent storytelling system)

AICS 2003, Dublin, Ireland  Temporal relations Allen’s interval relations Application in story-based interactive systems Temporal relations in technical orders domain (Badler et al., 1997)  Related research in NLP Sentence level temporal analysis Lexical vs. post-lexical temporal relations Lexical vs. post-lexical Lexical semantics Previous research

AICS 2003, Dublin, Ireland Allen’s interval relations

AICS 2003, Dublin, Ireland NLP in CONFUCIUS Coreference resolution Part-of-speech tagger Syntactic parser Morphological parser Semantic inference Pre-processing Connexor FDG parser WordNet LCS database LEXICON & MORPHOLOGICAL RULES FEATURES Disambiguation Temporal reasoning Lexical temporal relations Post-lexical temporal relations

AICS 2003, Dublin, Ireland Verb entailments Verb entailment: fixed truth relation between verbs with entailment given by part of lexical meaning, i.e. one verb entails another The implication logic relationship: if p then q (p  q)

AICS 2003, Dublin, Ireland Troponym  Elaborates manner of base verb (Fellbaum, 1998)  Examples: “trot”-“walk” (fast), “gulp”-“eat” (quickly) EVENT go (move) run cause … other action predicates walkclimbjumpmanner-of-motion verbs … limpstrideswaggertrot …

AICS 2003, Dublin, Ireland Temporal relations in verb entailment {p,m,o,s,f -1,≡} may also represent temporal relation between pair of cognate verbs and state of corresponding adjectives e.g. shorten-short, beautify-beautiful, clarify-clear

AICS 2003, Dublin, Ireland Representing procedural events Arguments of EVENT [EVENT agent: theme: space/time: manner: instrument: precondition: subactivities: result:] Relationship between definiendum verb and its subactivities act():- subact1(), …… subacti(), ……. subacti R act, i  N, R  {d,s,f,  } eatOut():- bookASeat(), goToRestaurant(), orderDishes(), eat(), pay(), leave(). a. Original definition eatOut():- bookASeat() {p} goToRestaurant(){p,m} orderDishes() {p} eat() {p,m} pay() {p,m} leave(). b. “eatOut” in restaurant eatOut():- bookASeat() {p} goToRestaurant(){p,m} orderDishes() {p} eat() {p,p -1,m} pay() {p,m} leave(). c. “eatOut” in restaurant/ fast food shop eatOut():- [bookASeat() {p}] goToRestaurant(){p,m} orderDishes() {p} eat() {p,p-1,m} pay() {p,m} leave(). d. Optional subactivities

AICS 2003, Dublin, Ireland Comparison with Badler’s temporal constraints Badler’s temporal constraints (technical orders domain) Sequential Parallel Jointly parallel Independently parallel While parallel Interval relations {p,m} {s,s -1,  } (act1 {s,s -1,  } act2) {p,m} act3 {f,f -1,  } act_domt {s -1,f -1,  } act_indomt compositional (e.g. jointly parallel); all 5 constraints are disjunctions of several interval relations consider other factors such as dominancy of action (e.g. while parallel) domain-specific

AICS 2003, Dublin, Ireland Comparison with Badler’s temporal constraints Badler’s temporal constraints (technical orders domain) Sequential Parallel Jointly parallel Independently parallel While parallel Interval relations {p,m} {s,s -1,  } (act1 {s,s -1,  } act2) {p,m} act3 {f,f -1,  } act_domt {s -1,f -1,  } act_indomt compositional (e.g. jointly parallel); all 5 constraints are disjunctions of several interval relations consider other factors such as dominancy of action (e.g. while parallel) domain-specific

AICS 2003, Dublin, Ireland Comparison with Badler’s temporal constraints Badler’s temporal constraints (technical orders domain) Sequential Parallel Jointly parallel Independently parallel While parallel Interval relations {p,m} {s,s -1,  } (act1 {s,s -1,  } act2) {p,m} act3 {f,f -1,  } act_domt {s -1,f -1,  } act_indomt compositional (e.g. jointly parallel); all 5 constraints are disjunctions of several interval relations consider other factors such as dominancy of action (e.g. while parallel) domain-specific

AICS 2003, Dublin, Ireland Comparison with Badler’s temporal constraints Badler’s temporal constraints (technical orders domain) Sequential Parallel Jointly parallel Independently parallel While parallel Interval relations {p,m} {s,s -1,  } (act1 {s,s -1,  } act2) {p,m} act3 {f,f -1,  } act_domt {s -1,f -1,  } act_indomt compositional (e.g. jointly parallel); all 5 constraints are disjunctions of several interval relations consider other factors such as dominancy of action (e.g. while parallel) domain-specific

AICS 2003, Dublin, Ireland Comparison with Badler’s temporal constraints Badler’s temporal constraints (technical orders domain) Sequential Parallel Jointly parallel Independently parallel While parallel Interval relations {p,m} {s,s -1,  } (act1 {s,s -1,  } act2) {p,m} act3 {f,f -1,  } act_domt {s -1,f -1,  } act_indomt compositional (e.g. jointly parallel); all 5 constraints are disjunctions of several interval relations consider other factors such as dominancy of action (e.g. while parallel) domain-specific

AICS 2003, Dublin, Ireland Achievement vs. accomplishment events Achievement events (Vendler, 1967): e.g. “find”, “arrive”, “die” punctual events occuring at single moment definite time instants never hold over intervals Why use interval relations instead of point-based relations? Pragmatic reasons (Verkuyl, 1993) Ontological reasons (Pinon, 1997) Practical reason for language visualisation achievement events depend on existence of context context + visual definitions → intervals find():- search(), eyesFixedOn(). arrive():- go(), stopAtDestination().

AICS 2003, Dublin, Ireland  Visual definitions of causative verbs (e.g. “kill”) must subsume result states (stative verbs) (e.g. “die”)  Represent distinction between launching causatives: causation of inception of motion entraining causatives: continuous causation of motion Temporal relations of lexical causatives disjunction set of interval relations between cause and effect adequate to define difference: {s,p,m,o} (launching) {≡,f -1 } (entraining)

AICS 2003, Dublin, Ireland Lexical and post-lexical repetition  Post-lexical level repetition e.g. “Roses come into bloom once a year.” “I visit the school every day.” or marked by “again", "continues to", "a second time”  Lexical level repetition Represent periodical repetition of subactivities walk():- [step()] R. hammer():- [hit()] R. Morphological prefix "re-"

AICS 2003, Dublin, Ireland Categories of action verb Action verbs Movement or partial movement Biped kinematics, e.g. go, walk, jump, swim, climb Face expressions, e.g. laugh, angry Lip movement, e.g. speak, say, sing, tell Lexical causatives Concerning single object, e.g. push, kick, bring, open Concerning multiple objects Bitransitive verbs, e.g. give, sell, show Transitive verbs with object & implicit instrument/goal/theme, e.g. cut, write, butter, pocket Verbs without distinct visualization when out of context trying verbs: try, attempt, succeed, manage helping verbs: help, assist letting verbs: allow, let, permit create/destroy verbs: build, create, assemble, construct, break, destroy verbs whose visualization depends on their objects, e.g. play (harmonica/football), make (the bed/trouble/a phone call), fix (a drink/a lock) High level behaviours (routine events) e.g. interview, eat out (go to restaurant), call (make a telephone call), go shopping involve speech modality

AICS 2003, Dublin, Ireland  Lexical Visual Semantic Representation (LVSR): necessary semantic representation between 3D model and language syntax  LVSR based on Jackendoff’s LCS adapted to task of language visualization (enhancement with Schank’s scripts)  Interval relations represent temporal relationship between subactivities of complex actions in LVSR  e.g. “The waiter approached me: ‘Can I help you? Sir.’” 3D animation “John walked towards the house.” 3D animation “Nancy ran across the field.” 3D animation Lexical Visual Semantic Representation

AICS 2003, Dublin, Ireland ♦ Temporal relation is a crucial issue in modelling action verbs, their procedures, contexts, presupposed and result states ♦ Temporal relation within verb semantics (lexical level) ♦ Semantic representation of verbs with temporal information based on Allen’s interval logic Conclusion

AICS 2003, Dublin, Ireland Future work  Quantitative factor  Action composition for simultaneous activities  Verbs concerning multiple characters’ synchronization & coordination Character can start a task when another signals pre- conditions are ready Two or more characters cooperate in shared task