1. SAI User-System Interaction U1, Speech in the interface: 6. Human communication1 Module u1: Speech in the Interface 6: Human Communication Jacques Terken

2. SAI User-System Interaction U1, Speech in the interface: 6. Human communication2 Motivation n provide knowledge about basic concepts of human communication relevant to the design of speech interfaces n explore use of knowledge about human communication as model for man-machine interaction

3. SAI User-System Interaction U1, Speech in the interface: 6. Human communication3 Contents n basic scheme for communication n the structure of messages n the structure of communication n “how to do things with words ”1: communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction n 1 J.L. Austin: “How to do things with words”, 1962

4. SAI User-System Interaction U1, Speech in the interface: 6. Human communication4 basic scheme n Context –Situational context  situated communication: time, place, objects –Social context –Cultural context –Dialogue context context sender channel receiver feedback context

5. SAI User-System Interaction U1, Speech in the interface: 6. Human communication5 intentions and effects n Sender: intention to convey a message  message needs to be encoded n Receiver: decoding  effect Ideal communication: intention = effect Distortions underway: intention  effect’ Furthermore: effect is more than intention (sender also communicates unintentional information) n complex messages require sequence of steps  communication as structured action / collaboration

6. SAI User-System Interaction U1, Speech in the interface: 6. Human communication6 Contents n basic scheme for communication n the structure of messages n the structure of communication n communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction

7. SAI User-System Interaction U1, Speech in the interface: 6. Human communication7 formal aspects (1) n Simple case: messages containing a single dialogue act (inform, request, confirmation etc) n Sentence/utterance –put the red block before the yellow block next to the ball –I herewith pronounce you man and wife –What time does the next flight to Boston leave? n Sentences made up of word groups n Word groups made up of words n Words made up of syllables and phonemes (~40) or graphemes (~25) / @, e, a, r, … / ; { e, e, a, r, … }

8. SAI User-System Interaction U1, Speech in the interface: 6. Human communication8 formal aspects (2) n rules determine well-formedness of representations at different levels: –pragmatics: sentence meaning in context (speech act, dialogue act)  contextual meanings *the king of France is bald (but may be okay in a story) –semantics: sentence meanings  possible meanings *the 50 year old bachelor surprised his wife –syntax: sentence structure  possible sentences –morphology: word structure  possible words –Dutch: loopje; *loopetje; kommetje; *kompje –Phonology: sound structure  possible sound sequences Dutch: herfst; Polish: przewalski

9. SAI User-System Interaction U1, Speech in the interface: 6. Human communication9 formal aspects (3) n mapping between levels not fully specified (not one - to - one):  ambiguity n Mapping sounds onto words: /Aiscri:m/  {I scream} or {Ice cream} n Mapping words onto sentence structure: Syntactic ambiguities i saw the girl with the binoculars

10. SAI User-System Interaction U1, Speech in the interface: 6. Human communication10 put the red block before the yellow block next to the ball (1) put the red block before the yellow block [ that is ] next to the ball (2) put the red block before the yellow block, next to the ball

11. SAI User-System Interaction U1, Speech in the interface: 6. Human communication11 Contents n basic scheme for communication n the structure of messages n the structure of communication n communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction

12. SAI User-System Interaction U1, Speech in the interface: 6. Human communication12 levels of analysis topic …topic …topic transaction exchange exchange exchange turn turn turn move move move

13. SAI User-System Interaction U1, Speech in the interface: 6. Human communication13 transaction n (part of) conversation dealing with a single topic n consists of one or more exchanges

14. SAI User-System Interaction U1, Speech in the interface: 6. Human communication14 exchange n smallest interactive unit n consists minimally of two turns produced by two different speakers n involves the negotiation of a single piece of information n examples: request - inform - confirm presentation - acceptance

15. SAI User-System Interaction U1, Speech in the interface: 6. Human communication15 n adjacency pairs: questionanswer requestaccept / turn down invitationaccept / declinegreeting apologysmoother n Interactive pressure: opening move induces obligation to answer with closing move (conversational obligation)

16. SAI User-System Interaction U1, Speech in the interface: 6. Human communication16 n nesting of adjacency pairs: s: question X what time is the next train to amsterdam a: question Y amsterdam central? s: answer Y yes a: answer X in seven minutes at platform 5 mixed initiative vs. system initiative

17. SAI User-System Interaction U1, Speech in the interface: 6. Human communication17 turn n Turn: everything the speaker says before the next speaker takes over n Rules for turn-taking behaviour 1. Speaker selects next speaker 2. Next speaker selects self Window of opportunity for turn switching (minimizes overlap and interruption; interruptions only permissible in particular circumstances n Turn consists of one or more moves

18. SAI User-System Interaction U1, Speech in the interface: 6. Human communication18 temporal aspects of turn-taking n dialog not as a sequence of well-formed non-overlapping turns: –time window for turn taking –overlap, interruptions

19. SAI User-System Interaction U1, Speech in the interface: 6. Human communication19 move/dialog act n no fixed inventory, but dependent on application n Common types: n Primary dialogue acts –request for action –inform –question n Game theory conceives of communication as a sequence of moves Expresses the assumption that communication is governed by a set of rules n Dialogue control acts – acknowledge – check – confirm – greeting – thanks

20. SAI User-System Interaction U1, Speech in the interface: 6. Human communication20 Indirect communication n No one-to-one mapping between form and function: indirect communication (politeness forms) n Questions: –Can you tell me what time it is (yes/no question) –I would like to know what time it is (statement) –Do you have a watch (yes/no question with indirect communication ) Direct communication: –What time is it (wh-question) –Please tell me the time (request)

21. SAI User-System Interaction U1, Speech in the interface: 6. Human communication21 spoken dialog systems n Identify dialog act performed by user in order to decide what dialog act to produce as an an answer n Machine learning techniques in order to learn best continuation in context x (and to learn best path)

22. SAI User-System Interaction U1, Speech in the interface: 6. Human communication22 Contents n basic scheme for communication n the structure of messages n the structure of communication n communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction

23. SAI User-System Interaction U1, Speech in the interface: 6. Human communication23 spoken communication as joint action/cooperation n Speech is volatile n Mechanisms to cope with volatile nature of speech n co-operation receiver must be in receiving mode, i.e., be open to message sender must adjust to channel and receiver characteristics

24. SAI User-System Interaction U1, Speech in the interface: 6. Human communication24 n Goal achievement is usually accomplished through a number of steps involving both participants: dialogue n Monitoring of transmission success by sender on the basis of conscious and unconscious feedback from receiver n Immediate action in case of problems either by user or sender –Role of eye gaze in staying in touch, monitoring and turn taking spoken communication as joint action/cooperation (2)

25. SAI User-System Interaction U1, Speech in the interface: 6. Human communication25 speech generation n two simultaneous processes: –conceptualisation (“what to say”: content) –formulation (“how to say it”: converting content into utterance) simultaneous, but formulation lagging behind (  spontaneous speech phenomena) –restarts, self-corrections, hesitations (silence, ehm), disfluencies –grammar of spoken language

26. SAI User-System Interaction U1, Speech in the interface: 6. Human communication26 message formulation: issues n continuity, coherence: what we say usually links to what was said before: context provides search space for interpretation and allows for efficient expression –pronouns, anaphora –prosody –bridging: establishing a relation between expressions in consecutive utterances can you take the picnic supplies out of the van? the beer is probably warm, so please put it in the fridge.

27. SAI User-System Interaction U1, Speech in the interface: 6. Human communication27 grice’s cooperativity principle human communication (supposed to be) governed by grice’s cooperativity principle, consisting of the following maxims (guidelines for communication): n quantity: make your contribution as informative as is required I want a flight; I want a flight from A to B because …. n quality: do not say what you believe is false or for which you lack adequate evidence n relation: be relevant n manner: be perspicuous, brief and orderly; avoid obscurity and ambiguity

28. SAI User-System Interaction U1, Speech in the interface: 6. Human communication28 speech comprehension n Stages in comprehension: –Perception: perceiving sounds and mapping sounds onto words –Interpretation –Evaluation –From a processing point of view perception and interpretation are closely interlinked keywords: on-line processing, hypothesis testing and integrating

29. SAI User-System Interaction U1, Speech in the interface: 6. Human communication29 backchannelling n “utterances” and other communicative signals guiding the progress of the communication, i.e., serving a control function –nodding –yes, hm-hm –“madrid?”

30. SAI User-System Interaction U1, Speech in the interface: 6. Human communication30 error recovery n on-line property allows fast and efficient recovery: –problems noticed immediately –cause of problem usually easily identified “what did you say?” “did you say …” “what do you mean”

31. SAI User-System Interaction U1, Speech in the interface: 6. Human communication31  Grounding agreeing about current state of the dialogue Important role of feedback  Shared knowledge + knowledge that speaker knows is available to hearer + world knowledge (“living memory”) + knowledge concerning context of communication and what has been established by previous part of discourse common ground

32. SAI User-System Interaction U1, Speech in the interface: 6. Human communication32 Contents n basic scheme for communication n the structure of messages n the structure of communication n communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction

33. SAI User-System Interaction U1, Speech in the interface: 6. Human communication33 n human communication no neat succession of turns: overlap in time: provision for speech recognition needed n Indirect communication: message function and form no 1:1 relation n interpretation dependent on domain knowledge: requires domain modelling: successful for restricted domains n interpretation dependent on context: requires discourse model

34. SAI User-System Interaction U1, Speech in the interface: 6. Human communication34 supporting the interaction n Prompting: system initiative n Zooming: increase amount of feedforward in system prompt –How may I help you –Which connection –Are you interested in connections or prices –Please respond with yes or no n Cooperative replies –User: “Do you have flights between 7:15 and 7:45?” –S: “No, but there is a flight at 7:55”

35. SAI User-System Interaction U1, Speech in the interface: 6. Human communication35 robustness issue n Lack of robustness creates need for verifcation n Explicit verification: –Did you say “amsterdam”? n Implicit verification –U: “I want to travel from.. to rotterdam” –S: “when do you want to travel to rotterdam?” –Cf: S: “when do you want to travel to amsterdam?”

36. SAI User-System Interaction U1, Speech in the interface: 6. Human communication36 n Verification is costly when you got it right n Feedback (positive feedback or implicit verification) is misleading when you good it wrong n Use verfication when uncertain n Use feedback when confident

37. SAI User-System Interaction U1, Speech in the interface: 6. Human communication37 confidence-based verification n High confidence: just accept or implicit verification n Medium confidence: explicit verification n Low confidence: explicit request for repetition n Precision/recall issue for correctness and confidence measure: –Confidence measure may not be right: either too high or too low

38. SAI User-System Interaction U1, Speech in the interface: 6. Human communication38 Contents n basic scheme for communication n the structure of messages n the structure of communication n communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction

39. SAI User-System Interaction U1, Speech in the interface: 6. Human communication39 n communication as joint action (cooperation) –taking into consideration listener needs –listening for intention n rule-governed behaviour at many different levels –structure of messages –structure of conversations (e.g. turn-taking behaviour) n grounding n feedback (back-channelling)

40. SAI User-System Interaction U1, Speech in the interface: 6. Human communication40 Contents n basic scheme for communication n the structure of messages n the structure of communication n communication as process n implications for man-machine interaction n key points n human-human communication as model for human-computer interaction

41. SAI User-System Interaction U1, Speech in the interface: 6. Human communication41 cooperative assistant n Characteristics of cooperative assistant –Interprets messages in terms of intentions not in terms of form (“c.a. is not a politician or lawyer”) –Cooperative replies (not blunt “no” but offers alternatives, e.g. through constraint relaxation) –Context-sensitive replies and help No global help manual –Proactive, triggered by implicit cues Does not await explicit request for help but anticipates

42. SAI User-System Interaction U1, Speech in the interface: 6. Human communication42 discussion n Is this a good model for human computer interaction? n Considerations: –Schneiderman c.s.: user should stay in control –Maes/Sidner: computers should be able to understand our intentions so that we don’t have to specify all the actions explicitly ourselves Computers should monitor our actions so that they can learn how to anticipate our needs and preferences