Alignment in human-human and human-computer interactions Holly Branigan Jamie Pearson University of Edinburgh.

Alignment in human-human and human-computer interactions Holly Branigan Jamie Pearson University of Edinburgh

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Collaborators University of Edinburgh Martin Pickering Stanford University Clifford Nass John Hu

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Outline Our approach Features of Human-Human interaction (HHI) –Audience design –Alignment Patterns of alignment Explanations for alignment Relevance to HCI Methodology for studying HCI Data from experiments investigating lexical/syntactic alignment in HCI

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Our approach Psycholinguistic accounts of HHI: –Characterising linguistic behaviour –Identifying underlying psychological mechanisms Focus on lexical and syntactic processing Applying to HCI: –Predicting human (user) behaviour –Simulating human behaviour –Modifying human behaviour

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Features of HHI Linguistic behaviour in HHI is highly flexible and adaptive: –An individual speaker’s behaviour is partially contextually-determined. –Linguistic choices (semantic, lexical, syntactic, phonological..) choices may vary according to context. –Relevance to HCI: Features of HCI communicative context may influence linguistic behaviour.

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Determinants of Behaviour Influence of addressee on speaker: Indirectly – speaker’s beliefs about addressee’s state of knowledge, interests etc –Audience Design Directly – via addressee’s own linguistic behaviour –Alignment Both influences may be active simultaneously

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Audience Design Tailoring utterance to fit addressee: –Their knowledge, interests, beliefs etc May be based on speaker’s a priori beliefs or assumptions about addressee Or may be based on direct evidence from addressee –E.g., addressee’s feedback

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Audience Design – lexical choice Fussell & Krauss (1992): –speakers adjust their choice of lexical terms to fit the assumed knowledge of their addressees: Proper names when addressee likely to know referent: Clint Eastwood More detailed descriptions when addressee less likely to know referent: Media mogul, was married to Jane Fonda, owns CNN, has a moustache and grey hair –Assessment of addressee knowledge is based on a priori judgements about social distribution of knowledge movie stars > industrialists

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Audience Design – lexical choice Isaacs & Clark (1987): –A priori assessment of addressee knowledge can be rapidly adjusted on basis of direct evidence from feedback Lexical choices then reflect this adjusted assessment –‘experts’ in a domain (New York landmarks) quickly adjust their choice of lexical terms to fit the apparent expertise of their addressees: –Proper names when addressee is apparently fellow expert Chrysler building –More detailed descriptions when addressee is apparently non- expert Building with a tall pointy roof and a spike on top

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Audience Design - syntax Audience design may influence syntactic choices: –Speakers may choose syntactic structures that are most easily understood by addressees. Ambiguity-avoidance: Put the penguin in the cup on the star vs Put the penguin that’s in the cup on the star When two penguins (and an empty cup), temporary ambiguity at penguin in the cup…

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Audience Design - syntax Audience design may influence syntactic choices: –Speakers may choose syntactic structures that are most easily understood by addressees. Ambiguity-avoidance: Put the penguin in the cup on the star vs Put the penguin that’s in the cup on the star When two penguins (and an empty cup), temporary ambiguity at penguin in the cup… Speakers include that’s more often (removing ambiguity) when there are two penguins than where is only one penguin. Haywood, Pickering & Branigan (2005)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Audience Design - summary Speakers in HHI may engage in strategic planning of aspects of utterances: –Calculations of addressee knowledge, beliefs etc –Based on a priori assumptions or direct evidence A priori assumptions are quickly updated in light of direct evidence; audience design reflects this –Strategic planning may affect lexical choice, syntax, semantic choices E.g., choice of reference frame (Schober, 1993)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment Speakers are affected directly by addressee’s linguistic behaviour: –Tendency for speakers in a dialogue to converge linguistic behaviour. –After hearing an interlocutor use particular linguistic behaviour, speakers tend to repeat that behaviour. –This tendency towards alignment is robust and highly pervasive. Rhetorical structure, semantic structure, syntactic structure, lexical choice, accent, speech rate, ….

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment Alignment can be implicit: –Almost always arises without explicit negotiation Speakers often align on different expression than negotiated expression –Speakers usually unaware of effects Can sometimes (but often cannot) report awareness of meaning-related alignment Usually unaware of alignment of form

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – semantic choices Alignment occurs for aspects of language associated with meaning: –e.g. reference frames (Watson, Pickering & Branigan, 2005) The dot is left of the camera vs The dot is below the camera

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – semantic choices Alignment occurs for aspects of language associated with meaning: –e.g. reference frames (Watson, Pickering & Branigan, 2005) The dot is left of the camera vs The dot is below the camera –e.g., description schemas (Garrod & Anderson, 1987) I’m at B5 vs I’m two along and two up

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – lexical choice Also alignment of lexical choice: –Use same words in same ways –box vs node –square = single node vs configuration of nodes

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – lexical choice May align on unusual/rare lexical choice: –e.g., if first person to name object is a non-native speaker

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – lexical choice May align on unusual/rare lexical choice: –e.g., if first person to name object is a non-native speaker Rocking chair

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – lexical choice May align on unusual/rare lexical choice: –e.g., if first person to name object is a non-native speaker Rocking chair The chair that can go back and forth

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – lexical choice May align on unusual/rare lexical choice: –e.g., if first person to name object is a non-native speaker Rocking chair The chair that can go back and forth You can shake your body Bortfeld & Brennan, 1997

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment Alignment at semantic and lexical levels may be linked to different types of meaning: –Alignment of perspectives on a situation (ways of thinking about the world) –E.g., rainbow trout vs coloured fish But other alignment seems to be unrelated to convergence on types of meaning –e.g., speech rate –e.g, syntax (alternatives express same meaning)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – syntactic choices

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – syntactic choices The artist is selling the dancer the gun (Double Object) vs The artist is selling the gun to the dancer (Prepositional Object)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment – syntactic choices The artist is selling the dancer the gun (Double Object) vs The artist is selling the gun to the dancer (Prepositional Object) A: The chef handing the book to the teacher B: The artist selling the gun to the dancer

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Branigan et al. (2000) Picture-description/-matching game –Stooge and naïve subject alternately describe pictures –Stooge produces scripted descriptions (50% PO, 50% DO) Stooge describes source picture to subject: –The nun showing the monk the banana Subject chooses matching picture: –Yup

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Branigan et al. (2000) Picture-description/-matching game –Stooge and naïve subject alternately describe pictures –Stooge produces scripted descriptions (50% PO, 50% DO) Stooge describes source picture to subject: –The nun showing the monk the banana Subject chooses matching picture: –Yup Subject describes target picture to matcher: –The artist selling the dancer the gun Matcher chooses matching picture: –Uh huh

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Confederate-scripting paradigm Confederate’s script Naïve participant Picture cards to be matched Confederate Blah Branigan et al (2000)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - syntax Strong tendency to align syntax with interlocutor: –More likely to produce a Prep Obj description after hearing a Prep Obj description than after a Double Obj description, and vice versa. –77% aligned descriptions when verb repeated –63% aligned descriptions when verb not repeated Chance = 50% –NB: speakers align sentence form: both alternatives have same denotational meaning (cf. semantic/lexical alignment) –Speakers align dynamically (produce both PO and DO)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - syntax Similar effects found: –for other structures: E.g., NP structure: a red cat vs a cat that’s red (Cleland & Pickering, 2003) –NB: Relative clause = strongly dispreferred in null context –in multi-party dialogues: Speakers syntactically align, whether directly addressed or not. A: The waitress is giving the monk the banana C: Yup

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - syntax Similar effects found: –for other structures: E.g., NP structure: a red cat vs a cat that’s red (Cleland & Pickering, 2003) –NB: Relative clause = strongly dispreferred in null context –in multi-party dialogues: Speakers syntactically align, whether directly addressed or not. A: The waitress is giving the monk the banana C: Yup B: The teacher is handing the cowboy the jug C: Uh huh Branigan et al. (in press)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - syntax Similar effects found: –in different communicative contexts: e.g., during walkie-talkie communication –in ‘special’ populations such as bilinguals, L2 learners, children…

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - syntax Similar effects found: –in different communicative contexts: e.g., during walkie-talkie communication –in ‘special’ populations such as bilinguals, L2 learners, children… El cerdo llora  El pingüino baila (SV)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - syntax Similar effects found: –in different communicative contexts: e.g., during walkie-talkie communication –in ‘special’ populations such as bilinguals, L2 learners, children… El cerdo llora  El pingüino baila (SV) Llora el cerdo  Baila el pingüino (VS) Flett, Branigan & Pickering, in prep

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment - summary Alignment with addressee’s linguistic behaviour: –is robust –can be very strong determinant of behaviour –occurs for many (all?) levels of linguistic structure –occurs for aspects of structure concerned with form, not just meaning

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment and audience design Alignment and audience design may co-occur. Put the penguin in the cup… vs the penguin that’s in the cup…

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment and audience design Alignment and audience design may co-occur. Put the penguin in the cup… vs that’s in the cup… –Priming: tendency to say The penguin that’s in the cup after hearing The sheep that’s on the plate –Audience design: tendency to say The penguin that’s in the cup when there are two penguins Haywood, Pickering & Branigan, 2005

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Explanations of alignment Social psychological approaches to alignment: –Identification with a particular social group (inc. addressee) –Reciprocity effects (linked to politeness norms?) –Such approaches explain why alignment of linguistic form occurs (in absence of meaning differences) –Perceived social identity of addressee is important e.g., politeness is only relevant for addressees that are perceived as social agents

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Explanations of alignment Alignment as audience design: –Alignment may reflect strategic effects (choosing to adopt other person’s perspective to enhance communication) –Choosing e.g., the same description schema or referential expression maximises the chances of effective communication –Such approaches do not really explain why alignment of linguistic form occurs (in absence of meaning differences) –Perceived identity of addressee is important e.g., depends on addressee being intentional agent inanimate entities do not have perspectives or beliefs

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Explanations of alignment Alignment as an automatic behaviour: –Primitive, default behaviour –children: align significantly more than adults (e.g., NP structure: 75% vs 30%) tend to align form, even when this leads to misunderstanding: –e.g., using same term with different reference (e.g., square) must learn to suppress tendency towards alignment where appropriate (Garrod & Clark, 1994).

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Explanations of alignment Alignment may be based on automatic priming mechanisms: activation-based account (Branigan et al., 2000) –comprehension of word or structure activates associated linguistic representations »VP  V NP PP –representations retain residual activation –residual activation facilitates subsequent selection

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Explanations of alignment Alignment may be fundamental to efficient communication (Pickering & Garrod, 2004) Efficient communication arises when interlocutors come to have the same understanding of relevant aspects of the world. –This arises from alignment of their situation models (e.g., Zwaan & Radvansky, 1998). –Alignment of situation models arises from alignment of other aspects of language (e.g., syntax, lexical choice). Alignment at one level promotes alignment at others Lexical alignment  syntactic alignment  semantic alignment

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Explanations of alignment Explanations are not mutually exclusive: multiple factors may underline alignment. –Most likely that at least some implicit element Participants generally report lack of awareness of alignment –But other factors may also contribute to overall effect: Basic (automatic) alignment effect may be enhanced by e.g. social factors (e.g., social status) Alignment at some levels of structure may be differentially susceptible to additional influences

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Implications for HCI Does alignment occur in HCI? –Pickering & Garrod (2004): alignment is fundamental to effective communication –So evidence that alignment is absent in HCI would imply that HCI is necessarily ineffective/inefficient. May highlight inherent difficulties in HCI –Conversely, evidence that alignment is present in HCI suggests possibility of effective communication in HCI

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Alignment in HCI? Why might we expect alignment in HCI? –Alignment seems to be a default linguistic behaviour: May be consequence of architecture of human language processor Might therefore expect to find it in any communicative context –Existing evidence that people treat computers as social actors Reeves & Nass (1996) They interact with them and evaluate them as they do other people  People should align with computers

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Leveraging alignment in HCI Alignment may be useful behaviour: –Facilitates predictions of user behaviour Reduces decision space in e.g., speech recognition –Allows simulation of human behaviour Users may feel more comfortable with systems that display human-like behaviour Users might feel more positively towards systems that align than those that do not –May allow [implicit] modification of human behaviour Users may modify their linguistic behaviour to align with system May allow modelling of desired input

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Patterns of alignment in HCI How might alignment in HCI pattern? If alignment is purely automatic priming: –Alignment would occur whenever a linguistic structure is encountered But in HHI, speakers’ (lexical and syntactic) choices are also influenced by: –a priori beliefs about addressee knowledge, capability etc –direct evidence about addressee knowledge, capability etc Hence beliefs about the knowledge, capability etc of a system might influence the extent to which speakers align with it.

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Relevant factors in HCI alignment A priori beliefs about social identity of system: –If systems are treated as social agents just like humans, then alignment should occur just as with humans. But which group of humans would count as comparable social agents? –Limited capability? –If systems are not treated as social agents just like to humans, then alignment may differ: Less alignment if alignment has substantial component related to e.g. politeness

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Relevant factors in HCI alignment Beliefs about system capability: –Users may assume computers to be less capable (generally and/or specifically linguistically). May increase likelihood of aligning, relative to HHI –Possibly different degrees of alignment with different systems May increase likelihood of aligning with unusual structure/expression –Effects of feedback: A priori beliefs may be updated in light of feedback Positive feedback of understanding may lead to comparable alignment as in HHI

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Relevant factors in HCI alignment Beliefs about addressee identity: –Modality may affect beliefs about addressee identity: Use of synthesised speech may emphasise non-humanness of addressee –Might enhance difference between HCI/HHI Use of (canned) speech may de-emphasise non-humanness of addressee –Might reduce differences between HCI/HHI

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Summary Alignment is potentially highly important in HCI – but there are many factors that might affect patterns of behaviour. –Specifically, many reasons why alignment in HCI might differ from alignment in HHI One important issue: –extent to which any differences between HCI and HHI are artifact of communicative situation rather than genuine differences between HCI and HHI Involvement of computer in communication

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Methodology Another person or a computer... or whatever Identical in all conditions ?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Methodology Picture-description experiments (as in HHI): –Single object: lexical choices –Actions: syntactic choices Manipulate participants’ beliefs about identity of addressee: –Human (at other end of remote connection) –Computer (at other end of remote connection) In fact, there is no interlocutor: –Participants always interact with a computer program that produces pre-scripted utterances –‘Reverse Wizard-of-Oz’

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Methodology Systematically control addressees’ utterances that participant encounters: –Actual linguistic behaviour of addressee is always identical in all conditions How do beliefs about addressee affect participants’ likelihood of lexical/syntactic alignment? –Differences in participants’ linguistic behaviour must be due to differences in beliefs about interlocutor. –NB: This method is also informative about alignment in computer-mediated HHI.

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Outline Series of experiments investigating: –Do people align with computers? –Focus on syntactic and lexical alignment Basic effects Influences of e.g., modality, a priori beliefs, feedback

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Possible patterns No alignment in HCI: –Only align with interlocutors that can experience positive affect or have beliefs/knowledge Equal alignment in HCI and HHI: –Compatible with cognitive economy accounts More alignment in HCI than HHI: –Beliefs about interlocutor’s affect alignment e.g. knowledge/capability (No alignment in HHI –People do not align in CMI)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic Interlocutor –Human vs. Computer Two separate experiments: Verb in interlocutor’s description –Same vs. Different in participant’s picture

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic Interlocutor’s description –Same PO: The pirate handing the cake to the sailor –Same DO: The pirate handing the sailor the cake

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic Interlocutor’s description –Same PO: The pirate handing the cake to the sailor –Same DO: The pirate handing the sailor the cake –Different PO: The pirate giving the cake to the sailor –Different DO: The pirate giving the sailor the cake

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic Interlocutor’s description –Same PO: The pirate handing the cake to the sailor –Same DO: The pirate handing the sailor the cake –Different PO: The pirate giving the cake to the sailor –Different DO: The pirate giving the sailor the cake Participant’s picture

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic alignment paradigm Interlocutor describes PO or DO Participant matches

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic alignment paradigm Participant describes PO or DO?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic alignment paradigm Interlocutor matches

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results Align with people and with computers –No difference in the magnitude of alignment 0 0.2 0.4 0.6 0.8 1 computerperson Proportion of aligned responses Different Verb

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 0 0.2 0.4 0.6 0.8 1 Results Align with people and with computers More alignment with computers than people computerperson Proportion of aligned responses Same Verb

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Syntactic alignment occurs in both CMI and HCI –Tendency to use same syntax as ‘human’ or ‘computer’ interlocutor When verb is repeated, more alignment in HCI than CMI –Only difference in in beliefs about interlocutor –Interlocutor’s behaviour identical

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Beliefs about interlocutor can affect strategic alignment –Overlies ‘basic’ (implicit, non-strategic) alignment –Syntactic choices not usually open to introspection, hence to strategic influences – only obvious in same-verb condition Lexical choices are usually more open to introspection –Hence should be generally more susceptible to strategic component

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical Interlocutor –Human vs. Computer Term used by the interlocutor –Preferred vs. Dispreferred term Experimental items pretested –Preferred term used more than 90% (e.g. bench) –Dispreferred term acceptable (e.g. seat)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical alignment paradigm Interlocutor describes Preferred or dispreferred term Participant matches

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical alignment paradigm Participant describes Preferred or dispreferred term?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical alignment paradigm Interlocutor matches

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results Align with people and with computers More alignment with computers than people computerperson 0 0.2 0.4 0.6 0.8 1 Proportion of aligned responses

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Lexical alignment occurs in both CMI and HCI Greater alignment with computers than people when the availability of such a strategy is obvious Beliefs about a system influence the extent that speakers align: –Strongly affects behaviour – >80% use of term that is strongly dispreferred (<10% use in null context)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Differences in alignment due to differences in a priori beliefs about interlocutor’s knowledge etc –Interlocutors’ linguistic behaviour identical in all conditions Alignment does not reflect beliefs based on feedback from computer’s earlier contributions –Received feedback from computer of understanding of syntax and lexical choices Conservative approach to HCI: A priori assumptions not updated in light of direct evidence –Cf. HHI evidence

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic: Modality Modality may affect salience of beliefs about interlocutor identity –Use of synthesised speech may emphasise non-humanness Hence promote alignment –Use of speech may deemphasise non-humanness Hence reduce alignment

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Syntactic: Synthesized Speech Syntactic alignment: PO/DO Interlocutor –Human vs. Computer Modality –Text vs. [Synthesized] Speech Interlocutor-text / Participant-text Interlocutor-text / Participant-speech Interlocutor-speech / Participant-text Interlocutor-speech / Participant-speech

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results More alignment with computers than people No effect of modality 0 0.2 0.4 0.6 0.8 1 computerperson Proportion of aligned responses

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Canned speech Same factors as previous –Spoken version Interlocutor’s “voice” pretested –Rated progressively distorted/impoverished versions for plausibility as a computer or person Chosen voice equally plausible as a computer or a person Voice gender-matched with participant

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results computerperson Proportion of aligned responses 0 0.2 0.4 0.6 0.8 1 Align with people and with computers More alignment with computers than people

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Combined analyses 0 0.2 0.4 0.6 0.8 1 Proportion of aligned responses 0 0.2 0.4 0.6 0.8 1 Interlocutor-text / Participant-textInterlocutor-spoken / Participant-spoken computerperson Although in right direction, no significant effect of modality

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Increased alignment in HCI is robust effect –Not reliably influenced by modality, whether synthesised or canned speech Beliefs about interlocutor that influence alignment seem to be impervious to relatively superficial and overt manipulations of ‘humanness’ –Not a dimension that needs to be worried about in systems design?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Persistence of alignment Alignment may persist over longer intervening material if interlocutor believed less capable –More effort to encode and recall interlocutors’ choices in order to maximise chances of effective communication If interlocutor believed to be capable: –Alignment with dispreferred term may occur in short-term (politeness) –But not over longer distances

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Persistence of alignment Interlocutor –Human vs. Computer Distance between interlocutor naming picture and participant subsequently naming it –Near vs. Far Near: Immediately preceding turn Far: Nine intervening filler turns

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results Align with people and with computers More alignment with computers than people 0 0.2 0.4 0.6 0.8 1 Proportion of aligned responses computer nearperson near person farcomputer far

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results Alignment with people decreases with increased distance 0 0.2 0.4 0.6 0.8 1 Proportion of aligned responses computer nearperson near person farcomputer far

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results Alignment with computers is maintained despite increased distance 0 0.2 0.4 0.6 0.8 1 Proportion of aligned responses computer nearperson near person farcomputer far

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Need for reciprocity/politeness to other people diminishes Maintain high level of alignment with computers –Conservative approach in HCI: Go to a great deal of effort to maximise the chances of effective communication

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Capability Alignment to computers: –Robust effect –Determined by a priori beliefs about addressee knowledge, capability etc –Not influenced by direct evidence about addressee knowledge, capability etc Can we manipulate participant’s beliefs about knowledge, capability etc of interlocutor?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Capability of computer HCI condition only Interlocutor –‘Basic’ computer vs. ‘Advanced’ computer Manipulate beliefs about capability –Actual behaviour identical in both conditions –Only difference: start-up screen for 10 seconds prior to the experiment

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Basic computer manipulation

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Advanced computer manipulation

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results Align with basic and advanced computers More alignment with basic than advanced computer basic computeradvanced computer 0 0.2 0.4 0.6 0.8 1 Proportion of aligned responses Pearson et al (CHI-2006)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Beliefs about computer’s capabilities influence alignment Beliefs can be manipulated through simple, superficial, non-functional aspects of system –Not affected by actual behaviour of system (evidence of understanding) Does conservatism wrt updating beliefs on basis of feedback in HCI reflect something specific to HCI? –Are speakers similarly conservative in HHI when interacting with humans of unknown capability? –Test group: Non-native speakers

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Capability of person CMI conditions only Interlocutor –Native person vs. Non-native person Manipulate beliefs about capability –Actual behaviour identical in both conditions –Only difference: participants told they were interacting with native or non-native person

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results No difference in alignment with non-native than native people 0 0.2 0.4 0.6 0.8 1 native personnon-native person Proportion of aligned responses

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interim summary Greater alignment with basic than advanced computer –Determined by a priori beliefs about interlocutor’s knowledge, capability etc –Not influenced by direct evidence about interlocutor’s knowledge/capability No greater alignment with non-native than native person –Influenced by direct evidence about interlocutor’s knowledge/capability What if we remove direct evidence about person interlocutor’s knowledge/capability?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Lexical: Capability of person(2) Same factors as previous Interlocutor –Native person vs. Non-native person No feedback from interlocutor when matching –No direct evidence about interlocutor’s understanding

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Feedback Interlocutor matches Participant has described

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 No feedback Interlocutor matches Participant has described

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Results 0 0.2 0.4 0.6 0.8 1 native personnon-native person Proportion of aligned responses Greater alignment with non-native than native people

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Beliefs about mental states Interlocutors establish a “common ground” of shared background knowledge Clark & Marshall (1981) –Each interlocutor works out what they assume the other knows about the situation Common ground can be accrued on the basis of: –Linguistic co-presence: What has been said previously –Community membership: What you can infer on the basis of them belonging to a particular social group

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Beliefs about mental states Greater alignment with computers than people and with basic than advanced computers –Computers have radically different mental states than us –Unclear what capabilities computers have, so align to maximise the chances of effective communication Greater alignment with non-native than native speakers only in absence of feedback –Non-natives have similar mental states to us With feedback: beliefs are up-dated. Based on their contributions, non-natives are capable enough Without feedback: cannot assume they are capable, so align to maximise the chances of effective communication

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Conclusions HCI is like HHI in important ways: –Tendency to align with interlocutor –Both lexical and syntactic alignment But important differences: –Differences in a priori beliefs about human vs computer interlocutors influences magnitude of alignment –Where different choices salient, speakers align more with computers than with people Strategic component additional to automatic component of alignment

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Conclusions Beliefs in HCI can be manipulated easily –Superficial non-functional aspects of system But beliefs in HCI are not readily updated on basis of behavioural evidence

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Interface design implications Users adapt their behavior to fit the system. This adaptation reflects beliefs about the system’s capability, and not its actual behaviour Improved natural language recognition –To elicit high degrees of alignment, a system should not claim “humanness” A computer that tries to act smarter than it actually is is disliked Wang et al (2005 ) –Consistent use of words/syntax by interface will implicitly limit users’ input, reducing burden on recognition systems –May also have affective implications

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Future directions: Align to listener Alignment/adaptation to one’s interlocutor increases: –Efficiency executing conversational tasks Clark & Wilkes- Gibbs (1986) –Likelihood of achieving higher level goals such as changing beliefs Carenini & Moore (2000) –Waitress’ tips van Baaren et al (2003) –Social Psychology: Increased liking; marking as in-group … Are people affected by computers aligning with them? Do people prefer computers that align with them ?

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Future directions: Align to listener Alignment to the user by the system might affect –Level of alignment by user (can be leveraged to improve recognition accuracy) –Liking of the system and the other users –Perceived credibility (expertise and trustworthiness) –Purchase behaviour –Task performance –Web stickiness (amount of time spent on a site) –Branding effectiveness (multiple visits to a site)

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Align to listener: Design Interlocutor –Human vs. Computer Interlocutor’s responses –Align vs. Misalign with participant’s Measure characteristics of participant’s responses –Align or misalign with interlocutor? Affective measures –Perceptions of the interaction, self, and the interlocutor

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Align to listener: Design Interlocutor:“boat”

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Align to listener: Design Interlocutor:“boat” Participant:“jacket”

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Align to listener: Design Interlocutor:“boat” Participant:“jacket” Interlocutor:“jacket”“coat” alignmisalign

Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006 Align to listener: Design Interlocutor:“boat” Participant:“jacket” Interlocutor:“jacket”“coat” Participant:align and say “boat”? alignmisalign

Alignment in human-human and human-computer interactions Holly Branigan Jamie Pearson University of Edinburgh.

Similar presentations

Presentation on theme: "Alignment in human-human and human-computer interactions Holly Branigan Jamie Pearson University of Edinburgh."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Alignment in human-human and human-computer interactions Holly Branigan Jamie Pearson University of Edinburgh.

Similar presentations

Presentation on theme: "Alignment in human-human and human-computer interactions Holly Branigan Jamie Pearson University of Edinburgh."— Presentation transcript:

Similar presentations

About project

Feedback