1. Interacting with AI Module 3 Working and living with AI
Amela Karahasanović

2. Module overview [3.1 & 3.2] Evaluation of interaction with AI
Task-oriented evaluation
Ability based evaluation
User Experience, Values and AI
[3.3 & 3.4 ] Human- AI partnership
Human-robots teams
Task distribution between humans and AI
Levels of automation
Human-in-the-loop and situation awareness)
[3.5 & 3.6] Lessons learned from studies of human – AI interaction
Personality of robots and trust
Empathy with algorithms
Evaluation in the lab and "in the wild"
Evaluation in industry context
[3.7 & 3.8] Writing workshop

3. [3.5 & 3.6 ] Lessons learned from studies of human – AI interaction
Some examples
Four tasks and discussions

4. EXAMPLES

5. Design patterns for exploring and prototyping human- robot interaction
Introductory Monologue
"Hi Welcome…"
Questions and answer pair
S: "Do you know who Marvin the Martian is?", L: "Oh yeah,…"
Generic comment and personal comment
"Interesting" "I had a similar experience once…"
Monologue and generic comment
S: "…and then, all of the sudden, thousands of these aliens appear at the Earth." L:"That's a lot"
Instructions and action
I: "Now connect the long pipe with the one shaped like an S" S: <locates the pipe and connect it>
Finished comment
P:"I think that's it, so we should be done
Wait
Based on a formative evaluation, 16 participants, 5 social interaction scenarios (conversation, collaboration, instruction, interview, storytelling), coding to discover patterns
Sauppe, A., Mutlu, B., CHI 2014

6. Design for Human - algorithmic experience
Storyboard key scenes with the algorithm present and observing, taking in various kinds of data and drawing certain conclusions
Act out being an algorithm
Write a play-like script for algorithmic dialog with users
Visually map out all the system touchpoint between users and smart objects/interfaces
U.Gajender, Empathizing with the smart and invisible – algorithms, IX, 23/4

7. Can robot (companion for children) express parenting styles?
A Robot with Style, because you are Worth it!, W. Johal, G. Calvary and S. Pesty, CHI 2014
Can robot (companion for children) express parenting styles?
88 participants
Two conditions (more dominant and less dominant – use of social cues)
Two robots: a robotic head and a humanoid robot
Users were asked to score the behavior of the robot based on a short video and answer the questionnaire (two parenting styles: authoritarian and neglectful)

8. To express dominance they used: blinking, body openness… Findings:
The effect of the style on perceived authoritativeness
The effect of the robot choice – one with only facial expressions was experienced as more authoritative

9. RoboCare A system that assists elderly at home Evaluation
Video based trials, 8 short movies with different domestic scenarios
User initiative and system initiative
100 participants (59 – 90 years
Questionnaire (background, evaluation)

10. Findings High usefulness and acceptability
Useful for personal and environmental safety, reminding to take medications and finding objects
Not useful for providing suggestions
Low scores for activities which are not central for everyday life
Appreciated help with cognitive problems (reminding them to do something, finding things)

11. Can robots manifest a personality
AIBO (robot dog)
48 participants based on the personality scores
Randomly assigned to the introvert or extrovert robot
(4 conditions)
Verbal cues
Asked to interact with AIBO (verbally and tactile commands)
AIBO respond (extrovert turn faster and wider)
Questionnaire on personality, intelligence and attractiveness of AIBO
Participant enjoyed the interaction more with a compatible robot

12. Tasks Design a study to evaluate: Describe
TASK 1 – A self-driving car (drivers still have control)
TASK 2 – A robot that distribute medicine in a hospital
TASK 3 – A recommendation engine (Netflix, Google…)
TASK 4 - A chatbot of an insurance company
Describe
Objectives
Special properties to think about (compared to other UX/usability studies)
Methods (what are you going to evaluate and how)
Advantages and disadvantages of the selected method
Alternatives for the evaluation

13. Things to consider Formative or summative Home versus working context
Inside versus outside
Who interacts with this AI agent
Lab and field studies
Duration of the study (longitudinal versus shorter studies)
Personality
Trust
How training of ML fits with the evaluation

14. Assignments and tasks Group Assignments and tasks
Task 1 (lesson 3.1&3.2) – problems with AI -> Appendix 3 (a video and a ½ page)
Task 2 (lesson 3.3&3.4) – human-machine partnership -> Appendix 4 (1 page)
Evaluation approach/plan and reflections on the proposed plan (one page max)
Individual assignment
Levels of automation (lesson 3.3&3.4, 1/2 – 1 page)