Technologies to Improve Quality of Life for People with Cognitive Disabilities Henry Kautz Department of Computer Science University of Rochester Rochester, NY

Assisted Cognition Systems Henry Kautz Department of Computer Science University of Rochester Rochester, NY

Vision Understanding human behavior from sensor data Actively prompting, warning, and advising Alerting caregivers as necessary Computer systems that improve the independence and well-being of people suffering from cognitive disabilities by…

Foundations Pervasive sensing Movement, location, object manipulation, physiology On user or embedded in the environment AI / machine learning Model everyday life Adapt to individual's behavior and goals Human computer interaction Keep focus on user Ease of use

Outline Designing an adaptive wayfinding system for cognitively impaired users Creating a context-aware task prompting system Challenges in research on assisted cognition systems

Outline Designing an adaptive wayfinding system for cognitively impaired users – U Washington Computer Science Alan Liu, Gaetano Borriello – U Rochester Computer Science Henry Kautz – U Washington Rehabilitation Medicine Kurt Johnson, Mark Harniss, Pat Brown Creating a context-aware task prompting system Challenges in research on assisted cognition systems

Wayfinding Difficulties Alice doesn’t go to new places by herself because even with practice, she still has trouble remembering directions. Bob gets lost easily, and cannot use a map to orient himself and figure out where he is or where to go. Carol takes a regular route between home and work, but one day a road along the way is closed and she doesn’t know how to adapt. 7

GPS Systems Why aren't regular GPS systems adequate? "One size fits all" route planning – Minimize distance or time when driving "One size fits all" presentation of directions – Map and turn-by-turn directions What makes a route safe and easy to follow by a cognitively impaired user?

Research Challenge Individuals with cognitive impairment can have vastly different abilities and disabilities – Designing for a "universe of one" (Kintsch & DePaula) However: – Can we uncover common features that lie behind successful solutions? – Can we incorporate these features into a general system that adapts to the individual? 9

Research Path Formative usability studies – Studied direction-following capabilities and priorities – Potential users interacting with prototypes in situ Landmark-based directions – Incorporated landmarks into wayfinding – Balancing cognitive effort and attention to environment Modeling individual variation – Implemented framework to generate customized and adaptable directions – Measure impact of adaptation on users

` Liu et al. ASSETS 2006, PervasiveHealth 2009 Formative Studies Indoor and outdoor wayfinding using a "Wizard of Oz" system Wide variation in user preference and expectation of directions – Modality (image, audio, text) – Complexity – Timing Attention demand for turn-by-turn directions problematic Alternative: landmark-based directions

Landmark-Based Directions Landmark-based directions – Encourage attention to world – Orient user – More natural What are features of good landmarks? 9 study participants – TBI, stroke, MS, Dev Dis. A.L. Liu, H. Hile, H. Kautz, G. Borriello, P.A. Brown, M. Harniss, K. Johnson, PervasiveHealth 2009

Landmark Selection Leverage collections of geo-tagged photos – Heuristic: more popular photos more useful Can augment / warp photos to make them more closely match user's perspective

Results 180 total directions given, 150 correctly followed – Much variation between participants Features of most successful landmark-based directions: – In perspective – Close – Path-aligned Making turns in relation to landmark required more cognitive effort – Turn-based directions should be user-centric

Modeling Individual Variation Studies showed how to create a wayfinding system that was usually better for most users However: system was still far from optimal for many users Can we design a system that adapts to the individual user?

Adaptation as Optimization Approach: for each user, learn a function: Generating directions = finding optimal solution to a sequential decision problem Issue: scarcity of user-specific training data – Solution: use data from all users, but weigh particular user's data more heavily

Adaptation Study 5 participants Compared performance on – Route generated with generic model – Route generated with personalized model Qualitative result: personalized routes considered easier – Need larger, longer study to quantify significance 17

Summary Studied wayfinding with real potential users – Learned range of abilities and preferences – Informed technological requirements Incorporated landmarks as wayfinding option – Explored landmark dimensions that affect usability Proof-of-concept of automatic adaptation – Creating a personal model using data from both the individual and the larger user base 18

Outline Designing an adaptive wayfinding system for cognitively impaired users Creating a context-aware task prompting system – Attention Control Systems: Richard Levinson – University of Rochester: Henry Kautz – VA Palo Alto: Dr. Harriet Zeiner – UC Santa Cruz: Beth Ann Hockey Challenges in research on assisted cognition systems

PEAT: handheld-based scheduling and cueing system for persons with executive function impairment (Attention Control Systems) – Multimedia prompts to start and end tasks – Assists with task (re)scheduling, by reasoning about deadlines, durations, and priorities PEAT

Problems with Prompting Systems Require input from user whenever a task begins or ends – Cost of use may discourage users Prompts may be out of sync – Self-initiation confuses system – Encourages dependency No support for contingent prompts – Remind the user to take her walker when she goes outside

Integrating Sensing and Prompting Goal: use information from sensors to improve prompting Determine user's current activity from – Location: Home automation motion sensors, GPS – Objects user is handling: Intel Research Seattle RFID reader bracelet – Schedule of expected activities

Context-Aware PEAT R&D funded by grants from DARPA, Office of Defense, and NIH Platform: Android cell phone – Internal sensors: GPS, acceleration – Home base station (laptop) for setup and relaying data from Insteon home automation sensors and Intel Research RFID reader – State estimator: employs probabilistic algorithms to infer user activity and learn user model State Estimator Cellphone Wireless connections GPS State DB Daily Activity Planning & Cueing PEAT User Interface PEAT RFID reader bracelet WiFi Relay RF ZigBee WiFi Home sensors User

Reduced Prompting Prompting cancelled if user self-initiates (or self- terminates) task – Reduces confusion / out of sync prompts – Encourages, rather than punishes, self-initiation Can system do even more to encourage self- initiation? – Perhaps: prompt later than scheduled time – If prompt is too late and user does not self-initiate, task will fail – What is the optimal time for system to prompt?

Optimal Prompting System can be configured to use a fixed prompting schedule (e.g. 5 minutes before scheduled time) or an adaptive schedule Caregiver specifies rewards associated with – Successful (timely) task completion – User self-initiation of task System – Learns model of user self-initiation – Computes prompting time that maximizes expected reward

Learning the User Model User model includes probability distribution over how long it takes user to self-initiate – Technical challenge: how to estimate this parameter quickly and accurately? – Can use statistical techniques developed for reliability analysis

System Trial In-home trials by Palo Alto VA start next month Help regularize patients' daily schedule for – Sleep – Meals – Exercise – Behavioral therapy "homework"

Outline Designing an adaptive wayfinding system for cognitively impaired users Creating a context-aware task prompting system Challenges in research on assisted cognition systems

Point versus General Solutions “[A] tension exists between satisfying the need for customized solutions that address the immediate needs of users, and identifying generalizable results that can be used to achieve long-term research goals.” [Generalizability in Research with Cognitively Impaired Individuals, Moffatt, Findlater, Allen, 2006]

Evaluating with a Universe of One “Twenty-eight papers presenting 25 studies were reviewed. The total number of participants was 423. Most identified papers described case reports or non-randomized clinical trials. Only one randomized controlled trial was identified.” [Efficacy and usability of assistive technology for patients with cognitive deficits: a systematic review, Joode, Heugten, Verhey, & Boxtel, 2010]

The Training Hurdle “The [studies] that do describe the exact time spent on training show a substantial amount of variation (from 30 minutes up to 9 hours).” [Efficacy and usability of assistive technology for patients with cognitive deficits: a systematic review, Joode, Heugten, Verhey, & Boxtel, 2010]

Promise of Context-Aware Systems Medication Prompting: “The context-based prompting resulted in significantly better adherence (92.3%) as compared to time-based (73.5%) or no prompting (68.1%) conditions (p < , χ2 = 17.0). In addition, subjects had better adherence in the morning than in the evening. We have shown in this study that a system that generates reminders at an opportune time to take the medication significantly improves adherence.” [A Study of Medication-Taking and Unobtrusive, Intelligent Reminding, Hayes, Cobbinah, Dishongh, Kaye, Kimel, Labhard, Leen, Lundell, Ozertem, Pavel, Philipose, Rhodes, & Vurgun, 2009]

Pitfalls of Context-Aware Systems “The participant was building a mental model of how the system behaved, and his model did not map well onto the behavior of the system. The problem was especially pronounced when an activity was not detected properly.... The participant further admitted to changing his behavior in order to ‘fool‘ the system, but since he didn’t understand how it worked, this behavior was counter-productive. [User-adaptive Reminders for Home-based Medical Tasks, Kaushik, Intille, Larson, 2008]