1. 1 Assisted Cognition Henry Kautz, Oren Etzioni, Dieter Fox, Gaetano Borriello, Larry Arnstein University of Washington Department of Computer Science & Engineering

2. 2 An Epidemic of Alzheimer’s Disease

3. 3

4. 4

5. 5 Statistics for United States 4.6 million people with Alzheimer’s 16 million people by 2050 Today costs $100 billion @ year for care Additional $61 billion in lost productivity from family members $ ½ Trillion total cost by 2050!

6. 6 Lost Competencies Short-term memory Ability to carry out complex tasks (driving, paying bills, cooking, house-hold tasks) Ability to orient self in time and space Memory of events Dressing, bathing, cooking, eating Memory of concepts Self-initiative Recognize friends, relatives

7. 7 Cognition in Context Can often compensate for physical disabilities by change in environment Wheelchairs Cognitive competence also depends on environment Physical Social

8. 8 Social Context

9. 9 cooking dressing gardening self-medicating personal grooming shopping exercise

10. 10 Problem Caregiver burnout ½ of all family caregivers suffer depression “The 36 Hour Day” Far too few professional caregivers to provide constant 1-on-1 help in institutional settings Already a nationwide shortage of good staff

11. 11 Assisted Cognition Systems Learn to interpret human behavior from low- level sensory data General commonsense knowledge Patterns of behavior idiosyncratic to the particular user External data sources Actively offer prompts and other forms of help as needed Alert human caregivers when necessary

12. 12 Computational Infrastructure Ubiquitous computing environment Sensors – position, motion, sound, vision Output – speech, graphics, robots Portable wireless computing devices Software layer on top of ubiquitous computing technology

13. 13 Architecture

14. 14 Applications The Activity Compass The Adaptive Prompter

15. 15 The Activity Compass Goal: help person safely and independently move about the community (including use of public transit) User carries GPS/wireless equipped PDA AC system tracks user’s position, predicts where user is going based on past experience System offers help when Inferred plan is likely to fail otherwise (e.g. miss bus) User is likely to be lost or disoriented (wandering, on wrong bus) User explicitly consults PDA

16. 16 Gathering Data

17. 17 green – GPS readings (10 sec), yellow – location estimation (probability distribution)

18. 18 Creating the User Model Training Data: 20,000 GPS readings 3 weeks of occasional use Reduce noise using Kalman filter Hand labeled by mode of transportation Walking, In Car, On Bus, Riding Bike, Inside Predicting current mode Input: current location/time/velocity Decision tree learning: 98.9% accuracy (10 FCV) Predicting next mode transition(s) Input: current mode/location/time/velocity Decision tree learning: 98.8% accuracy (10 FCV)

19. 19 Crisis Prediction (I) When might user need help? “Cutting it too close” Associated with each leaf of decision tree is a spatio-temporal window Compute expected position of next transition within that window If position is at or near upper temporal boundary, increased probability that expected transition will fail to hold (e.g., user will miss the bus!)

20. 20 Crisis Prediction (II) When might user need help? External changes in the world Some kinds of transitions (e.g. board bus) are enabled by external forces (the bus!) Real-time Seattle transit information available online Use information to more finely label training data Crisis = prediction that is inconsistent with external information

21. 21 Crisis Prediction (III) When might user need help? Novel events After-the-fact discovery that predicted behavior did not occur Ask user to confirm actions are intended Explicit error models Dangerous locations Wandering trajectory

22. 22 Intervention Strategies Must balance disutility of crisis cost of annoying user probability of crisis do not want to over-rely on negative reinforcement Qualitative preference language “Never let me miss a bus late at night”

23. 23 User Interface

24. 24 The Adaptive Prompter Goal: help a person carry out a multi- step task Smart home tracks residents and objects Hierarchical recognition model Simple behavior (sleeping, walking) Simple actions (get into bed) Meaningful patterns of actions – plans Model of possible failures and interventions

25. 25 Example 1. Joe enters bathroom at 9:00 am. 2. He turns on water, and picks up toothbrush. 3. Nothing happens for 15 seconds. AC system recognizes “tooth brushing” activity has stalled. 4. Prompts Joe to pick up toothpaste. Joe does so and completes task. 5. Joe leaves bathroom with water still running. AC system gently encourages Joe to go back and turn it off.

26. 26 Towards a Behavior Description Language Requirements probabilities on fluents and events continuous (or finely discretized) time probability distributions on temporal relations between events hierarchical events plans – intended complex events defective plans system interventions utilities of defects and interventions

27. 27 Approach Develop scenarios for AP in consultation with experts on Alzheimer’s care Prototype specification language Semantics via translation into Dynamic Bayesian Networks Interventions: Dynamic Decision Networks A terrific KR challenge! See work by Martha Pollack, Hung Bui, Geib & Goldman, Daphne Koller

28. 28 Data Source: Elite Care Oakfield Estates

29. 29 People University of Washington Computer Science & Engineering UW Medical Center Alzheimer’s Disease Research Center (ADRC) UW Institution on Aging Intel Research People & Practices – User studies of future technology needs Intel Research Seattle – Ubiquitous computing Elite Care Oakfield Estates assisted living http://assistcog.cs.washington.edu/

30. 30 Advertisement UbiCog 2002 – Workshop on Ubiquitous Computing for Cognitive Aids September 29, 2002 Gothenberg, Sweden Part of UBICOMP-2002, the major ubiquitous computing conference Slots for speakers still available, email Henry Kautz