1. Assisted Cognition Henry Kautz University of Rochester Computer Science

2. Vision Understanding human behavior from sensor data Actively prompting, warning, and advising Alerting caregivers as necessary Computer systems that improve the independence and safety of people suffering from cognitive limitations by…

3. Technical Foundations Inexpensive, easily deployed sensors GPS cell phones – location RFID tags – object manipulation Wearables – motion, sound, pulse,... Advances in algorithms for high-level analysis of sensor data

4. General Approach user profile general knowledge wearable sensors environmental sensors decision making user interface caregiver alerts behavior cognitive stateintentionsactivity

5. Examples Maintaining a daily schedule Compensating for memory problems Compensating for lowered self-initiative Navigation Indoors or outdoors Performing activities of daily living Step-by-step prompting Behavior regulation Improving self-awareness Safety and wellness Need for immediate help Long term health trends

6. Where is the Work Being Done? Field is coming together from many communities Computer science – artificial intelligence, robotics, ubiquitous computing Rehabilitation engineering Rehabilitation medicine Assisted-living & nursing homes Gerontology

7. ACCESS Assisted Cognition in Community, Employment, & Social Settings

8. Example: Community Access for the Cognitively Disabled

9. Problems in Using Public Transportation Learning bus routes and numbers Transfers, complex plans Recovering from mistakes

10. Current GPS Navigation Devices Designed for drivers, not bus riders! Should I get on this bus? Is my stop next? What do I do if I miss my stop? Destination manually entered High cognitive overhead Device decides which route is “best” Familiar route better than shorter one Catastrophic failure when signal is lost Frequent “dead zones” in urban areas

11. Idea User carries GPS cell phone System infers user’s state Walking? Getting on a bus? System learns about user Important places, routes Breaks from routine = user may be confused or lost Offer help Call caregiver

12. Transportation Routines BA Goal: intended destination Workplace, home, friends, restaurants, … Trip segments: Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot WorkplaceHome

13. GPS reading z k-1 zkzk Edge, velocity, position x k-1 xkxk  k-1 kk Data (edge) association Time k-1 Time k m k-1 mkmk Transportation mode t k-1 tktk Trip segment g k-1 gkgk Goal Cognitive mode { routine, novel, error } Dynamic Bayesian Network c k-1 ckck

14. Error Detection: Missed Bus Stop

15. GPS camera-phone “Knocks” when there is an opportunity to help Can I guide you to a likely destination? I think you made a mistake! This place seems important – would you photograph it? Prototype: Opportunity Knocks

16. CARE Cognitive Assistance in Real-world Environments

17. Goal A home monitoring system that Assists user in performing activities of daily living Tracks activities, and provides prompts and warnings as needed Can be deployed in an ordinary home

18. Initial Application Accurate, automated ADL logs Changes in routine often precursor to illness, accidents Human monitoring intrusive & inaccurate Image Courtesy Intel Research

19. Technical Requirements Sensor hardware that can be practically deployed in a ordinary home Methods for activity tracking from sensor data Methods for automated prompting that consider Probability of user errors Probability of system errors Cost / benefit tradeoffs

20. Object-Based Activity Recognition Activities of daily living involve the manipulation of many physical objects Kitchen: stove, pans, dishes, … Bathroom: toothbrush, shampoo, towel, … Bedroom: linen, dresser, clock, clothing, … We can recognize activities from a time- sequence of object touches

21. Sensing Object Manipulation RFID: Radio- frequency identification tags Small Semi-passive Durable Cheap Near future: use products’ own tags

22. Wearable RFID Readers Designed by Intel Research Seattle, samples given to a few academic partners

23. Hidden Markov Model Trained on labeled data 10-fold cross validation 88% accuracy

24. Improving Robustness Tracking fails if novel (but reasonable) objects are used Solution: smooth parameters over abstraction hierarchy of object types

25. Experiment: ADL Form Filling Tagged real home with 108 tags 14 subjects each performed 12 of 14 ADLs in arbitrary order Given trace, recreate activities

26. Results: Detecting ADLs ActivityPrior Work SHARP Personal Appearance92/92 Oral Hygiene70/78 Toileting73/73 Washing up100/33 Appliance Use100/75 Use of Heating84/78 Care of clothes and linen100/73 Making a snack100/78 Making a drink75/60 Use of phone64/64 Leisure Activity100/79 Infant Care100/58 Medication Taking100/93 Housework100/82 Legend Point solution General solution Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004 RFID

27. Current / Future Directions More detailed sensor data Machine vision fused with direct sensing More detailed models of cognitive state Affect recognition Automated inventions Prompting & guidance Interaction: Audio? Visual? Tactile? Interactive design/testing with various target populations TBI Alzheimers Autism