Assisted Cognition Henry Kautz University of Rochester Computer Science
Vision Understanding human behavior from sensor data Actively prompting, warning, and advising Alerting caregivers as necessary Computer systems that improve the independence and safety of people suffering from cognitive limitations by…
Technical Foundations Inexpensive, easily deployed sensors GPS cell phones – location RFID tags – object manipulation Wearables – motion, sound, pulse,... Advances in algorithms for high-level analysis of sensor data
General Approach user profile general knowledge wearable sensors environmental sensors decision making user interface caregiver alerts behavior cognitive stateintentionsactivity
Examples Maintaining a daily schedule Compensating for memory problems Compensating for lowered self-initiative Navigation Indoors or outdoors Performing activities of daily living Step-by-step prompting Behavior regulation Improving self-awareness Safety and wellness Need for immediate help Long term health trends
Where is the Work Being Done? Field is coming together from many communities Computer science – artificial intelligence, robotics, ubiquitous computing Rehabilitation engineering Rehabilitation medicine Assisted-living & nursing homes Gerontology
ACCESS Assisted Cognition in Community, Employment, & Social Settings
Example: Community Access for the Cognitively Disabled
Problems in Using Public Transportation Learning bus routes and numbers Transfers, complex plans Recovering from mistakes
Current GPS Navigation Devices Designed for drivers, not bus riders! Should I get on this bus? Is my stop next? What do I do if I miss my stop? Destination manually entered High cognitive overhead Device decides which route is “best” Familiar route better than shorter one Catastrophic failure when signal is lost Frequent “dead zones” in urban areas
Idea User carries GPS cell phone System infers user’s state Walking? Getting on a bus? System learns about user Important places, routes Breaks from routine = user may be confused or lost Offer help Call caregiver
Transportation Routines BA Goal: intended destination Workplace, home, friends, restaurants, … Trip segments: Home to Bus stop A on Foot Bus stop A to Bus stop B on Bus Bus stop B to workplace on Foot WorkplaceHome
GPS reading z k-1 zkzk Edge, velocity, position x k-1 xkxk k-1 kk Data (edge) association Time k-1 Time k m k-1 mkmk Transportation mode t k-1 tktk Trip segment g k-1 gkgk Goal Cognitive mode { routine, novel, error } Dynamic Bayesian Network c k-1 ckck
Error Detection: Missed Bus Stop
GPS camera-phone “Knocks” when there is an opportunity to help Can I guide you to a likely destination? I think you made a mistake! This place seems important – would you photograph it? Prototype: Opportunity Knocks
CARE Cognitive Assistance in Real-world Environments
Goal A home monitoring system that Assists user in performing activities of daily living Tracks activities, and provides prompts and warnings as needed Can be deployed in an ordinary home
Initial Application Accurate, automated ADL logs Changes in routine often precursor to illness, accidents Human monitoring intrusive & inaccurate Image Courtesy Intel Research
Technical Requirements Sensor hardware that can be practically deployed in a ordinary home Methods for activity tracking from sensor data Methods for automated prompting that consider Probability of user errors Probability of system errors Cost / benefit tradeoffs
Object-Based Activity Recognition Activities of daily living involve the manipulation of many physical objects Kitchen: stove, pans, dishes, … Bathroom: toothbrush, shampoo, towel, … Bedroom: linen, dresser, clock, clothing, … We can recognize activities from a time- sequence of object touches
Sensing Object Manipulation RFID: Radio- frequency identification tags Small Semi-passive Durable Cheap Near future: use products’ own tags
Wearable RFID Readers Designed by Intel Research Seattle, samples given to a few academic partners
Hidden Markov Model Trained on labeled data 10-fold cross validation 88% accuracy
Improving Robustness Tracking fails if novel (but reasonable) objects are used Solution: smooth parameters over abstraction hierarchy of object types
Experiment: ADL Form Filling Tagged real home with 108 tags 14 subjects each performed 12 of 14 ADLs in arbitrary order Given trace, recreate activities
Results: Detecting ADLs ActivityPrior Work SHARP Personal Appearance92/92 Oral Hygiene70/78 Toileting73/73 Washing up100/33 Appliance Use100/75 Use of Heating84/78 Care of clothes and linen100/73 Making a snack100/78 Making a drink75/60 Use of phone64/64 Leisure Activity100/79 Infant Care100/58 Medication Taking100/93 Housework100/82 Legend Point solution General solution Inferring ADLs from Interactions with Objects Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz IEEE Pervasive Computing, 4(3), 2004 RFID
Current / Future Directions More detailed sensor data Machine vision fused with direct sensing More detailed models of cognitive state Affect recognition Automated inventions Prompting & guidance Interaction: Audio? Visual? Tactile? Interactive design/testing with various target populations TBI Alzheimers Autism