The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine) Kurt Johnson, Pat Brown, Mark Harniss (UW Rehabilitation Medicine) Matthai Philipose (Intel Research Seattle)

Trend 1: Sensing Infrastructure  Robust direct-sensing technology oGPS-enabled phones oRFID tagged products oWearable multi-modal sensors  Rapid commercial deployment

Trend 2: Healthcare Crisis  Demand for community integration of the cognitively disabled year disabled by traumatic brain injury o7.5 million in US with mental retardation o4 million in US with Alzheimer’s  Family burnout  Nationwide shortage of professionals

Assisted Cognition  Technology to support independent living by people with cognitive disabilities oat home oat work othroughout the community by oUnderstanding human behavior from sensor data oActively prompting and advising oAlerting human caregivers when necessary

Building Partnerships  UW Assisted Cognition seminar oCSE, medicine, nursing, Intel  ACCESS oUW CSE & Rehabilitation Medicine oGrant from NIDDR (Dept. of Education) oHelp cognitively disabled use public transportation oPrototype: Opportunity Knocks  Intel Proactive Health effort oComputing for wellness & caregiving oPromote partnerships with government, universities, healthcare organizations oIntel Seattle: sensors for activity tracking

Example  Way-finding Assistant oHelp user travel throughout community  On foot  Using public transportation oDetect user errors  Proactively help user recover  “You missed your stop, so get off at the next stop and then wait for the #16 bus...” oPotential users  TBI, MR, mild memory impairment

Example  ADL Assistant oActivities of daily living  Eating, bathing, dressing,...  Cooking, cleaning, ing,... oMonitoring  Changes in ADLs signal changes in health oReminding / prompting  “Time to take your blue meds” oStep-by-step guidance  “Turn on the tap... now pick up the brush...” oPotential users  Disabled, ordinary aging

General Model user model common- sense KB geospatial DB wearables sensors environmental sensors intervention decision making user interface caregiver alerts

General Model common- sense KB geospatial DB wearables sensors environmental sensors intervention decision making user interface caregiver alerts physical motion & position cognitive stategoalsactivity

Deciding to Intervene A = system intervenes G = user actually needs help

ACCESS Way-finding Assistant supported by National Institute on Disability & Rehabilitation Research DARPA IPTO

The Need: Community Access for the Cognitively Disabled

Problems in Using Public Transportation Learning bus routes and numbers

Problems in Using Public Transportation Learning bus routes and numbers Transfers, complex plans

Problems in Using Public Transportation Learning bus routes and numbers Transfers, complex plans Recovering from mistakes

Result Need for extensive life-coaching Need for point-to-bus service

Result Need for extensive life-coaching Need point-to-bus service Isolation

Current GPS Navigation Devices  Designed for drivers, not bus riders! oShould I get on this bus? oIs my stop next? oWhat do I do if I miss my stop?  Requires extensive user input oKeying in street addresses no fun!  Device decides which route is “best” oFamiliar route better than shorter one  “Catastrophic failure” when signal is lost

New Approach  User carries GPS cell phone  System infers transportation mode oPosition, velocity, geographic information  Over time, system learns about user oImportant places oCommon transportation plans  Breaks from routine = possible user errors oAsk user if help is needed

GPS reading z k-1 zkzk Edge, velocity, position x k-1 xkxk  k-1 kk Data (edge) association Time k-1 Time k m k-1 mkmk Transportation mode t k-1 tktk Trip segment g k-1 gkgk Goal c k-1 ckck Cognitive mode { routine, novel, error } User Model

Error Detection: Missed Bus Stop

 GPS camera-phone  “Knocks” when there is an opportunity to help oCan I guide you to a likely destination? oI think you made a mistake! oThis place seems important – would you photograph it? Prototype: Opportunity Knocks

Status  User needs study  Algorithms for learning and predicting transportation behavior oBest paper award at AAAI-2004  Proof of concept prototype  Now: user interface studies oModality: Audio, Graphics, Tactile,... oGuidance strategies: Landmarks, User frame of reference, Maps,...

ADL Monitoring from RFID Tag Data UW CSE Intel Research Seattle demo at Intel this afternoon

Object-Based Activity Recognition  Activities of daily living involve the manipulation of many physical objects oKitchen: stove, pans, dishes, … oBathroom: toothbrush, shampoo, towel, … oBedroom: linen, dresser, clock, clothing, …  We can recognize activities from a time- sequence of object touches

Sensing Object Manipulation  RFID: Radio-frequency identification tags oSmall oLong-lived – no batteries oDurable  Easy to deploy  Bracelet touch sensor  Wall-mount movement sensor

Example Data Stream

Example Activity Model

Creating Models of ADLs  Hand-built  Learn from sensor data  Mine from natural-language texts  All of the above...

Experiment: Morning Activities  10 days of data from the morning routine in an experimenter’s home o61 tagged objects  11 activities oOften interleaved and interrupted oMany shared objects Use bathroomMake coffeeSet table Make oatmealMake teaEat breakfast Make eggsUse telephoneClear table Prepare OJTake out trash

DBN with Aggregate Features 88% accuracy 6.5 errors per episode

Improving Robustness  Tracking fails if novel objects are used  Solution: smooth parameters over abstraction hierarchy of object types

Status  Accurate tracking of wide variety ADLs  Active collaboration with Intel  Current work oDetecting user errors in ADL performance oLearning more complex ADLs  Preconditions/effects  Multi-tasking  Temporal constraints oReminding & prompting

Concluding Remarks  Research on Assisted Cognition going great guns at UW and (a few) other universities oCMU / Pitt / U Michigan (Nursebot, Autominder – M. Pollack) oGeorgia Tech (Aware Home, G. Abowd) oMIT (House N, Stephen Intille)

Some Thoughts on Funding  Getting funding for work in this area is currently challenging oWe were fortunate once with NIDRR, but less than 1% of their budget is for research oNIH & NIA spend relatively little on caregiving research  New NIH “Roadmap” for interdisciplinary exploratory research completely leaves out caregiving! oNIN has good people, but no real money

Some Thoughts on Funding  Getting funding for work in this area is currently challenging oNSF supports some of the underlying, multi- use technology, but not medically-oriented applications  Exception: helping disabled use computers oIndustry support is vital, but more for collaboration than actual dollars  Good industry grant = 1 grad student oThere’s a gap waiting to be filled...