2. The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine) Kurt Johnson, Pat Brown, Mark Harniss (UW Rehabilitation Medicine) Matthai Philipose (Intel Research Seattle)

3. Trend 1: Sensing Infrastructure  Robust direct-sensing technology oGPS-enabled phones oRFID tagged products oWearable multi-modal sensors  Rapid commercial deployment

4. Trend 2: Healthcare Crisis  Demand for community integration of the cognitively disabled o100,000 @ year disabled by traumatic brain injury o7.5 million in US with mental retardation o4 million in US with Alzheimer’s  Family burnout  Nationwide shortage of professionals

5. Assisted Cognition  Technology to support independent living by people with cognitive disabilities oat home oat work othroughout the community by oUnderstanding human behavior from sensor data oActively prompting and advising oAlerting human caregivers when necessary

6. Building Partnerships  UW Assisted Cognition seminar oCSE, medicine, nursing, Intel  ACCESS oUW CSE & Rehabilitation Medicine oGrant from NIDDR (Dept. of Education) oHelp cognitively disabled use public transportation oPrototype: Opportunity Knocks  Intel Proactive Health effort oComputing for wellness & caregiving oPromote partnerships with government, universities, healthcare organizations oIntel Seattle: sensors for activity tracking

7. Example  Way-finding Assistant oHelp user travel throughout community  On foot  Using public transportation oDetect user errors  Proactively help user recover  “You missed your stop, so get off at the next stop and then wait for the #16 bus...” oPotential users  TBI, MR, mild memory impairment

8. Example  ADL Assistant oActivities of daily living  Eating, bathing, dressing,...  Cooking, cleaning, emailing,... oMonitoring  Changes in ADLs signal changes in health oReminding / prompting  “Time to take your blue meds” oStep-by-step guidance  “Turn on the tap... now pick up the brush...” oPotential users  Disabled, ordinary aging

9. General Model user model common- sense KB geospatial DB wearables sensors environmental sensors intervention decision making user interface caregiver alerts

10. General Model common- sense KB geospatial DB wearables sensors environmental sensors intervention decision making user interface caregiver alerts physical motion & position cognitive stategoalsactivity

11. Deciding to Intervene A = system intervenes G = user actually needs help

12. ACCESS Way-finding Assistant supported by National Institute on Disability & Rehabilitation Research DARPA IPTO

13. The Need: Community Access for the Cognitively Disabled

14. Problems in Using Public Transportation Learning bus routes and numbers

15. Problems in Using Public Transportation Learning bus routes and numbers Transfers, complex plans

16. Problems in Using Public Transportation Learning bus routes and numbers Transfers, complex plans Recovering from mistakes

17. Result Need for extensive life-coaching Need for point-to-bus service

18. Result Need for extensive life-coaching Need point-to-bus service Isolation

19. Current GPS Navigation Devices  Designed for drivers, not bus riders! oShould I get on this bus? oIs my stop next? oWhat do I do if I miss my stop?  Requires extensive user input oKeying in street addresses no fun!  Device decides which route is “best” oFamiliar route better than shorter one  “Catastrophic failure” when signal is lost

20. New Approach  User carries GPS cell phone  System infers transportation mode oPosition, velocity, geographic information  Over time, system learns about user oImportant places oCommon transportation plans  Breaks from routine = possible user errors oAsk user if help is needed

21. GPS reading z k-1 zkzk Edge, velocity, position x k-1 xkxk  k-1 kk Data (edge) association Time k-1 Time k m k-1 mkmk Transportation mode t k-1 tktk Trip segment g k-1 gkgk Goal c k-1 ckck Cognitive mode { routine, novel, error } User Model

22. Error Detection: Missed Bus Stop

23.  GPS camera-phone  “Knocks” when there is an opportunity to help oCan I guide you to a likely destination? oI think you made a mistake! oThis place seems important – would you photograph it? Prototype: Opportunity Knocks

24. Status  User needs study  Algorithms for learning and predicting transportation behavior oBest paper award at AAAI-2004  Proof of concept prototype  Now: user interface studies oModality: Audio, Graphics, Tactile,... oGuidance strategies: Landmarks, User frame of reference, Maps,...

25. ADL Monitoring from RFID Tag Data UW CSE Intel Research Seattle demo at Intel this afternoon

26. Object-Based Activity Recognition  Activities of daily living involve the manipulation of many physical objects oKitchen: stove, pans, dishes, … oBathroom: toothbrush, shampoo, towel, … oBedroom: linen, dresser, clock, clothing, …  We can recognize activities from a time- sequence of object touches

27. Sensing Object Manipulation  RFID: Radio-frequency identification tags oSmall oLong-lived – no batteries oDurable  Easy to deploy  Bracelet touch sensor  Wall-mount movement sensor

28. Example Data Stream

29. Example Activity Model

30. Creating Models of ADLs  Hand-built  Learn from sensor data  Mine from natural-language texts  All of the above...

31. Experiment: Morning Activities  10 days of data from the morning routine in an experimenter’s home o61 tagged objects  11 activities oOften interleaved and interrupted oMany shared objects Use bathroomMake coffeeSet table Make oatmealMake teaEat breakfast Make eggsUse telephoneClear table Prepare OJTake out trash

32. DBN with Aggregate Features 88% accuracy 6.5 errors per episode

33. Improving Robustness  Tracking fails if novel objects are used  Solution: smooth parameters over abstraction hierarchy of object types

34. Status  Accurate tracking of wide variety ADLs  Active collaboration with Intel  Current work oDetecting user errors in ADL performance oLearning more complex ADLs  Preconditions/effects  Multi-tasking  Temporal constraints oReminding & prompting

35. Concluding Remarks  Research on Assisted Cognition going great guns at UW and (a few) other universities oCMU / Pitt / U Michigan (Nursebot, Autominder – M. Pollack) oGeorgia Tech (Aware Home, G. Abowd) oMIT (House N, Stephen Intille)

36. Some Thoughts on Funding  Getting funding for work in this area is currently challenging oWe were fortunate once with NIDRR, but less than 1% of their budget is for research oNIH & NIA spend relatively little on caregiving research  New NIH “Roadmap” for interdisciplinary exploratory research completely leaves out caregiving! oNIN has good people, but no real money

37. Some Thoughts on Funding  Getting funding for work in this area is currently challenging oNSF supports some of the underlying, multi- use technology, but not medically-oriented applications  Exception: helping disabled use computers oIndustry support is vital, but more for collaboration than actual dollars  Good industry grant = 1 grad student oThere’s a gap waiting to be filled...