Building an Aware Home: Understanding the symbiosis between computing and everyday activities Irfan Essa, Gregory Abowd Future Computing Environments College of Computing, Georgia Institute of Technology

© Irfan Essa and Georgia Institute of Technology, 1999 Who are we? 4 Faculty (Future Computing Env.): –Gregory Abowd, Chris Atkeson, Aaron Bobick, Irfan Essa, Blair MacIntyre, Beth Mynatt, Thad Starner 4 ~20 PhD students 4 Affiliations: CoC, GVU, BTC, ECE 4 Collaborations –ECE (Wireless, DSP), Architecture, Rehab Technologies, Psychology, etc.

© Irfan Essa and Georgia Institute of Technology, 1999 Outline 4 Motivation “Living Laboratory” 4 Focus Areas / Research Questions 4 Awareness 4 Context-Aware Applications 4 “Aging in place” 4 Conclusion / Discussion

© Irfan Essa and Georgia Institute of Technology, 1999 Research Objectives 4 Build a living laboratory in an everyday setting that is aware of its occupants’ activities 4.. supports the continuous interactions and activities of a small community. 4.. understand the usage of such a laboratory as it applies to symbiosis of computing and everyday living.

© Irfan Essa and Georgia Institute of Technology, 1999 Living Laboratory for Human-Home Symbiosis 4 How can we obtain ubiquitous and continuous connection? –testbed for technologies 4 How does this change life / living? –information at fingertips –support human communication –build community –unite over distance tech-centric user-centric

© Irfan Essa and Georgia Institute of Technology, 1999 An aware home 4 A home that is aware of its inhabitants and their activities 4 … can provide support for day-to-day activities 4 … do so without increasing the load on the inhabitants 4 … can augment daily functions 4 … provide connectivity 4 In and around the home

© Irfan Essa and Georgia Institute of Technology, 1999 Specific Applications 4 awareness and connectedness w/ others 4 augmentation (cognitive, memory) 4 education 4 monitoring 4 security / surveillance 4 Care Facility (Elder, Child, Health, …)

© Irfan Essa and Georgia Institute of Technology, 1999 Where are we? 4 Georgia Research Alliance (GRA) –~$600,000 4 Broadband Telecommunications Center (BTC) 4 Ground breaking May Occupancy by Jan. 2000

© Irfan Essa and Georgia Institute of Technology, 1999 SouthEast Outside

© Irfan Essa and Georgia Institute of Technology, 1999 Basement

© Irfan Essa and Georgia Institute of Technology, 1999 Living floors (2 floors)

© Irfan Essa and Georgia Institute of Technology, 1999 Other “smart homes” 4 Home automation –X10, –hobby 4 Many others 4 -info/smart_homes.html (Brad Stenger) 4 MSFT Research (Barry Brummit)

© Irfan Essa and Georgia Institute of Technology, 1999 Outline 4 Motivation “Living Laboratory” 4 Focus Areas / Research Questions 4 Awareness 4 Context-Aware Applications 4 “Aging in Place” 4 Conclusions / Discussion

© Irfan Essa and Georgia Institute of Technology, 1999 Research themes Human-Home Symbiosis Human-centered Technology-centered Aware Home

© Irfan Essa and Georgia Institute of Technology, 1999 Technological Challenges - I 4 Perception Technologies –make the environment aware of the users and their activities –what is happening? –ubiquitous sensing –interpret (rich) multi-modal streams –long-term vs. short term

© Irfan Essa and Georgia Institute of Technology, 1999 Technological Challenges - II 4 Ubiquitous Interfaces / Displays –“Off-the-desktop” –Context-aware applications –capture / integrate / access –anytime, anywhere, ease of use, … –diverse resources/media –software infrastructure –multimedia-based collaboration/interaction

© Irfan Essa and Georgia Institute of Technology, 1999 Technological Challenges - III 4 Systems & Networking –fast –distributed –secure –adaptive –storage –easy to deploy / configure wireless/wireline –inside, around, and to the home

© Irfan Essa and Georgia Institute of Technology, 1999 User-centric Challenges - I 4 Understand the needs of the domain –physical house vs. home (familial connections) awareness/connectedness with others –privacy / security –decrease cognitive load –What home activities are desirable can be improved through technology –Care facility (elderly, young, health)

© Irfan Essa and Georgia Institute of Technology, 1999 User-centric Challenges - II 4 Elderly home care / assistive healthcare –prolong independence in familiar surroundings –understand rhythms, patterns, deviations –provide contact –memory augmentation

© Irfan Essa and Georgia Institute of Technology, 1999 Outline 4 Motivation “Living Laboratory” 4 Focus Areas / Research Questions 4 Awareness 4 Context-Aware Applications 4 “Aging in Place” 4 Conclusions / Discussion

© Irfan Essa and Georgia Institute of Technology, 1999 Aware Spaces 4 Aware environments that know their inhabitants, their preferences, their activities –Who is there? –Where? –What is happening? –How it can best be supported?

© Irfan Essa and Georgia Institute of Technology, 1999 Computational Perception 4 Signal Interpretation/Coding –Computer Vision, –Audio/Speech, –Tactile / Contact, –RF/IR emitters, –Sonar, –Usage Sensor, …… 4 Instrument a Space with Sensors

© Irfan Essa and Georgia Institute of Technology, 1999 Perceptual Analysis 4 Signal Interpretation to determine –geometry, calibration, context –is anyone there?, who? –locate users/people –recognize their actions, activity, gestures, expressions –speech, non-verbal, communicative streams 4 Dynamic / Long-term / Interactive

© Irfan Essa and Georgia Institute of Technology, 1999 Sensors (Optical / Cameras) 4 High-end vs. low-end 4 Task / Resource specific 4 NEX V25 microprocessor, powerline modem,... 4 Analog / Digital Cameras to commercial PCs 4 Specific hardware solutions

© Irfan Essa and Georgia Institute of Technology, 1999 Experiences 4 Reconstruction of a Scene 4 Pose Estimation 4 Multiple Camera-Multiple Person Tracking 4 Context-based Activity / Object Recognition

© Irfan Essa and Georgia Institute of Technology, D models of rooms (Brostow & Essa, ICCV 1999) 4 Use motion information to model 3D scenes (models from movement).

© Irfan Essa and Georgia Institute of Technology, 1999 Video

© Irfan Essa and Georgia Institute of Technology, 1999 Multiple Cameras (Stillman, Essa, et al., AVBPA 1999) 4 Track multiple people with multiple cameras 4 Develop an architecture to support communication between multiple processors/cameras 4 Combine fixed and PTZ cameras to track and identify people

© Irfan Essa and Georgia Institute of Technology, 1999 Video

© Irfan Essa and Georgia Institute of Technology, 1999 System Architecture Video Locations Camera 1 (Fixed) Camera 2 (Fixed) Color Tracking Color Tracking Motion Tracking Motion Tracking Calibrated Video Camera 3 (PTZ) Camera 4 (PTZ) Color Tracking Face Recog. Face Tracking Expression Gesture Color Tracking Face Tracking PTZ locations Video More Cameras Server

© Irfan Essa and Georgia Institute of Technology, 1999 Pose tracking (Schödl, Essa, PUI 1998, PDPTA 1999) 4 Use a 3D model of head 4 Extract texture 4 Match texture on model to moving head (with non-linear optimization) 4 Repeat for every frame 4 Develop distributed/parallelized implementation

© Irfan Essa and Georgia Institute of Technology, 1999 Video

© Irfan Essa and Georgia Institute of Technology, 1999 Distributed Tracking Generate Test Parameters Estimate New Minimum Render Head Model Calculate Matching Error & Gradient Camera Compute Image Pyramid Compute Image Pyramid Render Head Model Calculate Matching Error & Gradient xtxt x t+1 Console Node 1 Node n 4X Time t Console + Parallel Nodes, n=7 in our tests

© Irfan Essa and Georgia Institute of Technology, 1999 Speed-up Curves Relative Frame Size Frame rate as a function of Cluster nodes and the size of image (1x = real frame size, 14.4KB).

© Irfan Essa and Georgia Institute of Technology, 1999 Example: Recognizing Activity (Moore, Essa, Hayes, AVBPA 1999 and ICCV 1999) 4 Develop an framework (architecture) for relating actions and objects 4 Track the relations between actions and objects for recognition 4 Use HMMs for temporal recognition

© Irfan Essa and Georgia Institute of Technology, 1999 Video

© Irfan Essa and Georgia Institute of Technology, 1999 Recognition Results

© Irfan Essa and Georgia Institute of Technology, 1999 Other Projects (not there yet!) 4 Smart Carpet –Recognizes people based on their footsteps 4 Audio-visual tracking –analysis of audio & visual-kinetic data –audio-visual tracking 4 Auto calibration (inside/outside) 4 Other sensors (wearable etc.)

© Irfan Essa and Georgia Institute of Technology, 1999 Outline 4 Motivation “Living Laboratory” 4 Focus Areas / Research Questions 4 Awareness 4 Context-Aware Applications 4 “Aging in Place” 4 Discussion / Future

© Irfan Essa and Georgia Institute of Technology, 1999 What is context? 4 Characterizing a situation 4 Sensed information 4 Identity, location, activity of people, places, things 4 Who? Where? When? What? Why?

© Irfan Essa and Georgia Institute of Technology, 1999 Context-aware applications 4 Present context information to users –Example: fridge informs the user of what is running out 4 Tailor the interaction according to context changes –Examples: Activity in kitchen and near dinner time  provide recipe help based on available food and preferences

© Irfan Essa and Georgia Institute of Technology, 1999 Easier said than done! 4 Designing and implementing such context-aware applications is difficult! 4 Goal: Provide software infrastructure to support rapid development

© Irfan Essa and Georgia Institute of Technology, 1999 The Context Toolkit (Salber, Dey, & Abowd CHI 1999) 4 Separation of concerns –context sensing from reaction –insulate sensors and applications from each other 4 An analogy to GUI development –separation of presentation and functionality 4 We want glue between perception and interaction

© Irfan Essa and Georgia Institute of Technology, 1999 Beyond the GUI analogy 4 Context widgets are distributed –They can be shared by applications 4 Context widgets are persistent –They store a history of context information 4 Context widgets may be unreliable –They must provide confidence factors

© Irfan Essa and Georgia Institute of Technology, 1999 Components 4 Context widgets –abstraction of a sensor –taxonomy of context types 4 Interpreters –translation between context values 4 Entity servers –persistence and aggregation of context

© Irfan Essa and Georgia Institute of Technology, 1999 Experience 4 Electronic In/Out Board 4 Informal capturing whiteboard 4 Mobile Conference Assistance 4 Home Monitoring system 4 More empirical experience needed

© Irfan Essa and Georgia Institute of Technology, 1999 Outline 4 Motivation “Living Laboratory” 4 Focus Areas / Research Questions 4 Awareness 4 Context-aware Applications 4 “Aging in Place” 4 Conclusions & Discussion

© Irfan Essa and Georgia Institute of Technology, 1999 “Aging in Place” 4 Design aware homes that support elderly –allow them to be independent, yet connected –supported, cared for –stay home (as opposed to move to an elder care facility) –health monitoring

© Irfan Essa and Georgia Institute of Technology, 1999 Connected Family 4 Is Mom doing well? Eating well? (peace of mind) 4 interface that leads to connectivity 4 see snapshots of “activities”, “day’s events” 4 active connection (in the periphery) 4 continuously updating “portrait” of Mom displaying how she is doing.

© Irfan Essa and Georgia Institute of Technology, 1999 Cognitive Support 4 Assist in daily routines to offset cognitive impairments 4 Aid memory –take medication –locate lost items –out of site / out of mind (connected) 4 Avoid institutionalization effects

© Irfan Essa and Georgia Institute of Technology, 1999 Requirements Analysis 4 What “matters” in free choice envs. 4 What is productivity? Quantify ??!! 4 Why do people move to assistive care facilities? 4 Why don’t they want to leave their homes 4 Ethnographic Interviews ……

© Irfan Essa and Georgia Institute of Technology, 1999 Outline 4 Motivation “Living Laboratory” 4 Focus Areas / Research Questions 4 Awareness 4 Context & Domains 4 “Aging in Place” 4 Conclusions & Discussions

© Irfan Essa and Georgia Institute of Technology, 1999 Test-beds 4 Future Computing Lab (5/1998) 4 Computational Perception Lab (1/98) 4 New Labs for “off-the-desktop” computing (7/1999) 4 “Aware Home” (1/2000) –Kitchen, Living Room, Entertainment Room, Home-office.

© Irfan Essa and Georgia Institute of Technology, 1999 Future 4 Pursue both technology-centric and human-centric approaches, understand the domain and build it –better sensing and perceptual analysis mechanisms –software, systems, networking infrastructure –evaluate the human-home symbiosis

© Irfan Essa and Georgia Institute of Technology, 1999 Summary 4 Described the Aware Home Project 4 Making it aware –sensing –context-enabled 4 Challenging Application –Care Facility.

© Irfan Essa and Georgia Institute of Technology, 1999 The end

© Irfan Essa and Georgia Institute of Technology, 1999 Why a living laboratory? 4 It is not sufficient to achieve technological breakthroughs. 4 Greater contribution lies in the understanding of impact on everyday life. 4 Domain specific.

© Irfan Essa and Georgia Institute of Technology, 1999 Living laboratory experience 4 Classroom 2000 (education) –a classroom used daily for 3 years –captured experiences in a classroom 4 Smart Spaces –Rooms, Offices, … 4 A large-scale test-bed for research in …