의미 모델링 Elaborating Sensor Data using Temporal and Spatial Commonsense Reasoning + Mining Models of Human Activities from the Web 지능 기반 시스템 응용 민준기

2 Agenda  B. Morgan and P. Singh, “Elaborating Sensor Data using Temporal and Spatial Commonsense Reasoning,” BSN The Problem Space LifeNet : A First-Person Model The Plug Sensor Network  M. Perkowitz, et al., “Mining Models of Human Activities from the Web,” WWW Introduction Proposed Technique Evaluation Summary and Future Work

3 The Problem Space  Two distinct directions for research Human-out (This paper) Telephone Technology-in (Much sensor network research) Text messaging on cell phones  Three topics LifeNet probabilistic human model The Plug sensor network An experimental design for evaluation of the LifeNet learning method

4 LifeNet : A First-Person Model  First-person common-sense inference model OpenMind Common Sense, ConceptNet, The PlaceLab data, Honda’s indoor common sense data  Attempts to anticipate and predict what humans do in the world  All of the reasoning in LifeNet is based on probabilistic propositional logic “I am washing my hair” before “my hair is clean”

5 The Plug Sensor Network  Using for both learning common sense and for recognizing and predicting human behavior  Using this sensor network to monitor how individuals interact with their physical environment  Nine sensor modalities: sound, vibration, brightness, current, wall voltage, acceleration

6 Agenda  B. Morgan and P. Singh, “Elaborating Sensor Data using Temporal and Spatial Commonsense Reasoning,” BSN The Problem Space LifeNet : A First-Person Model The Plug Sensor Network  M. Perkowitz, et al., “Mining Models of Human Activities from the Web,” WWW Introduction Proposed Technique Evaluation Summary and Future Work

7 Introduction : Recognize Humans Activities  Applications include activity-based actuation Dimming lights when a video is being watched Providing directions for someone using unfamiliar facilities etc.  Ubiquitous, proactive, disappearing computing Computers have to understand people’s needs by observing their physical activities (and to act autonomously) The cost of developing recognition infrastructure is too high Even small classes of activities is hard to recognize  A broadly applicable system should be general-purpose and easy to use

8 Motivation  Vision based systems None have reported detecting more than tens of activities in practice The features robustly detectable from vision are coarse Represent the relationships between “blobs” in the image rather than specific objects Each activity is expensive to model  Learning of the models The developers define the structure of the possible models System tunes the parameters of the model based on examples from the user The user is expected to label the patterns The variety of activities is quite restricted Introduction

9 Proposed Technique  RFID (Radio Frequency Identification) Cheap: Postage-stamp sized, forty-cent Wireless and battery free  Activity modeling Define an activity in terms of the probability and sequence of the objects Generate the models by translating textual definitions Structured like recipes Produced automatically by mining appropriate web sites Mining models is part of a larger activity recognition system, PROACT (Proactive Activity Toolkit)

10 Usage Model  Assumes that interesting objects in the environment contain RFID tags (tens ~ hundreds) Making a database entry mapping the tag ID to a name Within a few years, many household objects may be RFID- tagged before purchase, thus eliminating the overhead of tagging Medium-range readers (Tag-detecting Gloves) and Long-range readers (Run robots, Carts, …)  PROACT uses the sequence and timing of object to deduce what activity is happening Likelihood of various activities, details of those activities, degree of certainty, etc… Proposed Technique

11 System Overview  PROACT provides an activity viewer for debugging Real-time view of activities in progress The sensor data seen Changing of belief in each activity with the data  Inference Engine converts the activity models produced by the mining engine into Dynamic Bayesian Networks D. Patterson, L. Liao, D. Fox, H. Kautz, “Inferring High-Level Behavior from Low-Level Sensors,” Ubicomp Proposed Technique

12 Sensors and Models  Sensors Use two different kinds of RFID readers Long-range reader (mobile robot): map the location of objects Short-range reader (glove): determine the objects that are touched  Models Each model (activity) is composed of a sequence (step) s1 ~sn Each step si has optional duration ti and object oij involved along with the probability pij Proposed Technique

13 The Model Extractor  Builds formal models of activities using directions  Directions are written in natural language by human How-to (ehow.com), recipes (epicurious.com), training manuals, protocols, etc.  Syntactic structure of directions 1.A title t for the activity 2.A textual list r1~rm, Each step ri has: Possibly a special keyword delimiting duration di What to do during the step: subset of the objects and duration Proposed Technique

14 Converting Directions to Activity Models  Key steps 1.Labeling Set label of the mined model to title of the directions 2.Parsing steps Duration: Gaussian with mean = d, stdev = S(d, i, l ) Object Oi and Probability P 3.Tagged object filtering  Functions Object Object extraction: WordNet ontology Noun-phrase extraction: QTag tagger Probability Fixed probabilities Google conditional probabilities (GCP) Proposed Technique For example, [“making tea”] has 24,200 matches, and [“making tea” cup] has 7,340 matches, then conditional probability of a cup being involved in making tea is 7340/2400 = 0.3

15 Example Proposed Technique

16 Evaluation  Mined models ehow.com: 2300 directions ffts.com: 400 recipes epicurious.com: 18,600 recipes  Three strategies to approximate comprehensive evaluation Human activity-trace recognition Activities of Daily Living (ADLs) Inter-corpus consistency Making cookies recipes Intra-corpus distinguish-ability Distinguish-ability within activity domains

17 Distinguish-ability

18 Human and inter-corpus trace recognition  ADLs domain Many objects were not tagged, missed, and interleaved Models were not perfect  Cookie domain The identical recipe can have quite different structure For some of the recipes, there is no counterpart in the other corpus Evaluation

19 Impact of techniques on accuracy  ADLs Domain is fairly sparse, with many activities involving only few object  Cookie domain Each activity model involves many more objects Evaluation

20 Impact of techniques on compactness Evaluation

21 Summary and Future Work  An introduction to the idea of mining activity detection from the web  Future work Perform a more comprehensive evaluation Improving the effectiveness of mined models Include location Synonymous words Synsets (collections of synonymous words) can be extracted from WordNet