2. Introduction Time is very essential component in everyday life and would act as great source of information for any smart home. We hypothesize that anomaly detection and enhanced prediction is possible using temporal relations. TempAl suite:  Identify temporal relations in smart home datasets.  Use temporal information to aid prediction of events in a smart home environment.  Use temporal information to identify anomalous events in a smart home environment.

3. Smart Homes: Goals Adapt to Needs Cost Effective and Reliable Maximum Comfort and Security 3VJ AI@WSU © 2007

4. MavHome: Smart Home Project Project Unique – Focus on entire home House perceives and acts – Sensors – Controllers for devices – Connections to the mobile user and Internet Unified project incorporating varied AI techniques, cross disciplinary with mobile computing, databases, multimedia, and others VJ AI@WSU © 20074

5. Experimentation Environment 2 MavLAB Argus Sensor Network  around 100 Sensors.  include Motion, Devices, Light, Pressure, Humidity and more. Real Dataset and synthetic datasets consist timestamp of the activity with the activity name and the state it is in. 5VJ AI@WSU © 2007 Datasets Parameter Setting No of Days No of Events No of Intervals Identified Size of Data Synthetic6081729106KB Real60171623104KB

6. What is a temporal relation? Food “Contains” Water or Water “Before” Pills or Food “Meets” Pills or Food “Contains” Water “before” Pills “It is common to describe scenarios using time intervals rather than time points” - James F. Allen 6VJ AI@WSU © 2007 Reminder system based on temporal relations. Reminder Assistance If Pills are to be taken “After” Food, we can notice violation of this activity! Anomaly Detection If Cooker is Spoiled should we call emergency or a normal repair? Maintenance If Oven used for Turkey, Is turkey at Home? Temporary Need Analysis Increase prediction accuracy with association rules! Improve Prediction

7. Allen’s 13 Temporal Relations Before After During Contains Overlaps Overlapped-By Meets Met-by Starts Started-By Finishes Finished-By Equals 7VJ AI@WSU © 2007

8. Experimentation Process Experiment 1: Evaluate TempAl’s ability to detect anomalies using temporal relations. Experiment 2: Evaluate TempAl’s event prediction using association rules. Experiment 3: Evaluate TempAl’s event prediction using temporal relations in ALZ.

9. Architecture

10. Temporal Intervals Process raw data to form temporal intervals. Raw Sensor Data Timestamp Sensor State Sensor ID 3/3/2003 11:18:00 AM OFF E16 3/3/2003 11:23:00 AM ON G12 3/3/2003 11:23:00 AM ON G11 3/3/2003 11:24:00 AM OFF G12 Identify Time Intervals Date Sensor ID Start Time End time. 03/02/2003 G11 01:44:00 01:48:00 03/02/2003 G19 02:57:00 01:48:00 03/02/2003 G13 04:06:00 01:48:00 03/02/2003 G19 04:43:00 01:48:00 Associated Temporal Relations Date time Sensor ID Temporal Relation Sensor ID 3/3/2003 12:00:00 AM G12 DURING E16 3/3/2003 12:00:00 AM E16 BEFORE I14 3/2/2003 12:00:00 AM G11 FINISHESBY G11 4/2/2003 12:00:00 AM J10 STARTSBY J12 Raw Sensor DataInterval Data Temporal Relations Data

11. Experiment 1: Temporal relations based anomaly detection Step 1: Collect the temporal relations based data. Step 2: Identify the most frequent happening events in the data. Step 3: Use the frequency of temporal relations to calculate the probability of the observed event. Step 4: Flag observed events as anomalous which have a sufficiently low probability.

12. Find Frequent Events: The Apriori Algorithm Example Database D Scan D C1C1 L1L1 L2L2 C2C2 C2C2 C3C3 L3L3 12VJ AI@WSU © 2007

13. Anomaly Detection Process 1] Calculate the evidence of occurrence with other frequent events P(Z|Y) = |Before(Y,Z)| + |Contains(Y,Z)| + |Overlaps(Y,Z)| + |Meets(Y,Z)| + |Starts(Y,Z)| + |StartedBy(Y,Z)| + |Finishes(Y,Z)| + |FinishedBy(Y,Z)| + |Equals(Y,Z)| / |Y| Event Sequence: X A B P(B|A U X) = P(B ∩ (AUX) ) / P(AUX) = P(B ∩ A) U P(B ∩ X)/ P(A) + P(X) –P(A ∩ X) [Distributive Rule] = P(B|A)*P(A) + P(B|X)*P(X) / P(A) + P(X) –P(A ∩ X) [Multiplication Rule] 2] Calculate the anomaly. Anomaly z = 1 – P(Z|Y) 3] Check if the calculated anomaly is equal to or greater than (mean + 2 * St. Dev.), If yes, declare it as an Anomaly. VJ © AI LAB EECS@WSU 2007

14. Experimentation Results Frequent EventEvidenceAnomalyDetected J100.450.55No J110.320.68No A110.330.67No A150.240.76No A110.230.77No A150.220.78No I110.270.73No I140.340.66No Anomaly Mean0.7 Anomaly St. Dev.0.071764 Anomaly Cut-off Threshold0.8435 Anomaly Detection on Real Dataset Frequent EventEvidenceAnomalyDetected Lamp0.30.7NO Lamp0.230.77NO Lamp0.010.99YES Fan0.320.68NO Cooker0.290.71NO Lamp0.450.55NO Lamp0.230.77NO Lamp0.010.99YES Lamp0.230.77NO Fan0.30.7NO Cooker0.340.66NO Lamp0.330.67NO Lamp0.20.8NO Lamp0.020.98NO Lamp0.0020.998YES Fan0.340.66NO Cooker0.420.58NO Anomaly Mean0.763412 Anomaly St. Dev.0.135626 Anomaly Cut-off Threshold1 Anomaly Detection on Synthetic Dataset VJ © AI LAB EECS@WSU 2007

15. Experiment 2: Using Temporal relations for identifying association rules for prediction. Step 1: Collect the temporal relations based data. Step 2: Use Weka workbench to generate association rules on the temporal relations data. Step 3: Rules are used in the form “IF X THEN Y”. Use rules with the existing system for prediction.

16. Association rule generation using Weka Use association rule mining to find predictive rules in Weka workbench. Weka generates best rules with a given support and confidence. We use Apriori algorithm and vary the support and confidence and collect the output.

17. Parameter settings for rules generation using Weka in Real Datasets R UN M INIMUM SUPPORT MINIMUM CONFIDENCE N O OF B EST R ULES F OUND 10.000.5100 20.010.5006 30.020.5002 40.050.5001

18. Parameter settings for rules generation using Weka in Synthetic Datasets R UN M INIMUM SUPPORT MINIMUM CONFIDENCE N O OF B EST R ULES F OUND 10.000.5100 20.010.5010 30.020.5005 40.050.5003

19. Rule Generation Due to small datasets used, we use the top rules generated with a minimum confidence of 0.5 and a minimum support of 0.01. Confidence level above 0.5 and support above 0.05 could not be used, as they could not result in any viable rules. Sample of best rules observed in real datasets: Activity=C11 Relation=CONTAINS 36 ==> Activity=A14 36 Activity=D15 Relation=FINISHES 32 ==> Activity=D9 32 Activity=D15 Relation=FINISHESBY 32 ==> Activity=D9 32 Activity=C14 Relation=DURING 18 ==> Activity=B9 18

20. Step 3: Temporal Rules Enhancement to the Prediction. Input: Output of ALZ Predictor a, Best Rules r, Temporal Dataset Repeat If a! = null Repeat Set r1 to the first event in the relation rule If (r[i].relationoccur ==a) Then Read r[i].relationpredict, if any Then predict; If (a == testevent ) Then increment correctcount, Then insert a in trie; Else Continue; End if. Until end of rules. End if. Loop until End of Input.

21. Experiment 2 Results Comparing ActiveLeZi based prediction with and without temporal rules D ATA S ET A CCURACY %E RROR % R EAL (W ITHOUT R ULES ) 5545 S YNTHETIC (W ITHOUT R ULES ) 6436 R EAL (W ITH R ULES ) 5644 S YNTHETIC (W ITH R ULES ) 6931

22. Experiment 3: Using Temporal relations for prediction using Alz. Step 1: Collect the temporal relations based data. Step 2: Use existing ActiveLeZi Predictor and also calculate the temporal probability. Step 3: Use Prediction by Partial Match employed by ALZ with temporal probability for prediction.

23. ALZ Uses LZ78 compression algorithm and it uses Prediction by Partial Match (PPM) family of predictors (Alz-TDAG) Acts as sequential predictor and predicts the next most probable event based on trie. The way it calculates probability is that a is next event is

24. Why use temporal probability? At higher order in phrase we see that temporal probability would include more information compared to the sequential probability. Prediction c = P (C|B) = P (C|B) SEQ: Order-0 + P (C|B) TEMPORAL: Order 1-n (at each order in phrase) + P (C|B) SEQ: n P(B|A) = |After(B,A)| + |During(B,A)| + |OverlappedBy(B,A)| + |MetBy(B,A)| + |Starts(B,A)| + |StartedBy(B,A)| + |Finishes(B,A)| + |FinishedBy(B,A)| + |Equals(B,A)| / |A|

25. Prediction On Real Datasets Dataset (Learning Algorithm)TrainTestCorrect Prediction Accuracy (%) Prediction Error (%) Real (Alz)100100%100% Real (Alz+Tempal)10011100%0% Real (Alz)10010660%40% Real (Alz+Tempal)10010660%40% Real (Alz)750402972.50%27.50% Real (Alz+Tempal)750402972.50%27.50% Cross Validation (Alz)787834857.96%42.04% Cross Validation (Alz+Tempal)787834958.92%41.08%

26. Prediction on Synthetic Datasets Dataset(Learning Algorithm)TrainTestCorrect Prediction Accuracy Prediction Error Synthetic (Alz)10011100%0% Synthetic (Alz+Tempal)10011100%0% Synthetic (Alz)10010 100%0% Synthetic (Alz+Tempal)10010 100%0% Synthetic (Alz)1400908998.88%1.12% Synthetic (Alz+Tempal)140090 100%0% Synthetic (Alz)139051544153299.22%0.78% Synthetic (Alz+Tempal)139051544153299.22%0.78% Cross Validation (Alz)139051544129283.68%16.32% Cross Validation (Alz+Tempal)139051544129283.64%16.36%

27. Test Case Scenario To highlight the true potential of leveraging temporal relations for enhancing prediction. Test set consists of two typical test events which were also check by observation. one with more number of predictive temporal relations and other with less.

28. Test Case Scenario Results Alz mispredicts both the test events and TempAl predicts one correctly and mispredicts other due to lack of more temporal information in the form of temporal relations. Training Set: a ON a OFF a ON b ON a ON b ON a ON b ON c ON d ON d OFF c ON b ON a ON c OFF a OFF Test Set: a ON d OFF Temporal Relations on Training Set: a BEFORE a, a BEFORE b, a BEFORE b, a BEFORE b, a BEFORE a, a BEFORE b, a BEFORE c, a BEFORE d, a BEFORE c, a BEFORE a, a AFTER a, a OVERLAPS b, a BEFORE b, a BEFORE b, a BEFORE a, a BEFORE b, a BEFORE c, a BEFORE d, a BEFORE c, a BEFORE a, b AFTER a, b OVERLAPPEDBY a, b MEETS b, b BEFORE b, b BEFORE a, b BEFORE b, b BEFORE c, b BEFORE d, b BEFORE c, b BEFORE a, b AFTER a, b AFTER a, b AFTER b, b METBY b, b BEFORE a, b BEFORE b, b BEFORE c, b BEFORE d, b BEFORE c, b BEFORE a, a AFTER a, a AFTER a, a AFTER b, a AFTER b, a AFTER b, a BEFORE b, a BEFORE c, a BEFORE d, a BEFORE c, a BEFORE a, b AFTER a, b AFTER a, b AFTER b, b AFTER b, b AFTER b, b AFTER a, b CONTAINS c, b CONTAINS d, b OVERLAPS c, b BEFORE a, c AFTER a, c AFTER a, c AFTER b, c AFTER b, c AFTER b, c AFTER a, c DURING b, c BEFORE d, c BEFORE c, c BEFORE a, d AFTER a, d AFTER a, d AFTER b, d AFTER b, d AFTER b, d AFTER a, d DURING b, d AFTER c, d BEFORE c, d BEFORE a, c AFTER a, c AFTER a, c AFTER b, c AFTER b, c AFTER b, c AFTER a, c DURING b, c AFTER c, c AFTER d, c FINISHES a, a AFTER a, a AFTER a, a AFTER b, a AFTER b, a AFTER b, a AFTER a, a AFTER b, a AFTER c, a AFTER d, a FINISHESBY c ALZ Prediction: b b Alz +TempAl Prediction: a b

29. Conclusion Unique and new Approach by leveraging temporal information. It shows us that the temporal information aids the prediction and anomaly detection processes in a smart environment. Anomaly detection enables decision maker to identify change pattern in activities based on anomaly or simply discard the event. Also used for reminder assistance. Prediction can be used to predict the next event or next set of events (Modeled with spatial). It acts as a guiding foundation for spatio-temporal models for smart environments.

30. Future Direction Expansion of the temporal relations such as until, since, next, and so forth. Incorporate different dimensions of ON and OFF states. Larger real datasets should be collected and be experimented. Planning problems for smart home activities? Using time window or coming up with a cut off or threshold criterion mechanism. Expands to become a potential Graph mining problem and could involves link prediction. Can be extended to form Spatio-Temporal models. Temporal Relations Visualization problem. Pattern analysis for lifestyle and behavior improvements & feedback on interestingness of a pattern.

31. Acknowledgements I thank my advisor and all my professors & teachers whom I interacted with! I thank my family. I thank all my friends and peers in the research labs which I was a part of. I would like to thank Washington State University and University of Texas at Arlington. I would like to thank all other people who directly or indirectly played a role in encouraging me to work towards the completion of my Thesis.

32. Publications 2007 Vikramaditya R. Jakkula, Aaron S. Crandall, and Diane J. Cook, "Knowledge Discovery in Entity Based Smart Environment Resident Data Using Temporal Relations Based Data Mining", ICDM Workshop on Spatial and Spatio-Temporal Data Mining, Omaha, Nebraska, 2007 (acceptance rate: 20%). Vikramaditya R. Jakkula, "Predictive Data Mining to Learn Health Vitals of Residents in a Smart Home", ICDM IEEE Workshop of Data Mining in Medicine, Omaha, Nebraska, 2007. Vikramaditya R. Jakkula, and Diane J. Cook, "Mining Sensor Data in Smart Environment for Temporal Activity Prediction", Poster session of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Workshop on Knowledge Discovery from Sensor Data (sensor-KDD 2007), San Jose, August 2007 (acceptance rate: 32%). Vikramaditya R. Jakkula, and Diane J. Cook, "Using Temporal Relations in Smart Home Data for Activity Prediction", International Conference on Machine Learning (ICML) Workshop on the Induction of Process Models (IPM /ICML 2007), Corvallis, June 2007 Vikramaditya R. Jakkula, Diane J. Cook, and Aaron S. Crandall, "Temporal pattern discovery for anomaly detection in smart homes", Proceedings of the 3rd IET International Conference on Intelligent Environments (IE 07), Germany, September 2007 Vikramaditya R. Jakkula, and Diane J. Cook, "Learning Temporal Relations in Smart Home Data "Proceedings of the Second International Conference on Technology and Aging, Canada, June 2007 Vikramaditya R. Jakkula, Diane J. Cook, and Gaurav Jain, Prediction Models for a Smart Home Based Health Care System, Proceedings of 21st IEEE International Conference on Advanced Information Networking and Applications Workshops, Canada, May 2007 2006 Vikramaditya R. Jakkula, Michael G. Youngblood and Diane J. Cook, Identification of Lifestyle Behavior Patterns with Prediction of the Happiness of an Inhabitant in a Smart Home, AAAI Workshop on Computational Aesthetics: Artificial Intelligence Approaches to Beauty and Happiness, Boston, July 2006 Gaurav Jain, Diane J. Cook and Vikramaditya R. Jakkula, Monitoring Health by Detecting Drifts and Outliners for a Smart Environment Inhabitant, 4th International Conference on Smart Homes and Health Telematics, June 2006. JOURNALS (PENDING) Vikramaditya Jakkula and Diane J. Cook, “Anomaly Detection Using Temporal Data Mining in a Smart Home Environment ”, Medicine and Medical Informatics journal special issue on Smart Homes, 2008. Vikramaditya Jakkula and Diane J. Cook, “Prediction Models for a Smart Home Based Health Care System”, Special Issue of International Journal of Telemedicine and Applications (IJTA),2008. Diane J. Cook, Juan Augusto and Vikramaditya Jakkula “Ambient Intelligence: Current trends and technologies” PMC Journal, 2007. BOOK CHAPTERS (PENDING) Vikramaditya Jakkula and Diane J. Cook, “Mining Temporal Relations in Smart Environment Data using TempAl”, Knowledge Discovery from Sensor Data, Taylor and Francis/CRC Press, 2008. Vikramaditya Jakkula, Diane J. Cook and Aaron Crandall, “ENHANCING ANOMALY DETECTION FOR SMART HOMES USING TEMPORAL PATTERN DISCOVERY”, Advanced Intelligent Systems, 2008. Vikramaditya Jakkula and Diane J. Cook, “Learning Temporal Relations in Smart Home Data”, Technology and Aging, IOS Press: Assistive Technology Research Series, 2008. Gaurav Jain, Diane J. Cook and Vikramaditya R. Jakkula, ”Monitoring Health by Detecting Drifts and Outliners for a Smart Environment Inhabitant", Smart Homes And Beyond, IOS Press: Assistive Technology Research Series, volume 19, Pg 114-121,2006. ISBN 978-1-58603-623-2,2006.

33. VJ AI@WSU © 200733 Thank You

34. Test Case 2 Results Alz Alz + TempAlAlz Alz + TempAl # # of Instances# of Events # of Anomaly # of PatternsTrainTest # Correct Prediction # Correct Prediction Accuracy (%) Accuracy (%) 16000255205000100060358260.358.2 260001015205000100041142541.142.5 36000105505000100075776375.776.3 46000105205000100091391491.391.4 5325010520250075066871689.0695.4 P-Value 0.120213224 >0.05. Shows its not significantly better but with a closer value to 0.05, it would be a good enhancement model.

35. Visualization : Earlier Model

36. Visualization Enhanced