1. Context-Awareness based on Lifelog Sangkeun Lee Intelligent Database Systems Lab. Seoul National University

2. Copyright  2009 by CEBT Introduction  The whole story begins with my survey ‘A Survey of Context- Aware Systems’ Context & Context-awareness General Process in Context-Aware Systems History of Context-aware Systems – Generations of Context-aware Systems – Trend Analysis Let’s briefly revisit the survey first 2

3. Copyright  2009 by CEBT Context & Context-aware Systems  Context Any information that can be used to characterize the situation of entities relevant to the interaction between a user and an application (Dey and Abowd,2001)  Context-aware Systems The systems that use context to provide relevant information and/or services to the user, where relevancy depends on the user’s task (Dey and Abowd,2001) 3

4. Copyright  2009 by CEBT General Process in Context-Aware Systems  Generally, processing context-awareness can be summarized into following four steps Sense – Getting values from context sources (e.g. sensors) – Context-sources doesn’t have to be hardware devices – (e.g. virtual sensors, other applications) Abstraction – Context Aggregation Temperatures of previous 10 minutes -> Very Cold – Context Interpretation GPS signal -> City name (Recognize) – Understanding current context & trigger the most suitable action Action – Better user’s experience 4

5. Copyright  2009 by CEBT Trend Analysis 5

6. Copyright  2009 by CEBT Insights from the Survey  What makes context-Awareness so difficult? Realizing useful Application/Domain Specific context-aware systems is not a big problem – Using simple, non-flexible context models – In a restricted & known scenarios (e.g. museum, …) – Not a ‘true’ context-awareness for a ubiquitous environment However, realizing a flexible and general context-aware system is a big problem – If you use ontological context model, then you get flexibility and expressiveness of representing context information, but recognizing (understanding) the represented context data causes computational costs, less scalability – not practical! 6

7. Copyright  2009 by CEBT Approach to achieve feasibility and flexibility at the same time?  As I mentioned earlier, context-aware systems are The systems that use context to provide relevant information and/or services to the user, where relevancy depends on the user’s task (Dey and Abowd,2001) Recommender system – Recommender systems or recommendation engines form or work from a specific type of information filtering system technique that attempts to present information items that are likely to be of interest to the user. (Wikipedia) – Similar ! but there are many working systems They use user’s ratings, logs, previous histories, ….  By getting hints from recommender systems that are practically used in many domains, we present a practical way of processing context-awareness by separating ‘Recognize’ step into two steps ‘Collect’ & ‘Match’ Existing approach : Sense - Abstraction – Recognize – Action (Analytic) Practical Approach : Collect - Sense – Abstraction - Match – Action (Empiric) 7

8. Copyright  2009 by CEBT Profile Query Context t1t1 t2t2 t3t3 t4t4 t5t5 t2t2 t4t4 t6t6 Information t7t7 t8t8 t9t9 Service t 10 t 11 t 12 Sense Practical Context-awareness –an Empiric Approach Match Collect

9. Copyright  2009 by CEBT Practical Context-awareness –an Empiric Approach  Is it reasonable to do this? Yes! – Context history can be used to establish trends and predict future context values. [Anind K. Dey and Gregory D. Abowd, 2001] – “Context history is generally believed to be useful, but it is rarely used” [Guanling Chen and David Kotz, 2001] – “Context histories may be used to establish trends and predict future context values.” [Matthias Baldauf, 2007] Many systems keeps recording context history (although they do not utilize them) – Context Toolkit, CoBrA, CASS, SOCAM and CORTEX save sensed context data persistently in a database. 9

10. Copyright  2009 by CEBT Collect & Match  My research consists of two parts Collect – Sense and store context histories – LifeLogOn: Log on to Your Lifelog Ontology! (ISWC ‘09) – Entity-Event Lifelog Ontology Model (EELOM)for Lifelog Ontology Schema Definition (APWEB ‘10) – LifeLogOn: A Practical Lifelog System for Building and Exploiting Lifelog Ontology (UMC ‘10) Match – Find the most suitable information/services based on current context & context histories (logs) using available techniques such as Collaborative filtering, Context-aware collaborative filtering, contents based matching, Simple If-then rules, many other heuristics – A flexible framework for context-aware matching is needed 10

11. Copyright  2009 by CEBT Collect – LifeLogOn: A Lifelog System  Integrate currently available logs from different devices and create semantic relationships among logs  Transforms relational log data and metadata into instance-level Ontology and stores in knowledge base Music Listening Logs Movie Watching Logs GPS Logs Phone Call Logs E-mail History Schedules Music Listening Logs Movie Watching Logs GPS Logs Phone Call Logs E-mail HistorySchedules * Sematnic Relationships LifeLogOn

12. Copyright  2009 by CEBT Collect – ‘Entity-Event Lifelog Ontology Model’  EELOM consists of domain, entity and events A domain is a group of entities and events, and it has a domain name for representing it. An entity is a thing which has a separate existence and can be uniquely identified (e.g. a person, a location, a device, a timestamp) An entity is represented as a set of attributes. Entities can be shared among domains. An event is an activity or interaction that can be uniquely identified. It is represented as a set of entities, and each entity has its role for the event. Why not RDF? OWL? - It does not allow complex relationships (e.g. hierarchy of entities), -but it is still flexible and expressive enough to cover logs of many domains -Easy to construct index structure by considering everything as entities & events

13. Copyright  2009 by CEBT Collect – LifeLogOn Overview Log IDE-mailTimeDateEvent 45324121:11PM09/03/21Played 454215112:06AM09/04/22Played 455215112:10AM09/04/22Stopped TitleGenreArtistBPMTags 45324121:11PM09/03/21Played 454215112:06AM09/04/22Played 455215112:10AM09/04/22Stopped Music Domain Music Atmosphere AlbumGenre Artist BPM Title Play User TimeDate Other Context Music Location Stop User TimeDate Other Context Music Location TitleGenreArtistBPMTags 45324121:11PM09/03/21Played 454215112:06AM09/04/22Played 455215112:10AM09/04/22Stopped TitleGenreArtistBPMTags 45324121:11PM09/03/21Played 454215112:06AM09/04/22Played 455215112:10AM09/04/22Stopped TitleGenreArtistBPMTags 45324121:11PM09/03/21Played 454215112:06AM09/04/22Played 455215112:10AM09/04/22Stopped Last.fm Music Log iTunes Music Log AMG Metadata CDDB Metadata GPS Log from Cell phone Temperature Log from Forecast.com Music@1 Groove Hyori’s backKpop Hyori 90 10 minute Play@1 Location@1 Korea Kangwon 12:31:42 Chunchon User@1 Male HiphopSeoul,Korea Singing 27 Matt Time@1 30 12 02 Play@2 Time@2 12 9 01 Location@2 Korea Seoul 55:24:66 Seoul Music@2 Bright Yellow SubmarineOldpop Beatles 110 Yesterd ay take@1 Photo@1 SonyCD2 640KB Home.jpg 640x480 LifeLogOn Instance-Level Ontology Generator Log– Ontology Schema Mapping Ontology Schema Ontology Generator Ontology Instances Relational Log Data Ontology Schema Definition Tool Visualization & Browsing Tool Log to Ontology Mapper

14. Copyright  2009 by CEBT What can we use LifeLogOn for?  You can create your own lifelog ontology without understanding any ontology languages  You can search something not sure about Find a song you listened at your birthday party and but only know the filename of the photos at the party Find photos that you took when you are talking with your friend on the phone, saying "It's so beautiful here!" Find any events and entities those you know their context information such as time, location,... ... even more 14

15. Copyright  2009 by CEBT We have the context history collection… 15 Many issues remain -Building a lifelog ontology using real data set -Implementing Loggers (maybe on iPhone, …) -Log Ontology Abstraction/Summarization -Better Visualization -Entity Matching, …

16. Copyright  2009 by CEBT Profile Query Context t1t1 t2t2 t3t3 t4t4 t5t5 t2t2 t4t4 t6t6 Information t7t7 t8t8 t9t9 Service t 10 t 11 t 12 Sense Let’s Get Back to Context-awareness Match Collect

17. Copyright  2009 by CEBT Paper Reading  Context-Aware Collaborative Filtering System: Predicting the User’s Preference in the Ubiquitous Computing Environment - Annie Chen, 2005 predicts a user’s preference in different context situations based on past experiences  시간 상황 정보를 고려한 협업 필터링을 이용한 음악 추천 – 2009, 이동주  Context-Aware Recommendation by Aggregating User Context – 2009, 신동민  Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions – 2005, Gediminas Adomavicius  And so on, … 17

18. Copyright  2009 by CEBT Problem Definition  Recommender System Estimating ratings for the items that have not been seen by a user User (explicitly/implicitly - logs) rates Item Input: User, Ratings Output: Item  Context-aware Recommender System (similar but different!) Predicting a user’s preference in different context situations based on past experiences User (explicitly/implicitly) rates Item + each rating has Context Info Input: User, Current Context Output: Item 18

19. Copyright  2009 by CEBT Problem Definition  Context-awareness based on Lifelog Ontology In our Lifelog Ontology, there are no classification of users, items, contexts… All the things in the ontology are Events & Entities 19

20. Copyright  2009 by CEBT Problem Definition  Assume that the system knows the current context (situation)  Context-aware applications are such as Find out the most suitable songs to the current user Find out the most suitable person to be invited to the current location Find out the most similar songs to the song that currently user’s listening to Find out the most related anything to the current situation … Relatedness can be defined in various ways  We can conclude that a context-aware system based on lifelogs is a system that allows dynamically & flexibly composing recommendation based on lifelogs 20

21. Copyright  2009 by CEBT Context-Awareness Based on Lifelogs 21  Our Lifelog model Lifelog Ontology LO is a set of events EV A context history is represented by a event EV that are composed of a set of entities EN – e.g. {Sangkeun Lee, Yesterday, 2009/12/24, Summer, Seoul, SNU} A current context C is represented by a set of entities EN – e.g. {2009/12/25, Hyo-ri Lee, Winter, Hungry, Happy, Busan, …}

22. Copyright  2009 by CEBT Simple Flexible Match Algorithm (on-going work)  Collaborative Filtering (e.g. Music recommendation) 22 User Similar User Based on logs Songs User’s rating Rating prediction based on similar user’s raitings

23. Copyright  2009 by CEBT Simple Flexible Match Algorithm (on-going work)  Generalized Model Raiting is one possible relatedness between users & songs User similarity is one possible relatedness between users  Entity – Entity Relatedness Can be calculated by similarity of events related to each entity Is it only suitable for same type of entities? Not sure. 23 Entity Event Entity … … User Song User … … Song Location Time …

24. Copyright  2009 by CEBT Simple Flexible Match Algorithm 1 (on-going work)  Collaborative Context-aware Filtering 24 Entity Similar Entity Events Entity relatedness in current context Predict relatedness based on similar entity’s relatedness C C E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 C C E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 C C E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 Entity relatedness in current context

25. Copyright  2009 by CEBT Simple Flexible Match Algorithm 2 (on-going work)  Context-Aware Filtering Find out the most related entities to current context Context Similarity Can be defined as an aggregation of entity relatedness Entity Rank The most related entity to current context 25 Current Context can be represented in a set of entities (Assume that all the entities are the categorical values) C C E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 CH 1 E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 E1E1 E1E1 E2E2 E2E2 E3E3 E3E3 E4E4 E4E4 E5E5 E5E5 E6E6 E6E6 Event Rank UserTimeLocationTemperatu re … 1MattSummerChunchonHot… 2KangSummerChunchonHot… 3MattWinterChunchonHot… 4SteveFallSeoulHot… 5…………… Event ranking based on similarity to current context Entity Rank can be calculated by weighted sum

26. Copyright  2009 by CEBT Simple Flexible Match Algorithm (on-going work)  Context-Aware Filtering Flexible? – Context Similarity, Entity Relatedness can be defined in various ways by controlling Weight parameters of context similarity, weight parameters of entity relatedness – You can decide what to recommend (Any entity can be recommended) 26

27. Copyright  2009 by CEBT Simple Match is Not the Only Way – Graph Traverse  Assume that Current Context is {Matt Lee, DSC_2128, 2009/07/23, 018-2144-8842}  Result Entity: matt183/018-2144-8842/ Entity: SNU. Main Gate/37:27:14.60N:126:57:12.33E/37:27:14.60N:126:57:12.33E/ Entity: GS24-TB/ Entity: Green House/37:27:27.29N:126:57:62.52E/37:27:27.29N:126:57:62.52E/ Entity: Jay Yeon/Female/yeon@korea.pe.kr/101-1106 Suwon apt. Suwon Si, Republic Of Korea/Deejay09/010- 4255-5541/ Entity: 2009/01/04/ Entity: Lazy Rhapsody/The Duke: Creole Rhapsody (1931-1932) (disc 1)/Duke Ellington & His Orchestra/196440/ Entity: SNU. Rear Gate/37:27:13.79N:126:57:18.23E/37:27:13.79N:126:57:18.23E/ Entity: 12:35:45/ Entity: Warren Gong/gongstock@europa.snu.ac.kr/138-3 Naksung apt. BongChun dong, Kwanak-gu, Seoul Si, Republic Of Korea/gongfit/019-258-8163/ Entity: SNU Library/37:27:72.11N:126:57:11.32E/37:27:72.11N:126:57:11.32E/ 27

28. Copyright  2009 by CEBT Simple Match is Not the Only Way – Simple If-then Rules  Simple if then rules sometimes much powerful and useful then other recommendation algorithm Many context-aware actions can be defined as simple if-then rules – Alert me when ‘Sangkeun Lee’ is in ‘IDS Lab’ – Turn on light when ‘Sangkeun Lee’ arrives at home – …. CAUTION : should not require complex reasoning or inference 28

29. Copyright  2009 by CEBT Conclusion & Discussion  The Ideal Context-awareness Find out the most related anything to the current situation Considering everything useful Ignoring information noise … 29