2. IR in HDMs (Liadh Kelly)- 1 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Venturing into the Labyrinth: the Information Retrieval Challenge of Human Digital Memories Liadh Kelly Gareth J.F. Jones Centre for Digital Video Processing Dublin City University

3. IR in HDMs (Liadh Kelly)- 2 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Background and Overview Overview: It is now possible to capture and store one’s entire life experiences in a Human Digital Memory (HDM). HDMs are of little use if individuals cannot search through them to find important content. We suggest how an individual might search these potentially vast archives of personal information. Human Digital Memory Computersmobile phones Camerasvideo recorders audio recordersGPS technology sensor technology … items read, written, downloaded footage from life experiences photographs taken, videos seen, music heard details of places visited, details of people met … generated using containing

4. IR in HDMs (Liadh Kelly)- 3 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g HDMs HDM - Labyrinth of information: –Diverse range of item types. –Incompletely linked. –Complex and seemingly unintelligible labyrinth. => new IR techniques required - Backend algorithms. - User interface.

5. IR in HDMs (Liadh Kelly)- 4 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g We propose that rich sources of personal context can be exploited to trigger a person’s memory and to aid retrieval. Memory & Context Data Context data: Users lazy! Timestamps, GPS technology, Bluetooth, biometric sensors, data linking. Last saw photo – sunny day ?? On phone to Jack Here’s the photo!

6. IR in HDMs (Liadh Kelly)- 5 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Context data Timestamps – time of item creation/access. GPS technology – location of item creation/access. Timestamps & GPS – light status and weather conditions at creation/access time. Bluetooth tracking – people present at creation/access time. Biometric sensors (Polar Heart Rate Monitor, BodyMedia SenseWear Armband) – arousal, such as excitement or boredom, at creation/access time.

7. IR in HDMs (Liadh Kelly)- 6 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Item Linking Linking items based on: Past interactions. Content & context associations between items. Context Data Processor (output: Item & Associated Context data) User Access Patterns Database Item Linking Generator Raw Context Data Item Created or Accessed HDM with associated context data and linking Remembered context Retrieved Item(s)

8. IR in HDMs (Liadh Kelly)- 7 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Linked Structure Link structure used to measure page importance. –e.g. use PageRank to locate interesting items. Linked structure could allow for the suggestion of items which are related (in some sense) to the item currently being examined or worked on.

9. IR in HDMs (Liadh Kelly)- 8 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Interfaces for HDMs We anticipate that people’s remembering processes can be used as a basis for the design of effective interfaces to aid retrieval from HDMs. Great challenge to create interface for the HDM labyrinth, because HDM may be: –vast in size. –item may be labeled with many context features. –item may be linked to large number of other items. –space is personal to the user – may have specific memory cues that use to recall information. Interface – intuitive, simple to use, offer required functionality. Last saw photo – sunny day ?? On phone to Jack Here’s the photo!

10. IR in HDMs (Liadh Kelly)- 9 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Sample Prototype Interface file type people access location weather date recommend ……. …… ……. A B C D E FG new favorites

11. IR in HDMs (Liadh Kelly)- 10 - Centre for Digital Video Processing C e n t r e f o r D I g I t a l V I d e o P r o c e s s I n g Open Issues Research Objectives: To explore and evaluate the use of existing ranking algorithms, e.g. probabilistic model and PageRank, on HDMs. To investigate additional sources of context data. To propose new algorithms for effective retrieval from HDMs using context and content. To generate test collections and devise experiments to evaluate algorithms.