Lost in Memories Interacting With Large Photo Collections on PDAs Susumu Harada, Mor Naaman*, Yee Jiun Song, QianYing Wang, Andreas Paepcke Digital Library Project Stanford University

JCDL Motivation Small devices ubiquitous Storage, bandwidth: cheap The new photo wallet: PDA/Cellphone Can have access to all my photos Co-present sharing

JCDL Challenges Acquisition rate Small screen Find photos now! In addition… !

JCDL Personal Photo Collections Searching/browsing very difficult Little discernible structure to photo collections Warning: repeat!

JCDL Content-based retrieval –Basic, primitive (far from semantic) Manual labeling –Improved, yet cumbersome Visual methods for fast scanning (Zoom) –Don’t scale well, utilize unavailable screen space Managing Personal Photos Warning: repeat!

JCDL Our Approach Absolutely no human effort required Utilize automatically-captured, easy-to- get metadata, like time [and location] Warning: repeat!

JCDL 2004 Oy Vey! Hmm… I can do this! [JCDL 2002] Big Mama

JCDL Outline Timeline Browsing –What are “Events” –Timeline user interface Time clustering (detecting events) Experiment

JCDL Existing PDA Photo Browsers Require manual organization Even then, is it good enough?

JCDL Instead Use natural notion of time and event: –Photos that were taken in the same occasion and context.

JCDL Events: Time-based Structure Birthday Europe July 4th Feb Jul Sep Shoot bursts 2003 …

JCDL Events Birthday Europe July 4th Feb Jul Sep 2pm 5pm 6pm Feb 12 Cake Gifts Sub events

13 Timeline PDA Opening Game Automatically created “meaningful” events “Drill in”: Click on any box, or use timeline

Number of Photos Stack icon Time range Back button Scroll Negative space Go to Thumbnail view Dates

15 Zoom in time

16 Flipping Through a Pile in Place

17 That photo of the DB group members from WWW2003…

18 That photo of the DB group members from WWW2003…

JCDL Outline Timeline Browsing Time clustering (detecting events) Experiment

JCDL Event Detection in Personal Collections Graham et al (our project). Time as essence for photo browsing through personal digital libraries. JCDL U. Gargi. Time-based analysis and event clustering. HP Tech Report (2003). Platt et al. Phototoc: Automatic clustering for browsing personal photographs. MSR Tech Report (2003). Stent and Loui. Using event segmentation to improve indexing of consumer photographs. SIGIR Cooper et al. Automatically Organizing Digital Photographs Using Time and Content IEEE Image Processing (2004). More…

JCDL Events According to Graham (et al) Time 1 day, 11 hours –Detect threshold gaps (6-24 hours) –Within each segment Find outliers Split at outliers –Repeat recursively PDA: Merge at any level – limit to 10 events 10, 15, 13, 26, 400, 21, 55,…

JCDL Event Tree Days Months Years

JCDL Outline Timeline Browsing Time clustering (events) Experiment

JCDL Experiment: Basic vs. Timeline TL thumbTL time view Basic Timeline 15 subjects Average 1200 personal photos

JCDL Tasks Search: find randomly chosen photo in own collection as quickly as possible Browsing: build a collage of –Friends –Family –Trip –Special events “Make time span and set of occasions broad”

JCDL The Interface or the Organization? Controlled using an additional condition Use automatic organization (events) with basic interface

JCDL Even with basic interface, “automatic” matches “manual” After learning, Timeline improves search time by 29% over basic. Timeline success rate better Least backtracking in Timeline (after learning) Results

JCDL Time (sec) Number of Subject-Trials Year Month Week Day Thumbs Full View Dwelling Times

JCDL At least as good as manual organization Novel interface beneficial Learning effects considerable (even better) Conclusions

JCDL Geo-photos, of course Utilizing “off screen space” (classify, share) Data collection Current Work

JCDL Off Screen Space

JCDL Thank You! More details: Proceedings Google: Mor Naaman

JCDL Additional Slides

JCDL Three Phases of Photo Search Opening game: –Large-granularity search strategies Roughly when? [Where was it? Who was in the picture?] Middle game: –Check search assumptions Is this really the right year? No, Bobby wasn’t born yet. End game: –Look at individual photos in detail

JCDL Experiment Setup Browser Interface Baseline Timeline Manual organization Automatic organization BM BA TL1 TL2