1. Mor Naaman, Yee Jiun Song, Andreas Paepcke, Hector Garcia-Molina Digital Library Project, Database Group Stanford University Automatic Organization for Digital Photographs with Geographic Coordinates

2. JCDL 20042 Geo-Referenced Photos April 8 th, 2004 1:20:02pm Latitude: N34.3121 Longitude: W122.234

3. JCDL 20043 Geo-Photography Technology + 1)2)

4. JCDL 20044 Personal Photo Libraries Searching/browsing very difficult Little discernible structure to photo collections

5. JCDL 20045 Content-based retrieval –Basic, primitive (far from semantic) Manual labeling –Improved, yet cumbersome Visual methods for fast scanning (Zoom) –Don’t scale well Managing Personal Photos

6. JCDL 20046 Our Approach Absolutely no human effort required Utilize time and location –Automatically captured –Easy to get

7. JCDL 20047 Automatic Organization

8. JCDL 20048 Automatic Organization

9. JCDL 20049 Automatic Organization

10. JCDL 200410 Automatic Organization

11. JCDL 200411 Outline Requirements and challenges The algorithms Sample output Experiment results

12. JCDL 200412 Browsing by Location/Time Use a map/calendar –wwmx.org from MSR: Map issues –Lots of screen space –Sparse –Limited interaction? –Not intuitive for some

13. Using Hierarchies Time United States Yosemite N.P, Yosemite Valley, CA Location: Around: San Francisco, Berkeley, Sonoma CA San Francisco, Golden Gate Park, CA Seattle, WA … …… … Berkeley, Oakland CA 2003-01-01: Yosemite N.P. (2 Days) 2003-01-18: San Francisco (1 hour) Time:

14. JCDL 200414 Challenges Locations should be intuitive Events are tricky –3-days trip to NYC –The kid’s soccer game, followed by a birthday party Good names are important.

15. JCDL 200415 Outline Requirements and challenges The algorithms Sample output Experiment results

16. JCDL 200416 Process Diagram

17. JCDL 200417 Discovering Structure Location Hierarchy Initial Event Segmentation Location Clustering Final Event Segmentation Event Hierarchy Initial Event Segmentation Automatic Organization

18. JCDL 200418 Initial Event Segmentation Photos occur in bursts Identify bursts: semantically “connected”

19. JCDL 200419 Initial Event Segmentation Stream of photos More details: Graham et al, JCDL 2002 Tomorrow Proceedings

20. JCDL 200420 Discovering Structure Location Hierarchy Initial Event Segmentation Final Event Segmentation Event Hierarchy Location Clustering Automatic Organization

21. JCDL 200421 Location Clusters Cluster the bursts into locations A. Gionis and H. Mannila. Finding recurrent sources in sequences. In Proceedings, Computational molecular biology 2003. –Minimize: number of clusters –Minimize: error (distance to cluster centers)

22. Photo location Location Clusters: 2-D View

23. 2-D View: with Bursts

24. JCDL 200424 Location Clusters Location 4 - Location 3 - Location 2 - Location 1 -

25. Location 4 - Location 3 - Location 2 - Location 1 - Location Clusters (breakdown) Some clusters may be overloaded: –Many bursts / picture-taking days in one location San Francisco

26. JCDL 200426 Discovering Structure Location Hierarchy Initial Event Segmentation Location Clustering Event Hierarchy Final Event Segmentation Automatic Organization

27. JCDL 200427 Final Event Segmentation Again scan sequence, new events detected: –Whenever location context changes –In the same location, use adaptive time threshold

28. JCDL 2004 Final Event Segmentation Overnight trip to Yosemite Soccer game and dinner

29. JCDL 200429 Next - names Detected location and event structure Need to choose names for each node

30. 30 Assigning Names Photo location Stanford Palo Alto City Park Palo Alto Butano State Park Stanford42 Palo Alto30 Butano10 P.A. park8

31. 31 Assigning Names – Nearby? San Jose, 20 miles San Francisco, 30 miles What if photos occur sparsely within cities or parks?

32. JCDL 2004 Assigning Names - Nearby Which city has stronger “gravity”?

33. JCDL 2004 Assigning Names - Nearby San Jose is Closer

34. JCDL 2004 Assigning Names - Nearby San Jose is bigger* *larger population

35. JCDL 2004 Assigning Names - Nearby But San Fran is more important!* *greater Google count Final name for location cluster: “Stanford, 30 miles South of SF”

36. JCDL 200436 Assigning Names - Alexandria Using polygon-based dataset of administrative areas Alexandria gazetteer can be used for other prominent geographic features

37. JCDL 200437 Outline The requirement and challenges of automatic organization The algorithms Sample output Experiment results

38. JCDL 200438 Location Hierarchy Photoshop Album (at least 4 man-hours) Our system (about 0 man- seconds)

39. 39 Location Hierarchy (US) +San Francisco, Berkeley, Sonoma, CA -Stanford, Mountain View, Monterey, CA Monterey (58 miles S of San Jose) Mountain View (4 miles NW of San Jose) Stanford -Colorado (219 miles W of Denver) -Long Beach (35 miles S of Los Angeles, CA) -Philadelphia, PA -Seattle, WA -Sequoia N.P. (153 miles E of Fresno, CA) -South lake Tahoe; Bear Valley, CA -Yosemite N.P.; Yosemite Valley, CA

40. Events about 0 man- seconds:... 2003-06-28: Long Beach,CA (3 days) 2003-07-04: San Francisco,CA (3 hours) 2003-07-10: Colorado (3 days) 2003-07-15: San Francisco,CA(1 hours) 2003-07-18: Mountain View,CA (5 hours) 2003-07-27: San Francisco,CA (1 hours) 2003-09-28: Philadelphia,PA (1 hours) 2003-10-03: Sequoia NP (3 days)... Photoshop Album (at least 4 man-hours)

41. JCDL 200441 Event Names LOCALE: share automatically Check personal calendar Event Gazetteer Easy interface

42. JCDL 200442 Experiment Tested on 3 real-world geo-referenced photo collections Our system automatically generated the structure and names Tested with the owners

43. JCDL 200443 Experiment - Locations Accepted the automatic hierarchy Only minor edits requested –Merge/split few of the locations

44. JCDL 200444 Experiment - Events Compared to events as annotated by users 80-85% in both recall and precision Other metrics proposed (see paper)

45. JCDL 200445 Experiment - Naming Naming location clusters –For 76% of clusters, system and users pick at least one name in common –For the rest, “automatic” name was useful

46. Not yet published: Paid 13 participants to “geo-reference” their photos Loaded to WWMX and our browser –Most liked the map better, but… –Performed the same for search/browse tasks –Event notion helps overcome location handicap –Organization “made sense” P.S. Some didn’t touch the map, yet used our location hierarchy. P.S.2 This was on a BIG screen!

47. JCDL 200447 Thank You! More details: Proceedings Google: Mor Naaman mor@cs.stanford.edu http://www-db.stanford.edu/~mor/

48. JCDL 200448 Future Work User interface PDA Integrate with map Global photo libraries

49. JCDL 200449 Enhancing Personal Collections Browse/search for photos by location Detect events

50. JCDL 200450 Our System: PhotoCompas Location hierarchy Event hierarchy Our algorithm: –Creates them simultaneously Such that they inform each other –Assigns geo-names using gazetteers and Google

51. 48 != 48 48 hours in Seattle: One trip. 48 hours in San Francisco: Countless events. Monday 7am Monday 4pm Tuesday 8am Tuesday 1pm Monday 11am Monday 1pm Monday 7pm …

52. JCDL 200452 Why not use simple location hierarchy? What’s wrong with country state city? States division is usually perceived by users (in US) but not always City list likely to be too long Many countries do not have “states”

53. JCDL 200453 Initial Event Segmentation

54. JCDL 200454 Remember The Bursts?

55. JCDL 200455 Location Clusters 1 - 2 -

56. JCDL 200456 Location Clusters 1 - 2 - 3 - 4 - 6 - 5 - 7 - 8 -