1. 2005.03.31 SLIDE 1IS146 – SPRING 2005 Case Study: Cameraphones Prof. Marc Davis, Prof. Peter Lyman, and danah boyd UC Berkeley SIMS Tuesday and Thursday 2:00 pm – 3:30 pm Spring 2005 http://www.sims.berkeley.edu/academics/courses/is146/s05/ IS146: Foundations of New Media

2. 2005.03.31 SLIDE 2IS146 – SPRING 2005 Lecture Overview Review of Last Time –Understanding Visual Media Today –Case Study: Cameraphone Preview of Next Time –Databases

3. 2005.03.31 SLIDE 3IS146 – SPRING 2005 Lecture Overview Review of Last Time –Understanding Visual Media Today –Case Study: Cameraphone Preview of Next Time –Databases

4. 2005.03.31 SLIDE 4IS146 – SPRING 2005 What Are Comics? “Juxtaposed pictorial and other images in deliberate sequence, intended to convey information and/or to produce an aesthetic response in the viewer.” (p. 9) How do comics differ from –Photographs? –Movies? –Hieroglyphics? –Emoticons?

5. 2005.03.31 SLIDE 5IS146 – SPRING 2005 Old Comics: Mayan Codex Nuttall

6. 2005.03.31 SLIDE 6IS146 – SPRING 2005 Scott McCloud’s “Big Triangle” McCloud found that “The Big Triangle” as it came to be known, was an interesting tool for thinking about comics art... Picture Plane Reality Language

7. 2005.03.31 SLIDE 7IS146 – SPRING 2005 Cartoons and Viewer Identification

8. 2005.03.31 SLIDE 8IS146 – SPRING 2005 Closure: From Parts To The Whole

9. 2005.03.31 SLIDE 9IS146 – SPRING 2005 Closure: Bridging Time and Space

10. 2005.03.31 SLIDE 10IS146 – SPRING 2005 Closure in Comics

11. 2005.03.31 SLIDE 11IS146 – SPRING 2005 Types of Closure Moment-To-Moment Action-To-Action Subject-To-Subject Scene-To-Scene Aspect-To-Aspect Non-Sequitur

12. 2005.03.31 SLIDE 12IS146 – SPRING 2005 Questions for Today How do we interpret images and sequences of images? How do we read different visual representations of the world (especially different levels of realism and abstraction) differently? How does what is left out affect how we understand images and sequences of images?

13. 2005.03.31 SLIDE 13IS146 – SPRING 2005 Questions for Today What are some of the differences between how text and images function in comics? What would be lost/gained in moving between images and text?

14. 2005.03.31 SLIDE 14IS146 – SPRING 2005 Questions for Today How could we represent images and sequences of images in order to make them programmable? What could computation do to affect how we produce, manipulate, reuse, and understand images and sequences of images?

15. 2005.03.31 SLIDE 15IS146 – SPRING 2005 Lecture Overview Review of Last Time –Understanding Visual Media Today –Case Study: Cameraphone Preview of Next Time –Databases

16. 2005.03.31 SLIDE 16IS146 – SPRING 2005 What is the Problem? Today people cannot easily find, edit, share, and reuse digital visual media Computers don’t understand visual media content –Digital visual media are opaque and data rich –We lack structured representations Without metadata, manipulating digital visual media will remain like word-processing with bitmaps

17. 2005.03.31 SLIDE 17IS146 – SPRING 2005 Signal-to-Symbol Problems Semantic Gap –Gap between low- level signal analysis and high-level semantic descriptions –“Vertical off-white rectangular blob on blue background” does not equal “Campanile at UC Berkeley”

18. 2005.03.31 SLIDE 18IS146 – SPRING 2005 Signal-to-Symbol Problems Sensory Gap –Gap between how an object appears and what it is –Different images of same object can appear dissimilar –Images of different objects can appear similar

19. 2005.03.31 SLIDE 19IS146 – SPRING 2005 Computer Vision and Context You go out drinking with your friends You get drunk Really drunk You get hit over the head and pass out You are flown to a city in a country you’ve never been to with a language you don’t understand and an alphabet you can’t read You wake up face down in a gutter with a terrible hangover You have no idea where you are or how you got there This is what it’s like to be most computer vision systems—they have no context Context is what enables us to understand what we see

20. 2005.03.31 SLIDE 20IS146 – SPRING 2005 How We Got Here: Disabling Assumptions 1.Contextual (spatial, temporal, social, etc.) metadata about the capture and use of media are not available Therefore all analysis of media content must be focused on the media signal alone 2.Media capture and media analysis are separated in time and space Therefore removed from their context of creation and the users who created them 3.Multimedia content analysis must not involve humans Therefore missing out on the possibility of “human-in-the-loop” approaches to algorithm design and network effects of the activities of groups of users

21. 2005.03.31 SLIDE 21IS146 – SPRING 2005 Where To Go: Enabling Assumptions 1.Leverage contextual, sensory-rich metadata (spatial, temporal, social, etc.) about the capture and use of media content 2.Integrate media capture and analysis at the point of capture and throughout the media lifecycle 3.Design systems that incorporate human beings as interactive functional components and aggregate and analyze user behavior

22. 2005.03.31 SLIDE 22IS146 – SPRING 2005 METADATA Traditional Media Production Chain M E T A D A T A PRE-PRODUCTIONPOST-PRODUCTIONPRODUCTIONDISTRIBUTION Metadata-Centric Production Chain

23. 2005.03.31 SLIDE 23IS146 – SPRING 2005 Moore’s Law for Cameras 2000 Kodak DC40 Nintendo GameBoy Camera $400 $ 40 2002 Kodak DX4900 SiPix StyleCam Blink

24. 2005.03.31 SLIDE 24IS146 – SPRING 2005 Capture+Processing+Interaction+Network

25. 2005.03.31 SLIDE 25IS146 – SPRING 2005 Camera Phones as Platform Media capture (images, video, audio) Programmable processing using open standard operating systems, programming languages, and APIs Wireless networking Personal information management functions Rich user interaction modalities Time, location, and user contextual metadata

26. 2005.03.31 SLIDE 26IS146 – SPRING 2005 Camera Phones as Platform In the first half of 2003, more camera phones were sold worldwide than digital cameras By 2008, the average camera phone is predicted to have 5 megapixel resolution Last month Samsung introduced 7 megapixel camera phones with optical zoom and photo flash There are more cell phone users in China than people in the United States (300 million) For 90% of the world their “computer” is their cell phone

27. 2005.03.31 SLIDE 27IS146 – SPRING 2005 Campanile Inspiration

28. 2005.03.31 SLIDE 28IS146 – SPRING 2005 Mobile Media Metadata Idea Leverage the spatio-temporal context and social community of media capture in mobile devices –Gather all automatically available information at the point of capture (time, spatial location, phone user, etc.) –Use metadata similarity and media analysis algorithms to find similar media that has been annotated before –Take advantage of this previously annotated media to make educated guesses about the content of the newly captured media –Interact in a simple and intuitive way with the phone user to confirm and augment system-supplied metadata for captured media

29. 2005.03.31 SLIDE 29IS146 – SPRING 2005 Campanile Scenario Gathering of Contextual Metadata User Verification Image Capture Gathered Data: Location Data Time Date Username Processing Results: Location City: Berkeley (100%) Day of Week: Saturday (100%) Location Name: Campanile (62%) Setting: Outside (82%) Verified Information: Location Name: Campanile (100%) Setting: Outside (100%) Metadata (and Media) Similarity Processing Metadata and Media Sharing and Reuse

30. 2005.03.31 SLIDE 30IS146 – SPRING 2005 From Context to Content Context –When Date and time –Where CellID refined to semantic place –Who Cellphone user –What Activity as product of when, where, and who Content –When was the photo taken? –Where is the subject of the photo? –Who is in the photo? –What are the people doing? –What objects are in the photo?

31. 2005.03.31 SLIDE 31IS146 – SPRING 2005 SPATIAL SOCIAL Space – Time – Social Space TEMPORAL

32. 2005.03.31 SLIDE 32IS146 – SPRING 2005 What is “Location”?

33. 2005.03.31 SLIDE 33IS146 – SPRING 2005 Camera Location vs. Subject Location Camera Location = Golden Gate Bridge Subject Location = Golden Gate Bridge Camera Location = Albany Marina Subject Location = Golden Gate Bridge

34. 2005.03.31 SLIDE 34IS146 – SPRING 2005 Kodak Picture Spot

35. 2005.03.31 SLIDE 35IS146 – SPRING 2005 Location Guesser Weighted sum of features –Most recently “visited” location –Most “visited” location by me in this CellID around this time –Most “visited” location by me in this CellID –Most “visited” location by “others” in this CellID around this time –Most “visited” location by “others” in this CellID

36. 2005.03.31 SLIDE 36IS146 – SPRING 2005 Location Guesser Performance Exempting the occasions on which a user first enters a new location into the system, MMM guessed the correct location of the subject of the photo (out of an average of 36.8 possible locations): –100% of the time within the first four guesses –96% of the time within the first three guesses –88% of the time within the first two guesses –69% of the time as the first guess

37. 2005.03.31 SLIDE 37IS146 – SPRING 2005 MMM1: Context to Content Context Content When –Network Time Server Where –CellID Who –Cellphone ID What –Faceted Annotation

38. 2005.03.31 SLIDE 38IS146 – SPRING 2005 From MMM-1 To MMM-2 MMM-1 asked –“What did I just take a picture of?” MMM-2 adds –“Whom do I want to share this picture with?” Community Context Content Community

39. 2005.03.31 SLIDE 39IS146 – SPRING 2005 Sharing  Metadata From contextual metadata to sharing –A parent takes a photo of his child on the child’s birthday –Whom does he share it with? From sharing to content metadata –A birdwatcher takes a photo in a bird sanctuary and sends it to her birdwatching group –What is the photo of?

40. 2005.03.31 SLIDE 40IS146 – SPRING 2005 MMM2: Context to Sharing Context Community When –Network Time Server Where –CellID –GPS –Bluetooth Who –Cellphone ID –Bluetooth –Sharing History What –Faceted Annotation –Captions

41. 2005.03.31 SLIDE 41IS146 – SPRING 2005 MMM2: Context to Sharing

42. 2005.03.31 SLIDE 42IS146 – SPRING 2005 MMM2 Interfaces: Phone

43. 2005.03.31 SLIDE 43IS146 – SPRING 2005 MMM2 Interfaces: Web

44. 2005.03.31 SLIDE 44IS146 – SPRING 2005 MMM2 Image Map

45. 2005.03.31 SLIDE 45IS146 – SPRING 2005 More Captures and Uploads STATSMMM1MMM2DIFF Users38405% Days6339-38% Raw totals Personal photos uploaded1551478854% Total photos uploaded5351678214% Photos not uploaded10852-52% Average per user per day Personal photos uploaded0.060.951363% Total photos uploaded0.221.08381% Photos not uploaded0.050.03-26% Upload failure rate16.8%3.0%-82%

46. 2005.03.31 SLIDE 46IS146 – SPRING 2005 Reasons For 13.6 Times Increase Better image quality –VGA vs. 1 megapixel image resolution –Night mode for low light –Digital zoom Familiarity of the user population with cameraphones –12 prior cameraphone users this year vs. 1 last year The availability of only 1 rather than 2 camera applications in MMM2 vs. MMM1 Automatic background upload of photos to the web photo management application Automatic support for sharing on the cameraphone and on the web

47. 2005.03.31 SLIDE 47IS146 – SPRING 2005 More Sharing With Suggestions 0 20 40 60 80 100 120 140 160 11/211/9 11/1611/2311/30 12/7 UPLOADEDSHAREDRECEIVED

48. 2005.03.31 SLIDE 48IS146 – SPRING 2005 More Sharing With Suggestions MMM2 USER BEHAVIOR BEFORE SHARE GUESSER AFTER SHARE GUESSERDIFF TOTAL PHOTOS UPLOADED 688 990144% TOTAL PERSONAL PHOTOS UPLOADED 688 790115% TOTAL PHOTOS SHARED 249 791318% TOTAL PERSONAL PHOTOS SHARED 249 591237% PERCENTAGE OF PHOTOS SHARED36%80%221% PERCENTAGE OF PERSONAL PHOTOS SHARED36%75%207%

49. 2005.03.31 SLIDE 49IS146 – SPRING 2005 Sharing Graph

50. 2005.03.31 SLIDE 50IS146 – SPRING 2005 Number of Sources 100M 1 Photos Per Source 100K Scaling Up Photo Sharing 100K

51. 2005.03.31 SLIDE 51IS146 – SPRING 2005 MMM3: Context Content Sharing Context Community Content When –Network Time Server –Calendar Events Where –CellID –GPS –Bluetooth Who –Cellphone ID –Bluetooth –Sharing History What –Faceted Annotations –Captions –Weather Service –Image Analysis

52. 2005.03.31 SLIDE 52IS146 – SPRING 2005 MMM3 Research Questions MMM1 –Context  Content MMM2 –Context  Community MMM3 –Community  Context –Community  Content –Content  Context –Content  Community Community Context Content

53. 2005.03.31 SLIDE 53IS146 – SPRING 2005 Social Uses of Personal Photos Looking not just at what people do with digital imaging technology, but why they do it Goals –Identify social uses of photography to predict resistances and affordances of next generation mobile media devices and applications Methods –Situated video interviews –Review of online photo sites –Sociotechnological prototyping (magic thing, technology probes)

54. 2005.03.31 SLIDE 54IS146 – SPRING 2005 From What to Why to What

55. 2005.03.31 SLIDE 55IS146 – SPRING 2005 Preliminary Findings Social uses of personal photos –Creating and maintaining social relationships –Constructing personal and group memory –Self-presentation –Self-expression –Functional: self and others Media and resistance –Materiality –Orality –Storytelling

56. 2005.03.31 SLIDE 56IS146 – SPRING 2005 Photo Examples of Social Uses

57. 2005.03.31 SLIDE 57IS146 – SPRING 2005 Summary Cameraphones are a paradigm-changing device for multimedia computing Context-aware mobile media metadata will solve many problems in media asset management MMM1 –Content can be inferred from context MMM2 –Sharing can be inferred from context

58. 2005.03.31 SLIDE 58IS146 – SPRING 2005 Alex Jaffe on Cameraphone Uses Many of the users of cell phone cameras in this paper felt compelled to chronicle very "normal" aspects of their daily life, either to share with others or for personal memories. Do you think the ability to constantly record one's life satisfies an existing desire, or is the technology fulfilling a need it itself inspires in people? Regardless, can you think of examples where technology is used to do something not because there is a need, but simply because it becomes possible?

59. 2005.03.31 SLIDE 59IS146 – SPRING 2005 Alex Jaffe on Cameraphone Uses Respondents indicated that one of their favorite features unique to MMM(2) was their ability to send pictures to people immediately after they were taken. This created a sense of immediacy and "being there" in the viewer. How is communicating in this way reminiscent of orality, albeit in visual form? Might this be an important part of secondary orality in times to come?

60. 2005.03.31 SLIDE 60IS146 – SPRING 2005 Magen Farrar on Context-To-Content “Context-to-content” inferencing promises to solve the problems of the sensory and semantic gaps in multimedia information systems...By using the spatio-temporal-social context of image capture, we are able to infer that different images taken in the vicinity of the Campanile are very likely of the Campanile at UC Berkeley and know that they are not of, for example, the Washington Monument... So, how is the system of “context to content” inferencing changing to allow deciphering, or specifics, between similar content within the same context?

61. 2005.03.31 SLIDE 61IS146 – SPRING 2005 Magen Farrar on Context-To-Content Sharing metadata is exceptionally useful in inferring media content from context, but can potentially violate one's privacy. Other than the opt-in/opt-out mechanisms in the system, what other steps are being thought of to assure the preservation of privacy while sharing information in the Mobile Media Metadata system?

62. 2005.03.31 SLIDE 62IS146 – SPRING 2005 Lecture Overview Review of Last Time –Understanding Visual Media Today –Case Study: Cameraphone Preview of Next Time –Databases

63. 2005.03.31 SLIDE 63IS146 – SPRING 2005 Readings for Next Week Tuesday (Guest Lecture by Dr. Frank Nack) –Lev Manovich. Database as a Symbolic Form. 1999, p. 1-16. http://www.manovich.net/DOCS/database.rtf Discussion Questions –Dorian Peters –Joshia Chang Thursday (Guest Lecture by Prof. Yehuda Kalay) –Steve Harrison and Paul Dourish. Re-Place-ing Space: The Roles of Place and Space in Collaborative Systems. in: Proceedings of ACM Conference on CSCW. New York: ACM Press, 1996, p. 67-76. Discussion Questions –Vlad Kaplun –Annie Chiu