SLIDE 1IS 202 – FALL 2002 Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday 10:30 am - 12:00 am Fall SIMS 202: Information Organization and Retrieval Lecture 09: MPEG-7 et al.

SLIDE 2IS 202 – FALL 2002 Lecture 09: MPEG-7 et al. Methodological Considerations Media Streams Review and Demo MPEG-7 Future Work Assignment 4 Q & A

SLIDE 3IS 202 – FALL 2002 Lecture 09: MPEG-7 et al. Methodological Considerations Media Streams Review and Demo MPEG-7 Future Work Assignment 4 Q & A

SLIDE 4IS 202 – FALL 2002 Methodological Considerations Techne-centered methodology –Construction of theories informed by constructing artifacts –Construction of artifacts informed by (de)constructing theories –Practitioners: Kuleshov, Eisenstein, Papert, Narrative Intelligence Reading Group Inherently interdisciplinary activity –Information science, computer science, film theory and production, media studies, semiotics, user interface and interaction design and testing

SLIDE 5IS 202 – FALL 2002 Lecture 09: MPEG-7 et al. Methodological Considerations Media Streams Review and Demo MPEG-7 Future Work Assignment 4 Q & A

SLIDE 6IS 202 – FALL 2002 What is the Problem? Today people cannot easily create, find, edit, share, and reuse media Computers don’t understand media content –Media is opaque and data rich –We lack structured representations Without content representation (metadata), manipulating digital media will remain like word- processing with bitmaps

SLIDE 7IS 202 – FALL 2002 The Search for Solutions Current approaches to creating metadata don’t work –Signal-based analysis –Keywords –Natural language Need standardized metadata framework –Designed for video and rich media data –Human and machine readable and writable –Standardized and scaleable –Integrated into media capture, archiving, editing, distribution, and reuse

SLIDE 8IS 202 – FALL 2002 New Solutions for Creating Metadata After CaptureDuring Capture

SLIDE 9IS 202 – FALL 2002 After Capture: Media Streams

SLIDE 10IS 202 – FALL 2002 Media Streams Features Key features –Stream-based representation (better segmentation) –Semantic indexing (what things are similar to) –Relational indexing (who is doing what to whom) –Temporal indexing (when things happen) –Iconic interface (designed visual language) –Universal annotation (standardized markup schema) Key benefits –More accurate annotation and retrieval –Global usability and standardization –Reuse of rich media according to content and structure

SLIDE 11IS 202 – FALL 2002 Media Streams GUI Components Media Time Line Icon Space –Icon Workshop –Icon Palette

SLIDE 12IS 202 – FALL 2002 Media Time Line Visualize video at multiple time scales Write and read multi-layered iconic annotations One interface for annotation, query, and composition

SLIDE 13IS 202 – FALL 2002 Media Time Line

SLIDE 14IS 202 – FALL 2002 Icon Space Icon Workshop –Utilize categories of video representation –Create iconic descriptors by compounding iconic primitives –Extend set of iconic descriptors Icon Palette –Dynamically group related sets of iconic descriptors –Reuse descriptive effort of others –View and use query results

SLIDE 15IS 202 – FALL 2002 Icon Space

SLIDE 16IS 202 – FALL 2002 Icon Space: Icon Workshop General to specific (horizontal) –Cascading hierarchy of icons with increasing specificity on subordinate levels Combinatorial (vertical) –Compounding of hierarchically organized icons across multiple axes of description

SLIDE 17IS 202 – FALL 2002 Icon Space: Icon Workshop Detail

SLIDE 18IS 202 – FALL 2002 Icon Space: Icon Palette Dynamically group related sets of iconic descriptors Collect icon sentences Reuse descriptive effort of others

SLIDE 19IS 202 – FALL 2002 Icon Space: Icon Palette Detail

SLIDE 20IS 202 – FALL 2002 Video Retrieval In Media Streams Same interface for annotation and retrieval Assembles responses to queries as well as finds them Query responses use semantics to degrade gracefully

SLIDE 21IS 202 – FALL 2002 Media Streams Technologies Minimal video representation distinguishing syntax and semantics Iconic visual language for annotating and retrieving video content Retrieval-by-composition methods for repurposing video

SLIDE 22IS 202 – FALL 2002 New Solutions for Creating Metadata After CaptureDuring Capture

SLIDE 23IS 202 – FALL 2002 Active Capture Processing CaptureInteraction Active Capture Computer Vision HCI Direction/ Cinematography

SLIDE 24IS 202 – FALL 2002 Automated Capture: Good Capture

SLIDE 25IS 202 – FALL 2002 Jim Lanahan in T2 Trailer

SLIDE 26IS 202 – FALL 2002 Movies change from being static data to programs Shots are inputs to a program that computes new media based on content representation and functional dependency (US Patents 6,243,087 & 5,969,716) Central Idea: Movies as Programs Parser Producer Media Content Representation Content Representation

SLIDE 27IS 202 – FALL 2002 Media Production Chain PRE-PRODUCTIONPOST-PRODUCTIONPRODUCTIONDISTRIBUTION

SLIDE 28IS 202 – FALL 2002 Automated Media Production Process 2 Annotation and Retrieval Asset Retrieval and Reuse Web Integration and Streaming Media Services Flash Generator WAP HTML Print/Physical Media Automated Capture 1 Automatic Editing 3 Personalized Delivery 4 Annotation of Media Assets Reusable Online Asset Database Adaptive Media Engine

SLIDE 29IS 202 – FALL 2002 Lecture 09: MPEG-7 et al. Methodological Considerations Media Streams Review and Demo MPEG-7 Future Work Assignment 4 Q & A

SLIDE 30IS 202 – FALL 2002 MPEG History Moving Picture Experts Group (MPEG) Working group of ISO/IEC in charge of the development of standards for coded representation of digital audio and video Established in 1988, the group has produced –MPEG-1 Standard on which such products as Video CD and MP3 are based –MPEG-2 Standard on which such products as Digital Television set top boxes and DVD are based –MPEG-4 Standard for multimedia for the fixed and mobile web –MPEG-7 Standard for description and search of audio and visual content –MPEG-21 "Multimedia Framework" standard has started in June 2000

SLIDE 31IS 202 – FALL 2002 MPEG-7 Motivation Create standardized multimedia description framework Support range of abstraction levels from low-level signal characteristics to high- level semantic information

SLIDE 32IS 202 – FALL 2002 MPEG-7 Standard

SLIDE 33IS 202 – FALL 2002 MPEG-7 Standard

SLIDE 34IS 202 – FALL 2002 MPEG-7 Description Tools

SLIDE 35IS 202 – FALL 2002 MPEG-7 Top Level Hierarchy

SLIDE 36IS 202 – FALL 2002 MPEG-7 Conceptual Description

SLIDE 37IS 202 – FALL 2002 MPEG-7 Still Image Description

SLIDE 38IS 202 – FALL 2002 MPEG-7 Video Segments Example

SLIDE 39IS 202 – FALL 2002 MPEG-7 Segment Relationship Graph

SLIDE 40IS 202 – FALL 2002 MPEG-7 Applications Today

SLIDE 41IS 202 – FALL 2002 MPEG-7 Future Integration into media production and reuse cycle Better metadata creation Better metadata use

SLIDE 42IS 202 – FALL 2002 Lecture 09: MPEG-7 et al. Methodological Considerations Media Streams Review and Demo MPEG-7 Future Work Assignment 4 Q & A

SLIDE 43IS 202 – FALL 2002 Technical Research Challenges Develop end-to-end metadata system for automated media capture, processing, management, and reuse Creating metadata –Represent action sequences and higher level narrative structures –Integrate legacy metadata (keywords, natural language) –Gather more and better metadata at the point of capture (develop metadata cameras) –Develop “human-in-the-loop” indexing algorithms and interfaces Using metadata –Develop media components (MediaLego) –Integrate linguistic and other query interfaces

SLIDE 44IS 202 – FALL 2002 Lecture 09: MPEG-7 et al. Methodological Considerations Media Streams Review and Demo MPEG-7 Future Work Assignment 4 Q & A

SLIDE 45IS 202 – FALL 2002 Next Time Photo Project Presentations (MED) Assignment 4 –Due by Thursday, September 26 Group Presentation –2 minutes: Presentation of application idea –6 minutes: Presentation of classification and photo browser –2 minutes: residual time for completing explanations and Q + A Photo Browser Page (will be sent to you) Readings for next time (in Protected) –Catch up on what you may have missed

SLIDE 46IS 202 – FALL 2002 Next Tuesday XML and "Document Engineering" (Guest Lecturer: Bob Glushko) Readings –From A Technical Introduction to XML (Norm Walsh, October 13, 1998, xml.com)A Technical Introduction to XML "What is XML“ "What do XML Documents Look Like?" [don't stress over this -- read it lightly] "Validity“ –What is XSLT? (Ken Holman, XML.com, August 16, 2000) [read as much of this as you can, but at least through section ]What is XSLT? –XML Standards and Specifications for Interoperable E- commerce (Bob Glushko) [read as much of this as you can, but at least through "Horizontal Content Standards"]XML Standards and Specifications for Interoperable E- commerce