A Proposal for a Video Modeling for Composing Multimedia Document Cécile ROISIN - Tien TRAN_THUONG - Lionel VILLARD Presented by: Tien TRAN THUONG Project OPERA - INRIA Grenoble - France

Work Context * Need: composition of semantic video fragments with other basic media elements (image, text, sound,...) n Theme: Multimedia Document (Madeus) u Authoring system for multimedia structured documents u Basic media: sound, video, text, image, etc. u Document composed by relations

Temporal Synchronization Example INRIA’s positions document Pictures & Titles synchronized with video parts Video Presentation

Video Frames Logical organization of document InriaIntroduction Video Presentation Buildings Overview Image Rocq. Picture RhôneAlpes Picture Text Rocq. Title RhôneAlpes Title Rocq. appears R.A. appears Locations of INRIA’s units Rennes appears Lorraine appears S.A. appears

Locations of INRIA’s unitsConclusion Introduction Rocq. appearsRen. appearsS.A. app... Lorraine appears... RA app. Time line view of the document Time Rocquencourt Title & Picture Rennes Title & Picture Sophia-Antipolis Title & Picture Raw video Lorraine Title & Picture Rhône-Alpes Title & Picture Texts grow up Video fragments

Spatial Synchronization examples Ok HyperlinkTrackingThe text follows a character

Spatial layout of text follow video object document n Location of the video object region that is moving region in the video region Document Region (Left, Top, Width, Height) Text Region (Width, Height) Video Region (Left, Top, Width, Height) Ok Right-Top-Align Video Object Region {x(t), y(t)}

Objective and plan of that work n Research and development on the video modeling for the description of the video content relevant to multimedia applications: u Video modeling: video description for multimedia composition, u Multimedia application: our VideoMadeus is an editing and presentation system.

Video Description n Dublin core: the semantic indexing schema for video content description. n MPEG-7: the future standard tools will enable to define the semantic schemas for description of the audiovisual information. Video => Analysis -> Description -> Applications Scheme of audiovisual applications * Our video modeling for composing multimedia document.

Methodology n Specification of a modeling for the description of video content: u Multi-level structuration, u temporal and spatial relations, u actions interactive on the video elements. n Specification in XML n Experimentation in Madeus (VideoMadeus)

Video Content Description n Video Content Description Multi-level Structuration Video Structure u Structural Description Semantic u Semantic Description Thesaurus u Thesaurus <!ELEMENT VideoContent (MetaInfo, MediaInfo, Summary, Structure, Semantic, Thesaurus)> <!ELEMENT Thesaurus (ReferenceDictionary*, UserDictionary*)> Raw video Occ.1Occ.2Occ.3Occ.4 NabilIrene Structure Semantic Thesaurus Researcher

Video Structure Description n Motivation: for composition, the basis is to have the Structure description level. n Semantic and Thesaurus are more necessary for retrieval applications or as a support for structuration level. * First step is Structure description

High Level Description Video n Video Structure Video Structure u Sequences Sequences u Scenes Scenes u Shots Shots

Shot Content Description n Shot Content Shot u Transition Trans. u SpatialLayout SpatialLayout Reference u Event Event Semantic Index Background u Background Occurrence u Occurrence CameraWork u Camerawork <!ELEMENT Occurrence (Region+, Trajectory?, Occurrence*) > *

Occurrence Content Description n Occurrence Content Occurrence u Trajectory Trajectory u Regions Region u Occurrences Occurrence F Texture F Contour Contour Texture Centroid Region F Color Color F Regions

Model summary n The model focuses on the description of video elements useful for composing a multimedia document (shot, scene, occurrence, event, relation, etc.) n It has a XML specification that makes it independent and easy to apply to multimedia applications (ex. our VideoMadeus).

Experimentation of the model in Madeus - VideoMadeus

Madeus Architecture JAVA XercesJMF OUTILS Editor/Presentation Tools EXECUTION View TIME LINE View HIERARCHICAL View VIDEO STRUCTURED View... PARSERS LOGIC STRUCTURATION TEMPORAL STRUCTURATION SPATIAL STRUCTURATION EVENT MANAGEMENT MODEL MANAGEMENT MADEUS Madeus document n To extend Madeus to VideoMadeus, video content description is handled both in composition and in presentation parts. SAVE

Internal Document Madeus Document Model n Structured document organized according to the dimensions: Logical, temporal, spatial. Madeus Document Actor Content Temporal Spatial Logical …... Madeus Document Actor Content Temporal Spatial n Content that describes the content information of the document n Actor that defines how this basic information in the content part is used in the document (style information, link, etc.) n Temporal for the synchronization between document parts n Spatial for layout specification

Relations n Temporal relations (Allen extension) u meets, starts, equals, during, overlaps, parmin,etc. n Spatial relations u left_align, right_align, center_v, center_h, top_align, bottom_align, etc. … … … … d

Overview of VideoMadeus Video edition View " Structure View " Semantic View " Thesaurus View Element Management " Edit " Play " Search Execution View " Temporal " Spatial Synchronization Management " Hyperlink " Follow-up " Erase " Display, etc... Behavior Management Synchronization Video Index on video Requested descriptions Modified description Requested descriptions XML Description of video content Data Management Internal Structure (MODEL) Parser Modify Editing and Presentation Tools

VideoMadeus document

Editing features n Editing of the video description u shot detection (automatic or manual) u extract manually video objects, events, spatialLayout, etc. n Creating of semantic groups (manual) u group shots in a scene, group scenes in a sequence u detection occurrences of a character (group occurrences in objects) u creation of the other semantic indexing u classifying of the video elements (thesaurus) n scenario editing (composing) u Set temporal and spatial relations between video element and other media u Set actions on the video elements

Conclusion n Provide support for deeper access into video data in the multimedia authoring system: u temporal/spatial synchronization with the other media elements (image, text, sound, etc.), u actions on the video elements (hyperlink, follow-up, erasing, etc.) n Develop experimentally the video editing view to help the user create and modify descriptions of video data in accordance with our video model.

Perspectives n More experimentation for spatial synchronization, n Extension and experimentation of the semantic parts (Semantic and Thesaurus) -> semantic queries, n Use the MPEG-7 tools to specify our video model, n Develop the video content description editing tool: u Integration and adaptation of the video analyzing algorithms for generating more automatically possible the video elements, u Timeline editing view for video structure, etc. n Semantic queries for playing a part of video through network.

Video content description in Madeus document … … … …

Video element definition n The operations can be defined in the instance of the described video: Hyperlink, Tracking, Erasing, Jumping, etc.etc.... <VideoElement ID=«WesternScene» Content=«WesternDS.Seq.Scene1» TypeRenderer=«LightWeight»... > <VideoObject ID=«VO1» Object = «Shot2.ActorOcc1» Actions=«Follow-up;Hyrperlink;...» HRef =«file:///C:/Users/ttran/Multimedia/Madeus/opera.html» />

Temporal part of Inria introduction document... … …

Spatial part of Spatio-Temporal Relation Demo document...