1. 1 Multimedia Databases n How to represent and organize the databases containing a mix of media types? n MM DB architectures u The Principle of Autonomy: Use suitable index structures for each media type (may result in different index structures) u The Principle of Uniformity: Use same index structure for all media types u The Principle of Hybrid Organization: Hybrid of the above two n Organizing based on the principle of uniformity u associate with each media object o i, some meta-data, md(o i ) u md(o i ) may be produced by a human or program (image/video/text content retrieval engine) u If our archive contains objects o 1, … o n, then index the metadata md(o 1 ), … md(o n ) in a way that provides efficient ways of implementing the expected accesses that users will make

2. 2 Media Abstractions n Considering the content of mm data of different media types, What is common (or different) to all the media types n Media abstractions are mathematical structures representing such media content n Media abstractions may be implemented through a single data structure

3. 3 Media Abstractions n Media abstraction comprises of u a set of states (state is the smallest chuck of media that we wish to reason about) u a set of features (feature is any object in a state that is deemed to be of interest, in principle can include both objects and activities) u attributes associated with features u a feature extraction map that specifies what features occur in which states F in some cases, these are implemented as content extraction programs F in some other cases may involve humans manually specifying content u a set of state-dependent and state-independent relations F e.g., left-of is state-dependent, age is state-independent u a set of inter-state relations F e.g. before(s 1,s 2 ) may say that s 1 occurred before s 2

4. 4 Media Abstractions: Image data n States: {pic1.gif, … pic7.gif} n Features: Names of the people shown in the photographs (Bob, Jim, Bill, Charlie, Ed) n Extraction Map: state feature pic1.gif Bob,Jim pic2.gif Jim pic3.gif Bob … n Relations u state-dependent : left-of u state-independent: father n Inter-state relations may be empty

5. 5 Media Abstractions: Video data n States: frames 1.. 5 n Features: Jane Shady, Denis Dopeman, Dopeman-house, briefcase,.. n Extraction Map: state feature frame1 Jane Shady, Dopeman-house, briefcase frame2 Jane Shady, Denis Dopeman, Dopeman-house, briefcase frame3 Jane Shady, Denis Dopeman, Dopeman-house, briefcase frame4 Jane Shady, Denis Dopeman, Dopeman-house, briefcase frame5 Jane Shady, Dopeman-house

6. 6 Media Abstractions: Video data n Relations u state-dependent : have person object state Jane Shady briefcase 1 Jane Shady briefcase 2 Jane Shady briefcase 3 D.Dopeman briefcase 4 u state-independent: spouse person spouse Jane Shady Peter Shady Peter Shady Jane Shady n Inter-state relations: u before(s 1,s 2 ) holds iff s 1 occurs before s 2

7. 7 Simple Multimedia Database n Is a finite set of media extractions n simple multimedia databases are naive u a media abstraction may list “church” as a feature u when searching for “cathedrals” or “monuments” we may not find church because the system does not know that they are synonymous n users often search for media objects containing one or more features and then “refine” the search later when the returned objects (though correct) do not correspond to what they wanted

8. 8 Structured Multimedia Database n SMDS consists of u set of media abstractions M u equivalence relation (  )on features F u partial ordering (  ) of the set on F/equivalence classes on F u inh: a map that associates with each feature f, a set of features below f according to the ordering on features u subst: is a map from

9. 9 Example of SMDS media object part/frame features image photo1.gif - church,danube image photo2.gif - cathedral,melk image photo3.gif - church,st.paul, rome video video1.mpg 1-5 church,stream video video1.mpg 6-10 stream audio audio1.wav 1-20 st.peters,tiber n 3 media abstractions, one each associated with image, video and audio n features F = church, danube, cathedral, melk, st.paul, rome, stream, st.peters, tiber n equivalence u church  cathedral u river  stream n partial ordering  church cathedral river stream St.paul St.peterdanubeTiber rome melk