1. Multimedia Databases: Where are we?
Vincent Oria
Department of Computer Science
New Jersey Institute of Technology
2005 Knowledge Fusion Workshop

2. Multimedia databases: Challenges
Retrieval and Indexing
“Give me all video sequences showing Steve laughing”
Storage, Communication and performances
Standards (MPEG)
Streaming and Resource Scheduling
Continious data such as audio and video need continious transport (QoS)
Authoring
Presentation
2005 knowledge Fusion Workshop

3. 2005 knowledge Fusion Workshop
Outline
Mono or Multi-media databases (or data repositories)
Image Content Analysis and Description
Image Querying
Multimedia Metadata Standards
Conclusion
2005 knowledge Fusion Workshop

4. Multimedia Database Architecture
MM Data
Pre-
processor
<!ELEMENT ..>
.....
<!ATTLIST...>
Meta-Data
Recognized
components
Additional
Information
Query Interface
<article>
.....
</article>
MM Data
MM Data
Instance
MM Data
Instance
Users
Multimedia DBMS
Multimedia Data
Preprocessing System
Database Processing
2005 knowledge Fusion Workshop

5. Document Database Architecture
<!ELEMENT ..>
.....
<!ATTLIST...>
DTD
Manager
DTD files
DTD Parser
DTD
Type
Generator
Query Interface
<article>
.....
</article>
<!ELEMENT ..>
.....
<!ATTLIST...>
Document
content
SGML/XML
Parser
DTD
XML or SGML
Document
Instance
SGML/XML
Documents
Parse
Tree
C++ Types
Users
Instance
Generator
Multimedia DBMS
C++ Objects
Document
Processing System
Database Processing
2005 knowledge Fusion Workshop

6. Image Database Architecture
Image Analysis
and Pattern Recognition
Semantic
Objects
Syntactic
Objects
Image Content
Description
Meta-Data
Query Interface
<article>
.....
</article>
Image
Annotation
Image
Image
Users
Multimedia DBMS
Image
Processing System
Database Processing
2005 knowledge Fusion Workshop

7. Video Database Architecture
Video Analysis
and Pattern Recognition
Key
Frames
Video Content
Description
Meta-Data
Query Interface
<article>
.....
</article>
Video
Video
Annotation
Video
Users
Multimedia DBMS
Video
Processing System
Database Processing
2005 knowledge Fusion Workshop

8. Multimedia at the Confluence of Many Disciplines
DBMS
Modeling
Query processing …
Machine Learning
Neural Networks
Agents
Knowledge Representation …
Database Systems
Artificial Intelligence
Computer graphics
Human Computer
Interaction
3D representation …
Information Retrieval
Indexing
Inverted files …
Visualization
Networks
mathmatics
Communication
Timing
QoS …
Coding
Statistical and
Mathematical
Modeling …
Other
2005 knowledge Fusion Workshop

9. Image Content Analysis
Image content analysis can be categorized in 2 groups:
Low-level features: vectors in a multi-dimensional space
Color
Texture
Shape
Mid- to high-level features: Try to infer semantics
Semantic Gap
2005 knowledge Fusion Workshop

10. Image Content Analysis: Color
Color space:
Multidimensional space
A dimension is a color component
Examples of color space: RGB, CIELAB, CIEL*u*v*, YCbCr, YIQ, YUV, HSV
RGB space: A color is a linear combination of 3 primary colors (Red, Green and Blue)
Color Quantization
Used to reduce the color resolution of an image
Three widely used color features
Global color histogram
Local color histogram
Dominant color
2005 knowledge Fusion Workshop

11. 2005 knowledge Fusion Workshop
Color Histograms
Color histograms indicate color distribution without spatial information
Color histogram distance metrics
2005 knowledge Fusion Workshop

12. Image Content Analysis: Texture
Refers to visual patterns with properties of homogeneity that do not result from the presence of only a single color
Examples of texture: Tree barks, clouds, water, bricks and fabrics
Texture features: Contrast, uniformity, coarseness, roughness, frequency, density and directionality
Two types of texture descriptors
Statistical model-based
Explores the gray level spatial dependence of texture and extracts meaningful statistics as texture representation
Transform-based
DCT transform, Fourier-Mellin transform, Polar Fourier transform, Gabor and wavelet transform
2005 knowledge Fusion Workshop

13. Image Content Analysis: Shape
Object segmentation
Approaches:
Global threshold-based approach
Region growing,
Split and merge approach,
Edge detection app
Still a difficult problem in computer vision. Generally speaking it is difficult to achieve perfect segmentation
2005 knowledge Fusion Workshop

14. Salient Objects vs. Salient Points
Generic low-level description of images into salient objects and salient points
Original images
Segmented images with region boundaries
Extracted
salient points
2005 knowledge Fusion Workshop

15. Modeling Images – Principles
Support for multiple representations of an image
Support for user-defined categorization of images
Well-defined set of operations on images
An image can have (semantic, functional, spatial) relationships with other images (or documents) which should be represented in the DBMS
An image is composed of salient objects (meaningful image components)
2005 knowledge Fusion Workshop

16. Salient Object Modeling
Multiple representations of a salient object (grid, vector) are allowed
A salient object O is of a particular type which belongs to a user defined salient object types hierarchy
An image component may have some (semantic, functional, spatial) relationships with other salient objects
2005 knowledge Fusion Workshop

17. General Approach to Similarity Queries
A descriptor is a point in a multidimensional space
Querying consists in defining a metric in the space and computing distances between a query point and the points in the space.
2005 knowledge Fusion Workshop

18. The vector-based Similarity Searches
1) extract from each object N numerical features and map objects into points of a N-dimensional space
2) use a suitable distance (e.g., Euclidean) over such a space, and search for “close” objects using a multi-dimensional (“spatial”) index (low distance = high similarity)
2005 knowledge Fusion Workshop

19. User-defined similarity
Using the same distance function is not always appropriate
Example: retrieve (only) black points
2005 knowledge Fusion Workshop

20. 2005 knowledge Fusion Workshop
Query Processing
Integration Problem for Complex Image Queries
Limits of the Common Approach
How Qualitative Preferences Contribute to an Effective and Efficient Solution?
BesTop Operator
iMPO Algorithm
2005 knowledge Fusion Workshop

21. Integration of Sub-Query Results
Image similarity queries are processed by splitting the overall query into sub-queries. How to obtain an effective and efficient integrating result ? I1
f0,1
f0,3
f0,2
f1,1
f1,3
f1,2
Feature Extraction
The common approach is to define a monotonic scoring function (e.g. min and avg) that associates to each object Ii a numeric value (overall score) si
An object is better (preferred to) than another iff its s is higher
s1,1
s1,3
s1,2
Feature Comparison
Scoring Function s1
2005 knowledge Fusion Workshop

22. Common Approach: Efficiency
In order to minimize the number of DB accesses, “middleware” algorithms are applied to return only the Top-k highest scored objects
TA [FLN01]
MEDRANK [FKS03]
Upper [BGM02]
sub-query C
sub-query T C T
I2 S2,T
I2 S2,C
I1 S1,T
I3 S3,C
I3 S3,T
I4 S4,C
I5 S5,T
I1 S1,C
I4 S4,T
I5 S5,C
TEXTURE
COLOR
Top-k
STOP! …
2005 knowledge Fusion Workshop

23. 2005 knowledge Fusion Workshop
Skyline Operator
The Skyline filters out the set of objects that are not dominated by any other object
“An object dominates another object if it is equal or better in all its dimensions and better in at least one dimension”
Hotel name
Distance to beach (km)
Price ($)
Opera
0.5
300
Chariot
2.5
100
Star 3 50
Hilton
3.5
350 t1 t4 t2 t3 t3 t2 t4 t1 t3 t4 t2 t1
2005 knowledge Fusion Workshop

24. 2005 knowledge Fusion Workshop
Best Operator
Skyline filters out only the “first ” better results
…but what happens if the cardinality of Skyline is low and the user want more results?
The Best operator has been recently proposed
Independent from the selected partial order
Given a PO and the level n (n>0), Best computes
the best results (level 1)
the “second choices” (level 2) …
the “n-th choices” (level n) t3 t2 t1 t4
Skyline: computes only the first level
Best operator takes any partial order relation as input and returns the result level by level
2005 knowledge Fusion Workshop

25. 2005 knowledge Fusion Workshop
MM Example: Best … …
2005 knowledge Fusion Workshop

26. 2005 knowledge Fusion Workshop
MPEG-7 Objectives
MPEG-7, formally called ”Multimedia Content Description Interface”, will standardise:
A language to specify description schemes, i.e. a Description Definition Language (DDL).
A set of Description Schemes and Descriptors A scheme for coding the description
7.htm
2005 knowledge Fusion Workshop

27. Content Context and Objectives
Specification of a “Multimedia Content Description Interface”
Developed by the International Standard Organization and the International Electrotechnical Commission (IEC)
Standardized representation of multimedia metadata in XML (XML Schema Language)
Describes audio-visual content as a number of levels (features, structures, semantics, models,…)
2005 knowledge Fusion Workshop

28. MPEG 7 Context and Objectives
Content Description
format independent
may be applied to analogue media
different description granularities
Supplementary Data
Application Types

29. Content Description in MPEG-7
Metadata
Author, producer, copyright
Semantics
Objects, events, people
Structure
Region, segments
Features
Colors, textures, shapes
2005 knowledge Fusion Workshop

30. 2005 knowledge Fusion Workshop
MPEG-7 Framework
MM Content
User
Description
MPEG7 Description
Generation
Definition Language
(DDL)
Filter
Agents
MPEG7 Description
MPEG7
Description
Schemes (DS) & Descriptors (D)
Search /
Query
Engine
MPEG7
Coded
Description
Encoder
Decoder
2005 knowledge Fusion Workshop

31. 2005 knowledge Fusion Workshop
MPEG-7 Example
2005 knowledge Fusion Workshop

32. 2005 knowledge Fusion Workshop
MPEG-7 Example (cont)
<object_hierarchy type="PHYSICAL"> <!-- Physical hierarchy -->
<object_node id="10" object_ref="0"> <!-- Portrait -->
<object_node id="11" object_ref="1"> <!-- Father -->
<object_node id="12" object_ref="4"/> <!-- Father’s face -->
</object_node>
<object_node id="13" object_ref="2"> <!-- Mother -->
<object_node id="14" object_ref="5"/> <!-- Mother’s face -->
</object_hierarchy>
<object_hierarchy type="LOGICAL"> <!-- Logical hierarchy: faces in the image -->
<object_node id="15" object_ref="3"> <!-- Faces -->
<object_node id="16" object_ref="4"/> <!-- Father’s face -->
<object_node id="17" object_ref="5"/> <!-- Mother’s face -->
2005 knowledge Fusion Workshop

33. 2005 knowledge Fusion Workshop
MPEG-21
MPEG-7 does not take into account the aspect of the organization of the infrastructure of distributed multimedia systems
Initiated in 2000 to provide mechanisms for distributed multimedia systems design and associated services
A new distribution entity is proposed and validated: the Digital item
2005 knowledge Fusion Workshop

34. Conclusion: Towards more Semantics
Semantic Representation
MPEG-7
Semantic Acquisition
Automated approach for all purpose images and videos have failed
Manual interpretation is tedious
How to minimize human interventions?
2005 knowledge Fusion Workshop

35. Towards more Semantics: Challenges
“A picture is worth a thousand words”
“It is not so much that a picture is worth a thousand words, for many fewer words can describe a still picture for most retrieval purpose, the issue has more to do with the fact that those words vary from one person to another”
``In spite of almost fifty years of research, design of a general-purpose machine pattern recognizer remains an elusive goal....The best pattern recognizers in most instances are humans, yet we do not understand how humans recognize patterns".
2005 knowledge Fusion Workshop

36. 2005 knowledge Fusion Workshop
Conclusions
Gap between research and applications in general
Semantic Gap
What is Next?
Capture more semantics
Real multimedia databases and application
Indexing and dimensional reduction
Integration of Multimedia and Classical Data
2005 knowledge Fusion Workshop