1. CH 11 Multimedia IR: Models and Languages
Presented by:
Mohammed Al-Rashidi
Supervised by:
Dr. Mourad Ykhlef

2. Multimedia IR system
The architecture of a Multimedia IR system depends on two main factors
The peculiar characteristics of multimedia data
The kinds of operations to be performed on such data

3. Multimedia IR system Support variety of data Different kinds of media
Text, images (both still and moving), graphs, and sound
Mix of structured and unstructured data
Metadata
Semi-structured data
Data whose structure may not match, or only partially match, the structure prescribed by the data schema
The system must typically extract some features from the multimedia objects

4. Multimedia IR system Data retrieval
Exploiting data attributes and the content of multimedia objects
Basic steps
Query specification
Fuzzy predicates (Find all images similar to a car)
Content-based predicates (Find all objects containing an apple)
Object attributes (Find all red images)
Structural predicates (Find all multimedia containing a video clip).

5. Multimedia IR system Combine DBMS and IR technology
2. Query processing and optimization
Query is parsed and compiled into an internal form
3. Query answer
4. Query iteration
Combine DBMS and IR technology
DBMS: data modeling capabilities
IR system: similarity-based query capabilities

6. Data modeling Main tasks
A data model should be defined by which the user can specify the data to be stored into the system
Support conventional and multimedia data types
Provide methods to analyze, retrieve, and query such data
Provide a model for the internal representation of multimedia data

7. Object-oriented DBMS Provide rich data model Class Drawback
More suitable for modeling both multimedia data types and their semantic relationships
Class
Attributes +operations
Inheritance
Drawback
The performances of storage techniques, query processing, and transaction management is not comparable to that of relational DBMSs
Highly non-standard

8. Object-relational DBMS
Extend the relational model
Represent complex data types
Maintain the performance and the simplicity of relational DBMSs and related query languages
Define abstract data types
Allows one to define ad hoc data types for multimedia data

9. Internal representation
Using attributes is not sufficient
Feature
Information extracted from objects
Multimedia object is represented as a set of features
Features can be assigned manually, automatically, or using a hybrid approach

10. Internal representation
Values of some specific features are assigned to a object by comparing the object with some previously classified objects
Feature extraction cannot be precise
A weight is usually assigned to each feature value representing the uncertainty of assigning such a value to that feature
80% sure that a shape is a square

11. SQL3 Support extensible type system Collection data types
Provide constructs to define user-dependent abstract data types, in an object-oriented like manner
Collection data types
Sets, multisets, and lists
The elements of a collection must have compatible types

12. MULTOS MULTimedia Office Server
Client/server
Support filing and retrieval of multimedia objects
Each document is described by a logical structure, a layout structure, and a conceptual structure
Documents having similar conceptual structures are grouped into conceptual types

13. Conceptual structure of the type Generic_Letter
spring component type
Document
Place
Date
Receiver+
Sender
Address
Name
City
Street
Country
Letter_body
Conceptual structure of the type Generic_Letter

14. Product_Presentation
Document
Place
Date
Receiver+
Sender
Address
Name
City
Street
Country
Company_Logo
Image
Product_Presentation
Text
Product_Description
Product_Cose
Signature
Letter_body
Complete conceptual structure of the type Business_Product_Letter

15. Query languages Relational/object-oriented database system
Exact match of the values of attributes
Multimedia IR system
Similarity-based approach
Considers the structure and the content of the objects
Content-based query
Retrieve multimedia objects depending on their globe content

16. Query languages
In designing a multimedia query language, three main aspects require attention
How the user enters their request to the system
Which conditions on multimedia objects can be specified in the user request
How uncertainty, proximity, and weights impact the design of the query language

17. Request specification
Interfaces
Browsing and navigation
Specifying the conditions the objects of interest must satisfy, by means of queries
Queries can be specified in two different ways
Using a specific query language
Query by example
Using actual data (object example)

18. Search Using Sketch in QBIC
Sketch entry
Results of search

19. Search by Color in QBIC Color selection 15% yellow and 13% blue
Results of search

20. Search by User Drawn Pattern in QBIC
Results of search

21. Conditions on multimedia data
1. Attribute predicates
Concern the attributes for which an exact value is supplied for each object
Exact-match retrieval
2. Structural predicates
Concern the structure of multimedia objects
Can be answered by metadata and information about the database schema
“Find all multimedia objects containing at least one image and a video clip”

22. Conditions on multimedia data
3. Semantic predicates
Concern the semantic content of the required data, depending on the features that have been extracted and stored for each multimedia object
“Find all the red houses”
Exact match cannot be applied

23. Thanks
Any questions