1. Alexander Gelbukh www.Gelbukh.com
Special Topics in Computer Science Advanced Topics in Information Retrieval Lecture 5 (book chapter 11): Multimedia IR: Models and Languages
Alexander Gelbukh

2. Previous Chapter: Conclusions
Inverted files seem to be the best option
Other structures are good for specific cases
Genetic databases
Sequential searching is an integral part of many indexing-based search techniques
Many methods to improve sequential searching
Compression can be integrated with search

3. Previous Chapter: Research topics
Perhaps, new details in integration of compression and search
“Linguistic” indexing: allowing linguistic variations
Search in plural or only singular
Search with or without synonyms

4. Motivation Applications: Example: office, CAD, medical, Internet
Artists sings a melody and sees all the songs with similar melody

5. What’s different Different from text IR: Aspects:
Structure of data is more complex. Efficiency is an issue
Using of metadata
Characteristics of multimedia data
Operations to be performed
Aspects:
Data modeling: Extract and maintain the features of objects
Data retrieval: based not only on description but on content

6. Retrieval process Query specification
fuzzy predicates: similar to
content predicates: images containing an apple
data type predicates: video, ...
Query processing and optimization
Parsed, compiled, optimized for order of execution
Problem: many data types, different processing for each
Answer
Relevance: similarity to query
Iteration
Bad quality, so need to refine

7. Modeling

8. Data modeling
To model is to simplify, in order to make manageable. “We will represent an image as...”
From the user’s point of view
From the system’s point of view (technically)
A problem: very large storage size. Modeling needed
Objects are represented as feature vectors
Images / Video: shape. House, car, ...
Sound: style. Music: Merry, sad, ...
Features are defined directly or by comparison
Degree of certainty is stored

9. Multimedia support in commercial DBMSs (1999)
Variable length data.
Non-standard
Different and usually very limited sets of operations
SQL3:
provides user-extensible data types
Object-oriented
Implemented partially in many systems
Example: data blades of Informix
Content-based functions on text and images
E.g.: date = 1997 AND contains (car)

10. Spatial data types Informix: 2D, 3D data blades Boxes, vectors, ...
Operations: intersect, contains, center, ...
Text: containWords, ....
Supports query images by content

11. Example: MULTOS Multimedia document server Documents are described by:
logical structure: title, into, chapter, ...
layout structure: pages, frames, ...
conceptual structure: allows content-based queries
Docs similar in conceptual structures are grouped into conceptual types
Example: Generic_Letter

12. Example of conceptual structure...

13. ...continued

14. Image data in MULTOS Analysis Result of analysis:
low level: detect objects and positions
high level: image interpretation
Result of analysis:
description of objects found and their classes
certainty values
Indices are used for fast access to this info
Object index. Includes pointers to objects and certainty values
Cluster index, with fuzzy clusters of similar images

15. Internet How Google does it? No image processing. Textual context!
File names, nearby words
Distance from image to words
“give me images with flower in the file name or near the image”

16. Languages

17. Query languages
As a query, either a description of the object or an example object is submitted
“show me images similar to this one”
in what respects similar?!
Exact match is inadequate. Additional means are needed
Content is not a single feature

18. What defines query language
Interface. How to enter the query
Types of conditions to specify
Handling of uncertainty, proximity, weights

19. Interface Browsing and navigation
Search: description or query by example
Query by example:
specify what features are important. Give me all houses with similar shape but different colors
Libraries of examples can be provided

20. Conditions... Attribute predicates Structural predicates
structured content – the predefined types extracted beforehand
Exact match. E.g.: size, type (video, audio, ...)
Structural predicates
structure: title, sections, ...
metadata are used. Find objects containing an image and a video clip
Semantic predicates
unrestricted content.
Find all red houses: red = ?, house = ? Fuzzy

21. ... conditions Predicates
Spatial: contain, intersect, is contained in, is adjacent to ...
Temporal: Find audio where first politics and then economy is discussed
Spatial and temporal predicates can be combined: Find clips where the logo disappears and then a graph appears at the same place
A predicate can be applied to a part of document
As path expressions in OO databases

22. Uncertainty, proximity, weights
Similarity function
The user can assign importance weights to individual predicates in a complex query
This gives ranking, as in text IR
The same models can be used, e.g., probabilistic model

23. Examples of query languages: SQL3
Functions and stored procedures: user-defined data manipulation
Active database support: database reacts on the events, not only commands. This enforces integrity constraints
Good news: rather standard
Bad news: no ranking supported!
Effort to integrate SQL3 with IR techniques. SQL MM Full Text and other similar languages

24. ... examples: MULTOS One of design goals: easy navigation
Paths are supported
Identification of components by type, not by position
All images in the document, not the image in 3rd chapter
Types of predicates:
on data attributes, on textual components, on images (image type, objects contained, ...)
Example:

25. MULTOS example

26. Another example of MULTOS

27. Research topics How similarity function can be defined?
What features of images (video, sound) there are?
How to better specify the importance of individual features? (Give me similar houses: similar = size? color? strructure? Architectural style?)
How to determine the objects in an image?
Integration with DBMSs and SQL for fast access and rich semantics
Integration with XML
Ranking: by similarity, taking into account history, profile

28. Conclusions Basically, images are handled as text described them
Namely, feature vectors (or feature hierarchies)
Context can be used when available to determine features
Also, queries by example are common
From the point of view of DBMS, integration with IR and multimedia-specific techniques is needed
Object-oriented technology is adequate

29. Thank you!
Till ??, 6 pm