1. MPEG-7 Video Retrieval using Bayesian Networks
Luis M. de Campos
Juan M. Fernández-Luna
Juan F. Guadix
Departamento de Ciencias de la Computación e Inteligencia Artificial E.T.S.I. Informática
Universidad de Granada

2. MPEG-7 video retrieval using Bayesian Networks
Introduction
Brief overview about our work on the design of a
search engine based on Bayesian Networks
to retrieve MPEG-7 videos
using their text annotations.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

3. MPEG-7 video retrieval using Bayesian Networks
Overview
Preliminaries: introduction to
Information Retrieval
MPEG-7 standard
Bayesian Networks
MPEG-7 video retrieval models based on Bayesian networks.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

4. Preliminaries (I) – Information Retrieval
Information Retrieval is concerned with the representation, storage, organisation and accessing of information items.
Indexing + Querying + Retrieval
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

5. Preliminaries (II) – MPEG-7
Multimedia Content Description Interface
Standard to describe multimedia content using metadata:
The content of a multimedia file: concepts, objects in movement, who is speaking, ...
Aspects related to the management of the content, i.e., duration, structure, format and size of the file, number of frames per shot,...
Tools:
Descriptors, Schemes, Data Definition Language.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

6. Preliminaries (III) – MPEG-7
Descriptors: elements, data representation.
(Time to represent a duration, histogram to represent a colour or a string to represent a title)
Schemes: structure and semantic of the relationships among elements.
(A film divided into scenes and shots, including textual description in the scene level and description about colour, movement and audio amplitude in the shot level)
DDL (Data Definition Language): Language to extend or modify the previous set of tools. It is a variety of XML Schema. Therefore, descriptions files are XML files.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

7. Preliminaries (IV) – MPEG-7
<?xml version="1.0" encoding="UTF-8"?>
<Mpeg7>
<Description xsi:type="ContentEntityType">
<MultimediaContent xsi:type="VideoType">
<Video id="1">
<MediaTime>
<MediaTimePoint>T00:00:00:0F30000</MediaTimePoint>
<MediaDuration>PT16M33S11772N30000F</MediaDuration>
</MediaTime>
<TemporalDecomposition gap="false" overlap="false">
<VideoSegment id="shot1_1">
<MediaDuration>PT3S22112N30000F</MediaDuration>
<TextAnnotation confidence=" ">
<FreeTextAnnotation> A tv presenter is reporting information about a meeting of the security council in UN. </FreeTextAnnotation>
</TextAnnotation>
</VideoSegment>
<VideoSegment id="shot1_2">
<MediaTime>
<MediaTimePoint>T00:00:03:22112F30000</MediaTimePoint>
<MediaDuration>PT9S18288N30000F</MediaDuration>
</MediaTime>
<TextAnnotation confidence=" ">
<FreeTextAnnotation>
Collin Powell is speaking about the USA position
in the Iraq crisis.
</FreeTextAnnotation>
</TextAnnotation>
</VideoSegment>
</TemporalDecomposition>
</Video>
</MultimediaContent>
</Description>
</Mpeg7>
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

8. Preliminaries (V) – MPEG-7
From the point of view of IR, the structure of a video is seen conceptually:
Vídeo
Scene 1
Scene 2
Scene 3
Shot 1
Shot 2
Shot 3
Shot 4
Shot 5
Shot 6
Frame
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

9. Preliminaries (VI) – Bayesian networks
Graphical models able to represent and efficiently manipulate n-dimensional probability distributions.
The knowledge obtained from a problem is encoded in a Belief network by means of the quantitative and qualitative componets:
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

10. Preliminaries (VII) – Bayesian networks
Qualitative part: Directed Acyclic Graph G=(V,E):
V (Nodes)  Random variables, and
E (Arcs)  (In)dependence relationships.
Quantitative part:A set of conditional distributions:
Drawn from the graph structure,
representing the strength of the relationships,
stored in each node.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

11. MPEG-7 Video Retrieval Models based on Bayesian Networks (I)
Taking advantage of the structure of an MPEG-7 video:
Video, Scenes, Shots, Frames
And of free text annotation tags in the .xml file…
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

12. MPEG-7 Video Retrieval Models based on Bayesian Networks (II)
Sh1
Sh2
Sh4
Sh5
Sh6
Sh8
Sh3
Sh7 S1 S2 S3 V
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

13. MPEG-7 Video Retrieval Models based on Bayesian Networks (III)
Assesment of probability distributions:
Prior probability in term nodes: p(ti)=1/M.
Probability distributions in the rest of nodes: P(U | pa(U)).
Problem: Great number of parents.
Solution: Probability functions.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

14. MPEG-7 Video Retrieval Models based on Bayesian Networks (IV)
Query term instantiation.
Run a propagation algorithm: p(u | Q),U.
Generate a ranking.
Problem:
Great number of nodes in the graph.
Complex topology.
Solution:
Evaluation of probability functions in each layer.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

15. MPEG-7 Video Retrieval Models based on Bayesian Networks (V)
In shots:
In Scenes and Videos:
where vij and wij:
Exact propagation
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

16. MPEG-7 Video Retrieval Models based on Bayesian Networks (VI)
Once a relevance probability has been assigned to each unit,
Which units are offered to the user?
Those which present an accurate context, wider enaugh to be a good response to the query.
How?
Transforming the Bayesian Network into a
Influence Diagram
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

17. MPEG-7 Video Retrieval Models based on Bayesian Networks (VII)
Sh4
Sh5
Sh6 D1 D3 D2 U2 U3 U1 S4 S3 D5 D4 V2 U5 U4
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

18. MPEG-7 Video Retrieval Models based on Bayesian Networks (VIII)
Integrated Tool:
Video capture from tv.
Automatic annotations form subtitles.
Manual annotations based on ontologies.
Querying and obtaining the best units.
Automatic generation of a video with the results.
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

19. MPEG-7 Video Retrieval Models based on Bayesian Networks (IX)
Lalmas and Graves´ model: V S1 S2 S3
Sh1
Sh2
Sh3
Sh4
Sh5
Sh6
bbc C2 C3 C4 C5 C7 C8 C9
C10
C11
dog
MediaInformationDS
MediaProfileDS
MediaQualityDS
CreationInformationDS
MediaFormatDS
Creation
bbc
dog
and
29/11/2018
MPEG-7 video retrieval using Bayesian Networks

20. The end...
Thank you very much