MPEG-7 Video Retrieval using Bayesian Networks Luis M. de Campos Juan M. Fernández-Luna Juan F. Guadix Departamento de Ciencias de la Computación e Inteligencia Artificial E.T.S.I. Informática Universidad de Granada

MPEG-7 video retrieval using Bayesian Networks Introduction Brief overview about our work on the design of a search engine based on Bayesian Networks to retrieve MPEG-7 videos using their text annotations. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 video retrieval using Bayesian Networks Overview Preliminaries: introduction to Information Retrieval MPEG-7 standard Bayesian Networks MPEG-7 video retrieval models based on Bayesian networks. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (I) – Information Retrieval Information Retrieval is concerned with the representation, storage, organisation and accessing of information items. Indexing + Querying + Retrieval 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (II) – MPEG-7 Multimedia Content Description Interface Standard to describe multimedia content using metadata: The content of a multimedia file: concepts, objects in movement, who is speaking, ... Aspects related to the management of the content, i.e., duration, structure, format and size of the file, number of frames per shot,... Tools: Descriptors, Schemes, Data Definition Language. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (III) – MPEG-7 Descriptors: elements, data representation. (Time to represent a duration, histogram to represent a colour or a string to represent a title) Schemes: structure and semantic of the relationships among elements. (A film divided into scenes and shots, including textual description in the scene level and description about colour, movement and audio amplitude in the shot level) DDL (Data Definition Language): Language to extend or modify the previous set of tools. It is a variety of XML Schema. Therefore, descriptions files are XML files. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (IV) – MPEG-7 <?xml version="1.0" encoding="UTF-8"?> <Mpeg7> <Description xsi:type="ContentEntityType"> <MultimediaContent xsi:type="VideoType"> <Video id="1"> <MediaTime> <MediaTimePoint>T00:00:00:0F30000</MediaTimePoint> <MediaDuration>PT16M33S11772N30000F</MediaDuration> </MediaTime> <TemporalDecomposition gap="false" overlap="false"> <VideoSegment id="shot1_1"> <MediaDuration>PT3S22112N30000F</MediaDuration> <TextAnnotation confidence="0.500000"> <FreeTextAnnotation> A tv presenter is reporting information about a meeting of the security council in UN. </FreeTextAnnotation> </TextAnnotation> </VideoSegment> <VideoSegment id="shot1_2"> <MediaTime> <MediaTimePoint>T00:00:03:22112F30000</MediaTimePoint> <MediaDuration>PT9S18288N30000F</MediaDuration> </MediaTime> <TextAnnotation confidence="0.500000"> <FreeTextAnnotation> Collin Powell is speaking about the USA position in the Iraq crisis. </FreeTextAnnotation> </TextAnnotation> </VideoSegment> </TemporalDecomposition> </Video> </MultimediaContent> </Description> </Mpeg7> 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (V) – MPEG-7 From the point of view of IR, the structure of a video is seen conceptually: Vídeo Scene 1 Scene 2 Scene 3 Shot 1 Shot 2 Shot 3 Shot 4 Shot 5 Shot 6 Frame 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (VI) – Bayesian networks Graphical models able to represent and efficiently manipulate n-dimensional probability distributions. The knowledge obtained from a problem is encoded in a Belief network by means of the quantitative and qualitative componets: 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

Preliminaries (VII) – Bayesian networks Qualitative part: Directed Acyclic Graph G=(V,E): V (Nodes)  Random variables, and E (Arcs)  (In)dependence relationships. Quantitative part:A set of conditional distributions: Drawn from the graph structure, representing the strength of the relationships, stored in each node. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (I) Taking advantage of the structure of an MPEG-7 video: Video, Scenes, Shots, Frames And of free text annotation tags in the .xml file… 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (II) Sh1 Sh2 Sh4 Sh5 Sh6 Sh8 Sh3 Sh7 S1 S2 S3 V 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (III) Assesment of probability distributions: Prior probability in term nodes: p(ti)=1/M. Probability distributions in the rest of nodes: P(U | pa(U)). Problem: Great number of parents. Solution: Probability functions. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (IV) Query term instantiation. Run a propagation algorithm: p(u | Q),U. Generate a ranking. Problem: Great number of nodes in the graph. Complex topology. Solution: Evaluation of probability functions in each layer. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (V) In shots: In Scenes and Videos: where vij and wij: Exact propagation 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (VI) Once a relevance probability has been assigned to each unit, Which units are offered to the user? Those which present an accurate context, wider enaugh to be a good response to the query. How? Transforming the Bayesian Network into a Influence Diagram 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (VII) Sh4 Sh5 Sh6 D1 D3 D2 U2 U3 U1 S4 S3 D5 D4 V2 U5 U4 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (VIII) Integrated Tool: Video capture from tv. Automatic annotations form subtitles. Manual annotations based on ontologies. Querying and obtaining the best units. Automatic generation of a video with the results. 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

MPEG-7 Video Retrieval Models based on Bayesian Networks (IX) Lalmas and Graves´ model: V S1 S2 S3 Sh1 Sh2 Sh3 Sh4 Sh5 Sh6 bbc C2 C3 C4 C5 C7 C8 C9 C10 C11 dog MediaInformationDS MediaProfileDS MediaQualityDS CreationInformationDS MediaFormatDS Creation bbc dog and 29/11/2018 MPEG-7 video retrieval using Bayesian Networks

The end... Thank you very much