Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02

Outline  Background  Mpeg 7  Inference network model  Experiment  Conclusion

Background  CBVR: find the video upon demand  The semantic gap: high level concepts and low level features  Traditional method: retrieve by example  Relevance feedback: tedious for video retrieval  Retrieval by semantics?

Mpeg 7  Multimedia Content description interface  Attach metadata to multimedia content  Semantics: event, actor, place……  Structure: shot, scene, group……  Video as a structured document  The information contained in the Mpeg 7 annotation can be exploited to perform semantic based video retrieval  Mpeg 7 does not provide the solution to extract the annotation

Mpeg 7  Description definition language  Descriptors  Description schemes

Mpeg 7  Structured document and description    Basil attempts to mend the  car without success   Basil  Car  Mend  Carpark    ……  … …

Inference network  Perform a ranking given many sources of evidence  Document network (DN)  Constructed from the document data, represents all the retrievable units  Query network (QN)  Constructed from the query, represents the information needs

Inference network  The document network  Document layer (retrievable units collection)  Contextual layer (represents the contextual information about the document-concept links)  Concept layer (represent all the concepts in the network)

Inference network  Document network

Inference network  Link weight calculation  Structural: duration ratio (Between document nodes)  Contextual (Between contextual nodes) sibling number, context size, frequency  Context – concept (tf, idf)

Inference network  Query network  A framework of nodes that represents the information need  Concept nodes  Constraint operators (and, or, sum, not…), context-conceptual constraints

Inference network

 Attachment and Evaluation  Attachment: match the DN and QN, get a set of links between their nodes  All the constraint satisfied  Link inheritance: in the DN, document node can share the context nodes of its parent node  Evaluation: calculated quantized similarity for each document node in the DN

Inference network  Attachment  Link the QN and DN such that:  The concept nodes that contains same concepts (need a synonym dictionary) with weight 1 (firm)  For constraint queries, adjust the weight

Inference network  Attachment

Inference network  Evaluation  The evaluation process can be done toward all document nodes, it is calculated according to the query network  Can be used in different applications:  Retrieve scene, shot from one video  Retrieve video from video collections  …

Inference network  Evaluation  Back-propagate the QN-DN link weight back to each document nodes  All the node will have a value  Can derive a rank at different granularity level (Video, scene, shot)

Experiment  3 manually annotated video are employed as test data, totally 329 shots  Annotations:  for video shots and scenes  Avg. precision:  Similarity order are as expected

Experiment  The performance are quite dependent on the quality of the annotation data  Efficient annotation methods will be quite helpful

Conclusion  Propose a semantic video retrieval model based on Inference Network model, fully exploits the structural, conceptual, and contextual aspects of Mpeg 7.  Parry the semantic gap problem