1. Video retrieval using inference network A.Graves, M. Lalmas In Sig IR 02

2. Outline  Background  Mpeg 7  Inference network model  Experiment  Conclusion

3. Background  CBVR: find the video upon demand  The semantic gap: high level concepts and low level features  Traditional method: retrieve by example  Relevance feedback: tedious for video retrieval  Retrieval by semantics?

4. Mpeg 7  Multimedia Content description interface  Attach metadata to multimedia content  Semantics: event, actor, place……  Structure: shot, scene, group……  Video as a structured document  The information contained in the Mpeg 7 annotation can be exploited to perform semantic based video retrieval  Mpeg 7 does not provide the solution to extract the annotation

5. Mpeg 7  Description definition language  Descriptors  Description schemes

6. Mpeg 7  Structured document and description    Basil attempts to mend the  car without success   Basil  Car  Mend  Carpark    ……  … …

7. Inference network  Perform a ranking given many sources of evidence  Document network (DN)  Constructed from the document data, represents all the retrievable units  Query network (QN)  Constructed from the query, represents the information needs

8. Inference network  The document network  Document layer (retrievable units collection)  Contextual layer (represents the contextual information about the document-concept links)  Concept layer (represent all the concepts in the network)

9. Inference network  Document network

10. Inference network  Link weight calculation  Structural: duration ratio (Between document nodes)  Contextual (Between contextual nodes) sibling number, context size, frequency  Context – concept (tf, idf)

11. Inference network  Query network  A framework of nodes that represents the information need  Concept nodes  Constraint operators (and, or, sum, not…), context-conceptual constraints

12. Inference network

13.  Attachment and Evaluation  Attachment: match the DN and QN, get a set of links between their nodes  All the constraint satisfied  Link inheritance: in the DN, document node can share the context nodes of its parent node  Evaluation: calculated quantized similarity for each document node in the DN

14. Inference network  Attachment  Link the QN and DN such that:  The concept nodes that contains same concepts (need a synonym dictionary) with weight 1 (firm)  For constraint queries, adjust the weight

15. Inference network  Attachment

16. Inference network  Evaluation  The evaluation process can be done toward all document nodes, it is calculated according to the query network  Can be used in different applications:  Retrieve scene, shot from one video  Retrieve video from video collections  …

17. Inference network  Evaluation  Back-propagate the QN-DN link weight back to each document nodes  All the node will have a value  Can derive a rank at different granularity level (Video, scene, shot)

18. Experiment  3 manually annotated video are employed as test data, totally 329 shots  Annotations:  for video shots and scenes  Avg. precision: 69.25  Similarity order are as expected

19. Experiment  The performance are quite dependent on the quality of the annotation data  Efficient annotation methods will be quite helpful

20. Conclusion  Propose a semantic video retrieval model based on Inference Network model, fully exploits the structural, conceptual, and contextual aspects of Mpeg 7.  Parry the semantic gap problem