Prof. Thomas Sikora Technische Universität Berlin Communication Systems Group Thursday, 2 April 2009 Integration Activities in “Tools for Tag Generation“

Core partners: TUB (coord.), EPFL, TUD, QMUL Aims: Generation of tags and metadata (signal processing and/or users’ annotation) Keyframe extraction for image search engines Video classification, clustering and summarization Propagation of tags (object detection/ similarity) 2 Goals

Integration Activities in “Tools for Tag Generation“ Preparation: Setup database from unstructured channel “Travel” on YouTube.com –100 videos + affiliated metadata (keywords, comments, user information etc.) –Annotation of shot boundaries (by TUD and TUB) 3 Database

Integration Activities in “Tools for Tag Generation“ 4 Youtube Downloader Tool also used by the IRP “Social Media Data Collection“

Integration Activities in “Tools for Tag Generation“ 5 Youtube Downloader Information also interesting for IRP “Social Media Data Collection“

Integration Activities in “Tools for Tag Generation“ -Integration of different expertise: -Image search engines, audio signal processing, video signal processing and text detection/ recognition 6 Overview

Integration Activities in “Tools for Tag Generation“ 7 Overview Shot Subshot Key Frame Video stream: 1.Temporal segmentation Detection of hard cut, fade and dissolve 2.Subshot boundary detection Special importance for single shot videos and for long shots Detection of new visual content appearing through camera operations or undetected transitions. 3.Extraction of key frames efficient key frame extraction as a handshake between existing image search engines and video domain

Integration Activities in “Tools for Tag Generation“ 8 Key Frame Extraction

Integration Activities in “Tools for Tag Generation“ 9 Future Work

Integration Activities in “Tools for Tag Generation“ 10 Future Work Key framing [TUB] + Object Replica Detection [EPFL] + Reranking [TUD] + others

Integration Activities in “Tools for Tag Generation“ 11 Future Work Audio-Visual Source Localization in Videos [EPFL] localization can be used for automatic tag generation/propagation

Integration Activities in “Tools for Tag Generation“ 12 Future Work Quality Tags [EPFL] on Key Frames [TUB] Key frames are used to produce quality tag by no reference video quality assessment.

Integration Activities in “Tools for Tag Generation“ 13 Future Work Multimodal Video Reranking [TU Delft] Task: Retrieval of shots treating a particular topic (i.e., a semantic theme) Already done: Search over speech transcripts is improved by pseudo- relevance feedback. Selection of feedback documents is informed by visual features. Future Work: Expand reranking ideas developed on VideoCLEF 2008 dataset to YouTube-like video collections. Reranked results list

Integration Activities in “Tools for Tag Generation“ 14 Conclusion IRP continues until end May 2009 So far very good research progress – new tools Good integration of partners Tools of much interest and use for other IRP‘s Key framing will be finalized end of May Clustering will continue in new IRP Demos: