IST DIVAS Presentation 1 Advanced search technologies for digital audio-visual content

 Divas represents the combined efforts of eight companies and institutions to: › design and develop a multimedia search engine › based on advanced direct video and audio search algorithms applied directly on encoded (compressed) content.  IST DIVAS (FP6 IST ) was officially launched at the 1 st of January 2007, with a duration of 24 months. IST DIVAS Presentation 2

 Availability of huge and ever expanding distributed repositories of media in various formats  how can a system efficiently and reliably identify content fragments captured from various streams?  Only techniques for indexing and searching raw (uncompressed) content are available today (and text-based techniques) IST DIVAS Presentation 3

› provides the capability to the user to locate captured video and audio feeds with missing additional context  like title, filename, origin, location, service provider etc. › or in situation where metadata based queries are inapplicable IST DIVAS Presentation 4

› Search of multimedia libraries using the techniques for uncompressed content is a heavy duty/ costly solution  because for the search each item has to be decompressed. IST DIVAS Presentation 5

› Metadata annotation may be a heavy duty/costly solution to content owners. › A complementary solution should therefore also be made available IST DIVAS Presentation 6

 Audio-visual signature/ fingerprint extraction directly from compressed resources  Extend Search Techniques By supporting content queries, DIVAS extends the state of the art beyond nowadays pursued search techniques based on metadata.  Improve the reliability of audio-visual content detection By its multimodal (video & audio content) approach, and by combining the query results obtained from both modalities. IST DIVAS Presentation 7

8

 DIVAS proposes characterization, feature extraction and direct search of compressed video › as opposed to cognitive-level metadata annotation from uncompressed video streams.  “Video fingerprinting”, › as envisaged (but not extensively exploited) in the MPEG-7 standard is the term approximately fitting to our approach.  DIVAS will pursue: › Mpeg-2 compliant implementation › a H.264 compliant implementation. IST DIVAS Presentation 9

 Already a relatively mature technology on uncompressed audio  Based on the extraction of fingerprints, which capture the characteristic features of an audio clip. › These fingerprints are then compared to the fingerprint of a query (an audio clip to search for). IST DIVAS Presentation 10

 DIVAS system search techniques incorporate in parallel both audio and video based searching.  In terms of functional decomposition the system will address audio and video in a different way.  DIVAS system utilizes two different engines: › a/generate unique indexes from each clip › b/search among the aforementioned identifiers, providing a match/no match answer to the user. IST DIVAS Presentation 11

 Open architecture  Future-proof design  Scalability  Interoperability  Expandability  Modularity IST DIVAS Presentation 12

IST DIVAS Presentation 13

IST DIVAS Presentation 14

Compressed Audio Signal Direct conversion into the suitable time/frequency domain Feature Extraction Speech RecognitionMusic Information Retrieval Decoding Conversion to suitable time/frequency domain DIVASConventional

Tool “A” Content uploading Content index Tool “C” Administration Updating Tool “B” Content search Result of content search Indexes (fingerprints) DB Writing Reading DIVASENGINE

Multiplexed Content Multiplexed indexes Content demultiplexer Index multiplexer Video features extraction engine Audio features extraction engine Engine of text/meta features extraction Video/audio/text content Video/audio/text indexes Plug-ins See next slide

Video content Video Decoder (Transcoder) Features extractor Video index Plug-ins Plug-ins supporting video fingerprints Scene change plug-in Brightness change plug-in Frame content plug-in Plug-ins supporting video formats MPEG2 MPEG4 AVC/H.264 VC1 etc.

Query content CONTENT FEATURES EXTRACTION ENGINE COMPARISON ENGINE Query index Search result

Query index Search result Index reader Index comparer Plug-ins Indexes (fingerprints) DB Search result Searched index Read Query index Plug-ins for statistics comparison Plug-in for scene change statistics comparison Plug-in for brightness statistics comparison Plug-in for time characteristics statistics comparison Plug-in for average brightness and contrast statistics comparison

Query index Monitoring result Index comparer CONTENT FEATURES EXTRACTION ENGINE Content stream reader Plug-ins Content stream Monitored content Index Read Query index Plug-ins of content stream reading Plug-ins of read stream from capture devices Plug-ins of read stream from remote source