Technologies: for Enhancing Broadcast Programmes with Bridgets Karsten Müller Fraunhofer HHI 1st BRIDGET Workshop Paris – 16th December 2013

Overview “Bridgetised” Broadcast Broadcast Media databases Analysis, Search, Recognition AR Authoring Media databases (archives, internet servers, social networks, etc) Mobile Terminal 3D Modelling System Architecture, Interfaces

Visual Search, Recognition and Media Analysis Reliable and fast suggestion of content bridges Recognising objects, buildings and scenes selected by the user Instantaneous matching with available images, video, and other content from a broadcaster or external databases Video-to-video matching, based on the improved local descriptors and description aggregation (as employed in MPEG’s Video Signature tools) Modifications and improvements to selected elements of the processing pipeline: Local descriptors (targeting binary resulting in ultra-fast matching), Hierarchical scoring strategies Descriptor aggregation techniques Novel methods of geometric verification

3D Scene Modelling, Coding and Reconstruction Coarse 3D scene model creation from available monoscopic, stereoscopic and 3D images, video material, and 3D Computer Graphics (CG) data in a broadcasters database Continuous model refinement Newly available material from other sources, like the web Uploaded content from individuals or user groups, e.g. images taken from other viewpoints. Addition of CG 3D models, contributed by professional artists or end users, whenever available Newly developed VS tools and media descriptors, for extracting scene calibration information from images and videos Device-dependent view synthesis, viewpoint selection and baseline adaptation for stereo playout Visual compression adaptation for authoring server and user terminal

Augmented Reality Authoring Integration of navigation information into temporal video structure Association of identification descriptors to user-defined spatial regions, e.g. for possible reconstruction of multi-view videos, 3D objects and scenes. Management of descriptor dictionaries (equivalent of a large index where (binary) descriptors will be associated with references to media from private and open libraries containing the same object) Cloud-friendly Authoring Tool (AT) with exploitation of the current physical infrastructure of IMT’s MyMultimediaWorld (www.MyMultimediaWorld.com) platform for establishing augmentation areas by using natural markers (images, videos, etc.) that will be recognised in real time, and building the behaviour of the application by using pre-programmed scripts. New formalism for representing the scene, the assets, the user interaction, the connection with the sensors and the access to remote resources.

Mobile Terminal Creation of high quality 3D models of the scenes for optimising video-plus-depth rendering and view synthesis for content and baseline retargeting. Lightweight and real-time methods for retargeting, that do not require content post processing to be adapted to the terminal viewing set-up for cases, where high quality models cannot be computed (e.g. where very limited views of a scene are available) Sparse features and saliency detection for most characteristic scene objects Hierarchical grouping approach for points with similar colour and depth for bigger “splat” rendering during the visualisation phase 3D audio rendering over small speakers for mobile terminals

System Architectures and Interfaces Mechanisms for hybrid services on the main screen towards second screens in the home network (e.g., tablets and smart phones) Usually not under direct control of broadcasters or service providers,  Design of according signalling and content retrieval processes Investigation of video object recognition as an alternative with less requirements and constraints in terms of device functions exposed to the applications (e.g., a UPnP SW stack), Provision of a reference architecture to support second screen services in an open, component-based way