102-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures Overview French Consortium SE Demonstrator MAM Demonstrator French Partners Presentation

202-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures Overview French Consortium SE Demonstrator MAM Demonstrator French Partners Presentation

302-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA  General goals: Intelligent video delivery applications Development of innovative content analysis technology  Involved technologies Computer vision Networks, storage

402-OCT-02CANDELA presentation STSIConfidential information/proprietary Consortium: 35 Partners Project leader WP leader French  Industry: Bosch CSI NL Philips SLE NL LuTech I Philips Philips PDSL NL Philips Res. NL Siemens D Thomson F Thales F Telecom Italia I Ibermatica E TXT I  263,5 MY in 7 countries: Belgium (31), Finland (46), France (49), Germany (36), Italy (35,5), Netherlands (40), Spain (23)  Universities: U Paderborn D TU Berlin D Free U Brussels B KU Leuven B TU Eindh. EESI NL Cefriel I ESI E INRIA F VTT Fi Multitel B  SME: Solid Fi Hantro Fi capVidea B Ciaolab I IT Optic B MicroGenesis E RheaSystem B Retevision Movil E AlgoVision/ LuraTec D Empolis D TecMath D Vitec F Quadrox B Vartec B

502-OCT-02CANDELA presentation STSIConfidential information/proprietary Costs: 33,8 Mio EUR Belgium 11% Germany 14% Spain 7% France 19% Finland 19% Italy 15% Netherlands 15%

602-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Structure Video server Terminal meta data base Content Analysis Network WP3: Architectures WP1: Content AnalysisWP2: Networked Delivery

702-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Structure Content analysis Coding Integration Platform Application Validation Requirements Delivery Security WP1 WP2 WP3 Storage Retrieval

802-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Working Packages  WP0: Project management WP leader: Bosch CSI (NE), Paul Merkus  WP1: Content Analysis WP leader: Philips SLE (NE), Ronald Begeer  WP2: Networked Delivery WP leader: Solid (SME, Fi), Pauli Berg  WP3: Architectures WP leader: LuTech (SME, I), Simona Costa

902-OCT-02CANDELA presentation STSIConfidential information/proprietary WP 1: Content Analysis  Object tracking (TMM, F) Object-based metadata generation  Feature extraction (CapVidea, B) Metadata generation from video  Video structure (Vitec, F) Metadata generation on temporal video structure  Semantic metadata (INRIA, F) High-level metadata Metadata representation language  Semantic video encoding (VUB, B) metadata-driven video encoding

1002-OCT-02CANDELA presentation STSIConfidential information/proprietary WP 2: Networked Delivery  Efficient video storage and retrieval technology (Philips Research, NL) User storage, provider storage, distributed storage Scalable and reliable storage Database retrieval by mobile user, by provider  Intelligent video delivery technologies (TU Eindhoven, NL) Adaptive video delivery Profiled delivery Video delivery technologies  Video network delivery (Thales, F) Video Network Delivery architecture Video delivery and data management Adaptive and Intelligent video services

1102-OCT-02CANDELA presentation STSIConfidential information/proprietary WP 3: Architectures  Applications (TXT, I) Application requirements Demonstrator specifications  Platform & security (Cefriel, E) middleware standard compliant implementations  Methodology (ESI, S)  Integration & validation (Lutech, I) Standard compliance Test, demo  Exploitation, operation (Rhea, B)

1202-OCT-02CANDELA presentation STSIConfidential information/proprietary Deliverables 2004 Application Spec Progress report Integration Final Demonstrator Technology Demonstrators Technologies Progress report 2005 Technology Spec. Platform

1302-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures Overview French Consortium SE Demonstrator MAM Demonstrator French Partners Presentation

1402-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA: French Consortium  Total French effort: 49 MY, 6.5 Mio EUR INRIA Thales 00WP 2 Networked Delivery 22WP 3 Architectures 714WP 1 Content Analysis VITECTMMMY Total Total 49 MY

1502-OCT-02CANDELA presentation STSIConfidential information/proprietary French Consortium  May 2003:40 MY (Thales, TMM)  June 2003:36 MY (Thales, TMM)  July 2003:56 MY (Thales, TMM, VITEC, INRIA)  August 2003:49 MY (Thales, TMM, VITEC, INRIA) Reduction Equilibration Reduction

1602-OCT-02CANDELA presentation STSIConfidential information/proprietary MAM Demonstrator SE Demonstrator French Demonstrators Terminal Video server meta data base Content Analysis Network

1702-OCT-02CANDELA presentation STSIConfidential information/proprietary French Demonstrators TMMThalesINRIAVITEC MAM Mutimedia Content Management SE Smart Encoding MAM SE MAM SE  New partner for MAM: content provider?

1802-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures Overview French Consortium SE Demonstrator MAM demonstrator French partners presentation

1902-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Demonstrator

2002-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Demonstrator Context  Video-surveillance application Requires video coding scheme adapted to the video-surveillance needs – Video coders have been developed mainly for broadcast like applications  Static details are privileged  Moving objects are coded with reduced quality and resolution – In contradiction with video-surveillance requirements where the best quality and resolution has to be ensure for mobile objects. Need of enhanced network delivery to cope with video-surveillance network constraints –Heterogeneous secured networks –QoS –Fast and easy/automatic reconfiguration  This demonstrator results from WP1, WP2 and WP3 works

2102-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Issues  Smart video encoding Region of interest adapted video coding – Object tracking and scene identification  To segment and track moving objects in video scenes such as persons.  The information on moving objects will provide new type of semantic metadata for multimedia asset management (MAM)  Metadata on moving objects will allow object-based bandwidth and system resource allocation in video surveillance and mobile applications. – Evolution and adaptation of existing video coding schemes (MPEG-4, H.264 ….)  To exploit high-level information extracted from the audiovisual content analysis to drive the subsequent lossy compression mechanisms  Target solutions that are compliant with standard compressed video stream formats, (MotionJPEG 2000, MPEG and/or H26x …), to ensure a widespread dissemination of the project’s results.

2202-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Region of Interest Adapted Video Coding Mobile object detection & tracking Video encoding ROI driven better quality Background lower quality Metadata Coded video  Merging Region of interest adapted video coding Object detection and tracking

2302-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Networking Issues  The necessity of delivering with good quality different video content on different type of network implies addressing : Real time networking Tight traffic control Optimisation of the link utilisation  The need to deliver information in a secure way means addressing the following security issues: Confidentiality, integrity, authentication, protection against replay, access control  Necessity for reconfiguration and reactivity implies Simple and homogeneous management of multiple network services Dynamic network management (security, QoS)

2402-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Issues Core Network QoS &Security management Reconfigurability management

2502-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Tasks & Partners Involved  Object-based metadata generation (WP1 Task 1.1, 1.2 and 1.4) Event detection (Scenes identification and object tracking) French partners involved –TMM –THALES –INRIA  Metadata-driven video encoding (WP1 Task 1.5) Evolution and adaptation of existing encoding schemes –MPEG-4/H.264 –MotionJPEG 2000 – ….. French partners involved –VITEC –THALES  Other european partners involved VUB, Multitel, AlgoVision...

2602-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Tasks & Partners Involved  Video delivery network management (WP2 Task 2.2) Qos and security network architecture Qos and security Policy based management  Intelligent video services (WP2 Task 2.3) End to end QoS Dynamic security routing  European partners involved VUB, Solid, TUE, Ciaolab, Philips NL, Telecom Italia,...

2702-OCT-02CANDELA presentation STSIConfidential information/proprietary Smart Encoding (SE) Expected Results  Contributions to Standards MPEG-4/H.264, MotionJPEG 2000 –Additional Guidelines Annexes –Amendements Following IETF recommendations (draft, FRC)  Technology Merge of two different/separate technological domains (image coding & Object detection/tracking ). Enhanced network cope with video-surveillance constraints  Products VITEC –Know How for product line evolutions THALES –New functionalities for video surveillance systems

2802-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures Overview French Consortium SE Demonstrator MAM demonstrator French partners presentation

2902-OCT-02CANDELA presentation STSIConfidential information/proprietary Multimedia Asset Management (MAM) Demonstrator

3002-OCT-02CANDELA presentation STSIConfidential information/proprietary Multimedia Asset Management (MAM) Repeater Editing Ingestion Playout serverVideo server Low quality video server Annotation terminal meta data base playout Internet Encoder Logging Ingestion Creation Retrieval

3102-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA MAM Workflow  Ingest Development Digitizing Automatic technical metadata (date, time, format Incoming content: –Dailies from film production –Live events –reused content  Logging Manual Annotation  Creation Editing, cutting Color correction  Archiving/Retrieval Storage Ingestion Video server Annotation terminal meta data base Logging Ingestion Retrieval Editing Creation

3202-OCT-02CANDELA presentation STSIConfidential information/proprietary MAM Issues  Automatic metadata very limited  Time costly manual annotation  Inefficient re-use of stored content  Need of cost-efficient metadata generation  Need of semantic metadata to enable content re-use

3302-OCT-02CANDELA presentation STSIConfidential information/proprietary New MAM Technologies  Intelligent ingest: Automatic generation of semantic-level metadata Temporal video summary  video chapters Semantic categories  keywords Description language  interpretation  Object-based logging: Semi-automatic object-based annotations Object segmentation  object-based annotations Object recognition  keywords  Task 1.3 Video structure (VITEC)  Task 1.2 Object description (INRIA)  Task 1.4 Semantic Metadata (INRIA)  Task 1.1 Object tracking (TMM)

3402-OCT-02CANDELA presentation STSIConfidential information/proprietary Video Structure Generation unstructured video Shot detection Keyframe extraction Classification Scene detection keyframes Video chapters

3502-OCT-02CANDELA presentation STSIConfidential information/proprietary Object Tracking Image region tracking Object annotation Object detection video frame Annotations: horse

3602-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures Overview French Consortium SE Demonstrator MAM demonstrator French partners presentation

3702-OCT-02CANDELA presentation STSIConfidential information/proprietary Thomson Multimedia Contribution

3802-OCT-02CANDELA presentation STSIConfidential information/proprietary … through content distribution… From content Production… Video Image Chain > Movies: Production of distribution copies, marketing support, digital movie systems > Video: Production and delivery of VHS and DVD packaged media; integrated services > Networks: Professional video equipment for satellite, cable, telecommunications and ground networks; associated services > Receiver products: TV sets, VCRs, DVDs, audio systems, decoders, etc. > Interactive services: Electronic program guides, « back office » services for broadcasters / advertisers, interactive TV > Components: Flat screens, high-definition, plasma, optical components, electronics, etc. Thomson all along the Video Chain > Development of film negatives > Post-production services: image processing (digitalization, colorization, audio and video montage, etc.) > Management of digital content: indexing, compressing, formatting, archiving, routing images > Studio services and equipment: cameras, integrated studio video servers, etc. … until content access Group businesses involved: SBU Digital Media Solutions SBU Patents and Licensing Research and Innovation Group businesses involved: SBU Broadband Access Products and Systems SBU New Media Services SBU Patents and Licensing Research and Innovation Group businesses involved: SBU TV, Accessories and Aftersales SBU Audio, Video & ATLINKS SBU Displays and Components SBU Patents and Licensing SBU New Media Services Research and Innovation

3902-OCT-02CANDELA presentation STSIConfidential information/proprietary Thomson Goals in CANDELA  Goals Acquire technologies for automatic metadata generation Exploit intra-image information (object segmentation) Rise the semantic level of metadata  Technologies Image classification (e.g. sports, indoor, family) Object tracking (e.g. cars and faces) Object recognition (e.g. faces)  Aimed applications: MAM Ingest with automatic semantic annotation Object tracking for semi-automated logging Semantic browsing of multimedia assets

4002-OCT-02CANDELA presentation STSIConfidential information/proprietary TMM Contribution  WP1 Content Analysis Task 1.1 Object Tracking (task leader)10 MY Task 1.3 Video Structure4 MY  WP3 Architecture Task 3.1 Applications1.5 MY Task 3.4 Integration & Validation0.5 MY 16 MY 2200 kEUR

4102-OCT-02CANDELA presentation STSIConfidential information/proprietary TMM in Task 1.1 Object Tracking  Task leadership State-of-the-art report Requirements Object tracking for simple scenes Object tracking for complex scenes Demonstration, report

4202-OCT-02CANDELA presentation STSIConfidential information/proprietary TMM in Task 1.1 Object Tracking  Object Tracking Simple Scenes Tracking of object silhouettes  Object Tracking Complex Scenes Motion-based object tracking (general objects) Pattern-based object tracking (faces, logos) Combined tracking (persons, cars)  Innovation Domain specific knowledge Combination of motion (image analysis) and pattern clues (statistics) video frame Annotations: horse

4302-OCT-02CANDELA presentation STSIConfidential information/proprietary TMM in Task 1.3 Video Structure  Video structure  Problem: Semantic gap User: Show me the whole forest sequence! User metric: Semantic similarity MAM System: Here are all green frames System metric: Visual similarity video keyframes Video chapters

4402-OCT-02CANDELA presentation STSIConfidential information/proprietary TMM in Task 1.3 Video Structure  Innovation: Semantic classification of video scenes OUTDOOR INDOOR

4502-OCT-02CANDELA presentation STSIConfidential information/proprietary TMM in WP3 Architectures  Task 1.1 Applications MAM application specification MAM application requirements  Task 3.4 Integration & validation MAM demonstrator integration  TMM skills Business units in professional video Customers in content creation and archiving

4602-OCT-02CANDELA presentation STSIConfidential information/proprietary Thales Contribution

4702-OCT-02CANDELA presentation STSIConfidential information/proprietary THALES Presentation  Thales Corp. Leading European & worldwide positions in defence and aerospace electronics Leading-edge technologies in IT&S for high-tech B2B markets 3 business Areas: Defense Aerospace Information Technology & Services  THALES Communications We design and supply global information and communications systems solutions Seamless Secured Standard and interoperable Thales Security &System: Design, development and turnkey installation of integrated security systems dedicated to the protection of sensitive civil and military sites High-tech systems integrator and prime contractor for large-scale security projects 20 years of extensive experience and technical expertise in electronic security (access control, video surveillance, intrusion detection, perimeter protection, global supervision

4802-OCT-02CANDELA presentation STSIConfidential information/proprietary THALES Contribution : Goals & Points of Interest  Thales Goals in CANDELA Goals Acquire technologies for Smart Video encoding Exploit standardized Technologies (Video &Network) for specific uses Technologies Object tracking Object recognition (e.g. person) Content delivery (Network) Aimed applications: videosurveillance...  Thales Points of Interest: WP1 : Content analysis Content Analysis for Event detection and Flexible compression WP2 : Content Delivery Network secured delivery and QoS Management WP3 : Architecture Definition of user requirements (for video surveillance) Definition of the network architecture

4902-OCT-02CANDELA presentation STSIConfidential information/proprietary THALES Contribution WP1 Content Analysis  Focus on Smart Encoding Task 1.1 (Object tracking) –State of the art –Requirements –Object tracking in simple scenes (specially in out door condition) –Participation in object tracking in complex scenes –Demonstration Task 1.5 (Semantic video encoding) –State of the art –Smart encoding –Flexible lossy compression –Scalable encoding –RC Region Of Interest (ROI) –Demonstration

5002-OCT-02CANDELA presentation STSIConfidential information/proprietary THALES Contribution WP2 Content Delivery  Focus on network management Task (Video Network delivery management) –State of the art report concerning QoS and security on network side and Policy based network management for QoS –Participation to the definition of the Multimedia network reference architecture for QoS and security –QoS and Security Policy based management –Deployment of this architecture in a research platform –Test with WP1 streaming applications Task (Intelligent Video Delivery Technologies) –Develop new mechanisms management for providing end to end QoS –Develop new mechanisms to assure the mobile user to benefit of permanent security –Participation to the “intelligent delivery solutions” report (presentation of the QoS and security solutions) –Deployment of those solutions in a research platform

5102-OCT-02CANDELA presentation STSIConfidential information/proprietary Multimedia Receiver => Bad Reception Network Load THALES Contribution WP 2 QoS Themes Classification in the edge (EF) PHB in core routers => Good reception Step 2 : dynamic management Step 3 : RSVP

5202-OCT-02CANDELA presentation STSIConfidential information/proprietary THALES Contribution WP 2 Security Themes PKI : Clear text : Encrypted text Authentication Encryption Tunnel IKE Session Negotiation of the security parameters Exchange of the certificates PKI : Public Key Infrastructure PM : Policy Manager AD : Access Device PMPM PMPM Step 2: Dynamic management Step 3 : Dynamic tunnel rebuild in case of mobility

5302-OCT-02CANDELA presentation STSIConfidential information/proprietary THALES Contribution WP3 Architecture  Application definition and user requirement analysis (in the context of video surveillance)  Definition of the platform (in term of network architecture)  Integration and validation of a video surveillance demonstrator  Participation to the security task (by bringing the results obtained in the WP2 security task)

5402-OCT-02CANDELA presentation STSIConfidential information/proprietary VITEC Multimedia Contribution

5502-OCT-02CANDELA presentation STSIConfidential information/proprietary VITEC Presentation VITEC Multimedia  Company established in 1988  Independent company  Revenues: 3M€  Capital: 1,5M€  25 people

5602-OCT-02CANDELA presentation STSIConfidential information/proprietary Market of VITEC Multimedia Digital video Video ComputersTelecom

5702-OCT-02CANDELA presentation STSIConfidential information/proprietary VITEC Objectives in CANDELA Productivity tools for mass video image databases (MAM & SE): - image content extraction and classification (content editing) - video structuring and data reduction (scalable content delivery) - sequence identification and retrieval (content query) - MPEG encoding enhancement (content-driven coding)

5802-OCT-02CANDELA presentation STSIConfidential information/proprietary WP1 VITEC Contribution (1) Task 1.3: Video Structure (Task Leader: 3.50 Mys) - Automatic chapter generation : -Scene identification ( video shot grouping) -Chapter generation (story-telling principles) - Video data reduction: -Video summarization -Video digest generation

5902-OCT-02CANDELA presentation STSIConfidential information/proprietary WP1 VITEC Contribution (2) Task 1.4: Semantic description (1.50 Mys) Image sequence segmentation and indexing based on content extraction and classification Task 1.5: Semantic video encoding (2.00 Mys) MPEG encoding enhancement: image sequence coding using a compression rate driven by image content

6002-OCT-02CANDELA presentation STSIConfidential information/proprietary WP3 VITEC Contribution Task 3.1: Requirements and Overall Application (1.00 Mys) - user’s requirements gathering from one’s own customers base - design of architecture and video processing modules Task 3.4: Integration and Validation (1.00 Mys) - video structuring and semantic description software at MAM demonstration disposal - participating to integration and validation of SE demonstrator

6102-OCT-02CANDELA presentation STSIConfidential information/proprietary INRIA Contribution

6202-OCT-02CANDELA presentation STSIConfidential information/proprietary ORION team: Research directions Intelligent Reusable Systems for Cognitive Vision Intelligent: explicit knowledge, reasoning and learning capabilities Reusable Systems: for different applications or problems Cognitive Vision: image understanding Multidisciplinary team: artificial intelligence, software engineering, computer vision INRIA Activities Monique THONNAT Sophia Antipolis

6302-OCT-02CANDELA presentation STSIConfidential information/proprietary  Objectives: real time and automated analysis of video sequences video understanding = from people detection and tracking to behavior recognition  Examples: recognition of bank agencies scenes for visual surveillance recognition of metro scenes for visual surveillance INRIA Video Understanding

6402-OCT-02CANDELA presentation STSIConfidential information/proprietary INRIA Video Understanding Our goal: to model the interpretation process of video sequences from pixel up to behaviour. time.. Recognised scenario Video stream Mobile object detection & tracking “ Vandalism ?” “Two people are fighting ?” Scenario recognition

6502-OCT-02CANDELA presentation STSIConfidential information/proprietary Behaviour recognition: detection of an agitated group in metro INRIA Video Understanding lively Agitated behaviour

6602-OCT-02CANDELA presentation STSIConfidential information/proprietary  Behaviour recognition: approach based on a priori knowledge model of the empty scene (3D geometry and semantics) models of predefined scenarios a language for representing scenarios based on combination of states and events 10 states and 21 events can be used a reasoning mechanism for real time detection of states, events and scenarios (e.g. temporal reasoning, constraints solving techniques) INRIA Video Understanding

6702-OCT-02CANDELA presentation STSIConfidential information/proprietary INRIA Video Understanding n Description of a bank attack scenario Scenario(Attack, Characters((cashier : Person), (robber : Person)) Constraints((exists ( state(s1, inside_zone, cashier, "Back_Branch") event(e, changes_zone, robber,"Gate","Infront_Branch") state(s2, inside_zone, cashier, "Safe_zone") state(s3, inside_zone, robber, "Safe_zone")) ((s1 before s2) (s1 before s3) (e before s3) ) ) ) Production((name of sc : Scenario Attack) (Start of sc := Start of s1) (End of sc = End of s2)))

6802-OCT-02CANDELA presentation STSIConfidential information/proprietary INRIA Video Understanding The recognition of the scenario “Attack” in a bank agency

6902-OCT-02CANDELA presentation STSIConfidential information/proprietary  Description of a vandalism scenario. Scenario(vandalism_against_ticket_machine, Characters ((p : Person), (eq : Equipment, Name = “Ticket_Machine”) ) Constraints((exist ((events1: p move_close_to eq) (state s2: p stay_at eq) (event s3: p move_away_from eq) (event s4: p move_close_to eq) (state s5: p stay_at eq) ) (s1 before s2) (s2 before s3) (s3 before s4) (s4 before s5) ) ) ) Production( (sc : Scenario) ( (Name of sc := "vandalism_against_ticket_machine") (StartTime of sc := StartTime of s1) (EndTime of sc := EndTime of s5) ) ) ) INRIA Video Understanding

7002-OCT-02CANDELA presentation STSIConfidential information/proprietary INRIA Video Understanding

7102-OCT-02CANDELA presentation STSIConfidential information/proprietary INRIA Activities  INRIA will participate to WP1 on video content analysis task 1.1 Object Tracking 2 mys task 1.2 Feature Extraction/Object Description 1 my task 1.4 Semantic Metadata (task leader) 3 mys  INRIA activities in SE demonstrator to detect interesting events and regions of interest for smart encoding using intelligent video understanding techniques for fixed visual surveillance (task 1.1, 1.2 and 1.4)  INRIA activities in MAM demonstrator to help metadata generation (logging) by: + proposing a standard vocabulary for video annotation (task 1.4) + attempt automating video description (task 1.2 and 1.4)

7202-OCT-02CANDELA presentation STSIConfidential information/proprietary April 3/ kick-off meeting INRIA Sophia Antipolis