1.  2023! ITEA CANDELA Content Analysis and Networked DELivery Architectures International Review Helsinki, Finland June 16, 2005 Home Multimedia Demonstrations

2. CANDELA 2 Project Presentation CANDELA........................... Outline Introduction & Positioning Video / Multimedia Content Analysis (VCA / MCA) and Networks for Home MultimediaIntroduction & Positioning Video / Multimedia Content Analysis (VCA / MCA) and Networks for Home Multimedia WP1: VCA / MCA for Home MultimediaWP1: VCA / MCA for Home Multimedia Service Unit AV segmentationService Unit AV segmentation Service Unit Face detectionService Unit Face detection Service Unit Sport sequence analysisService Unit Sport sequence analysis WP2: System ArchitectureWP2: System Architecture Distributed content analysis (encapsulated components – Service Units), system architecture and system solutions for prototyping in-home networksDistributed content analysis (encapsulated components – Service Units), system architecture and system solutions for prototyping in-home networks WP2: AV content and meta databaseWP2: AV content and meta database Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features Integration with other CANDELA application domainsIntegration with other CANDELA application domains Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia MPEG7 MPEG2/4

3. CANDELA 3 Project Presentation CANDELA........................... WP1WP2WP3 Networked infrastructure Interoperability of media and devices Demonstrators Surveillance Home Multimedia Medical systems Applications CANDELA introduction & positioning Focus of the Home Multimedia application demonstrator Philips Research, TU/e

4. CANDELA 4 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... WWW record play Cable EPG Content management Aim of CANDELA Home Multimedia Content Management in Consumer Electronics (CE) Networks The Future of Consumer Electronics In-Home Networks –storage capacity of >1 Tbyte  consumer desire for ease-of-use –Distributed processing GigaFlops  Problem: retrieving and managing AV content  Required:  Metadata (content descriptors) & their generation (VCA / MCA)  System solutions for distributed content analysis (DCA) including media database solutions and metadata standards for interoperability

5. CANDELA 5 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... User Interface Cable WWW AV Content & descriptors Content Management HDD Database MPEG7 Storage VCA for ‘Smart’ content management VCA for ‘Smart’ content management Real-time Content Analysis dissemination  Demo at ICME 2005 (Amsterdam, The Netherlands)  Collaboration projects CANDELA / MultimediaN

6. CANDELA 6 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... WP1: VCA / MCA for home multimedia Service Unit (SU) AV Segmentation & Face Detection AV SegmentationAV Segmentation –Auto AV chaptering for CE devices –Annotation tool for Home Videos SU Cut Detection (CD) + SU Audio Segmentation (AS)... results in SU AV Segmentation (AVS)SU Cut Detection (CD) + SU Audio Segmentation (AS)... results in SU AV Segmentation (AVS) SU CD SU AS SU ScBD Media Database MPEG-7 conversion User e.g. SOLID / MICROGENESIS Content from e.g. MICROGENESIS / Bosch SU FD Face Detection Face Detection –Still & video content management –Surveillance & Automotive & Basis for Face Recognition: SU Face Detection (FD) Face Detector on a commercial Philips productsFace Detector on a commercial Philips products Benchmarked of various Viola-Jones-based Face DetectorsBenchmarked of various Viola-Jones-based Face Detectors CANDELA: content from Co. Microgenesis / SOLID  MPEG-7 output to SOLID / BOSCH / Microgenesis MPEG-7 output to SOLID / BOSCH / Microgenesis

7. CANDELA 7 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... WP1: VCA / MCA for home multimedia Service Unit Automatic Sports AV Analysis (Video Coding and Architectures Group / TU Eindhoven) Court detection and camera calibration Player detection and tracking Event analysis Video input Playing scene detection 1. Get local area 2. White pixel rate 3. Threshold difference 1. Detect white pixel 2. Detect line 3. Camera model 1. Build background 2. Produce binary map 3. Extract players 4. Track players 1. Real world speed 2. Real world position 3. Speed change Output

8. CANDELA 8 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... Demo

9. CANDELA 9 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... Outline Introduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for Home MultimediaIntroduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for Home Multimedia WP1: AV CA for Home MultimediaWP1: AV CA for Home Multimedia Service Unit AV content segmentationService Unit AV content segmentation Service Unit Face detectionService Unit Face detection Service Unit Sport sequence analysisService Unit Sport sequence analysis WP2: System ArchitectureWP2: System Architecture Distributed content analysis (encapsulated components – Service Units), system architecture and system solutions for prototyping in-home networksDistributed content analysis (encapsulated components – Service Units), system architecture and system solutions for prototyping in-home networks WP2: AV content and meta databaseWP2: AV content and meta database Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features Integration with other CANDELA application domainsIntegration with other CANDELA application domains Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia MPEG7 MPEG2/4

10. CANDELA 10 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... WP2: networked delivery System architecture for the in-home systemSystem architecture for the in-home system –database, terminals, processing components –interconnect –protocols The connected devices form the physical platformThe connected devices form the physical platform –content sharing –shared processing –shared resources Software architectureSoftware architecture –service units encapsulate functionality –this functionality is exposed on the network –applications as compositions of service units (e.g. VCA applications)

11. CANDELA 11 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... Example: commercial block detector Input Required metadata: -commercial block detection AVDecoder SU Meta YUV SU AFX SU MetaSync SU Low-level features Silence Detector SU Cut Detector SU AV Segmentor SU Mid-level features Commercial Block Detector SU High-level features MDB GUI metadata audio video PC2 PC4 PC3 Network PC5 PC6 metadata video audio video metadata

12. CANDELA 12 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... Focus: reliability Applications rely on the cooperation of several devicesApplications rely on the cooperation of several devices –devices may disappear –more points of failures –integration of 3 rd party VCA/MCA components Considered errorsConsidered errors –Service Units Logical (catchable by a Service Unit itself)Logical (catchable by a Service Unit itself) Physical (failstop of process running the service)Physical (failstop of process running the service) –Host (device) breakdown –Communication (network) failure

13. CANDELA 13 Project Presentation CANDELA Final Review, June 16, 2005, Helsinki, FI........................... Example Error detection and recovery 1.Unplug a network cable from the system 2.The health keeper detects the malfunction 3.Recovery: the missing component is started on another node and reconnected after a timeout has exceeded SU1 HM1 PC1 SU2 HM2 PC2 PC3 Health Keeper SU1_new PC4

14. CANDELA 14 Project Presentation CANDELA........................... Outline Introduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for Home MultimediaIntroduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for Home Multimedia WP1: AV CA for Home MultimediaWP1: AV CA for Home Multimedia Service Unit AV content segmentationService Unit AV content segmentation Service Unit Face detectionService Unit Face detection Service Unit Sport sequence analysisService Unit Sport sequence analysis WP2: System ArchitectureWP2: System Architecture Distributed content analysis, system architecture and system solutions for prototyping in-home networksDistributed content analysis, system architecture and system solutions for prototyping in-home networks WP2: AV content and meta data databaseWP2: AV content and meta data database Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features Integration with other CANDELA application domainsIntegration with other CANDELA application domains Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia MPEG7 MPEG2/4

15. CANDELA 15 Project Presentation CANDELA........................... WP2: AV content and meta database Allows persistent storage of featuresAllows persistent storage of features Allows nonlinear features browsing and searchingAllows nonlinear features browsing and searching Allows non real-time features processingAllows non real-time features processing Enables features distribution using metadata standardsEnables features distribution using metadata standards Enables new applicationsEnables new applications Serves as a reference to evaluate new algorithmsServes as a reference to evaluate new algorithms

16. CANDELA 16 Project Presentation CANDELA........................... Media Database (MDB) Framework Persistent storage and management of:Persistent storage and management of: –digital asset and asset collections –AV features –info about people and devices Data representation:Data representation: –multiple formats (DIDL, MPEG7, ID3) –dynamic views (annotations) –views on remote datasets (snapshots) Adaptable :Adaptable : –support for ‘unknown’ metadata –extensible (API, model) Interfaces:Interfaces: –relational –object-oriented Mobile Stationary wrappers APIs objects/ schemas Asset Mng Person Mng AV Features EPG Mng DBMSes databases MySQL Solid SQLite XML B-tree local DB snapshots ODBC SQL libdbi wrappers APIs objects/ schemas Asset Mng Person Mng AV Features EPG Mng DBMSes databases MySQL Solid SQLite XML B-tree local DB snapshots ODBC SQL libdbi wrappers APIs objects/ schemas Asset Mng Person Mng AV Features EPG Mng DBMSes databases MySQL Solid SQLite XML B-tree local DB snapshots ODBC SQL libdbi

17. CANDELA 17 Project Presentation CANDELA........................... Analysis Content Metadata A/V Signal (Solid, Microgenesis) MPEG7 Integration with other CANDELA demonstrators Home Multimedia meets Mobile Multimedia e.g. face detection Mobile multimedia demo Home multimedia demo DB

18. CANDELA 18 Project Presentation CANDELA........................... Summarizing: results VCA algorithms for generating metadataVCA algorithms for generating metadata –video annotation with e.g. face recognition, shot detection, tennis scene analysis Different semantic levels: low, mid and high-level featuresDifferent semantic levels: low, mid and high-level features –mid level: semantic AV segmentation, face detection –mid & high level: tennis player position & behavior Data management system for storing and retrieving AV featuresData management system for storing and retrieving AV features –adaptable for heterogeneous devices –interoperable with multiple data formats System architectureSystem architecture –flexible and reliable composition of components into applications –distributed processing