aceMedia Personal content management in a mobile environment Jonathan Teh Motorola Labs

2 The Figures FP6 Integrated Project 9 countries 45 staff (approx.) 6 industrial partners  4 large  2 SMEs 7 academic partners  4 Universities  3 National Research Centres Co-ordinator : Motorola Ltd Started : 1 Jan 2004 Duration : 4 years Total Budget: €17M

3  aceMedia aims to discover and exploit knowledge inherent in multimedia content, making it more relevant for the user and automating annotation. Intelligent Search & Presentation Content Analysis & Annotation Storage Transmission Content Creation aceMedia vision  aceMedia will implement a full content value chain which will enable content and knowledge creation, update, transmission, and manipulation & exploitation (through advanced search and retrieval and intelligent content behaviour)

4 The Autonomous Content Entity Metadata Layer Intelligence Layer Content Layer Programmable layer, enabling the ACE to be self-sufficient, self- organizing, self-analysing Knowledge-based Automatic Semantic analysis and annotation using ontologies and Semantic Web technologies. Scalable content for reuse in different devices, different situations and user needs. ACE

5 ACE lifecycle Natural Language Processing Natural Language Processing Knowledge-Assisted Analysis aceToolbox Person Detection Face Detection Knowledge-Assisted Analysis aceToolbox Person Detection Face Detection Multimedia Indexing NLP Visual Search Relevance Feedback Personalised Retrieval NLP Visual Search Relevance Feedback Personalised Retrieval Cross-Media Engine Content delivery tools Content self-organization Cross-Media Engine Content delivery tools Content self-organization Self-governance

6 ACE creation Metadata Layer Intelligence Layer Content Layer ACE Empt y  Digital content is created and imported into an aceMedia device. 1. ACE content layer is created  Scalable content through aceMedia SVC and JPEG2000  Provide support for transmission across different networks and rendering on different devices 2. ACE intelligent layer possibly defined at ACE creation  Personalising the ACE (presentation, sharing rules, visibility), making it self- organising, …

7 ACE transmission and presentation ACE User, Terminal & Network Profiles User, Terminal & Network Profiles CME Adapted ACE  ACEs are transmitted through heterogeneous networks and presented on aceMedia devices  aceMedia Cross-Media Engine (CME) acts as a processing node to:  Select the appropriate parameters and adaptation modality based on user, device and network profiles  Select the appropriate available content adaptation tool  Configure and manage that tool to create the adapted content  Both content and metadata are scalable

8 ACE pro-activeness  The ACE intelligent layer  Provides the ACE with pro-active behaviour  As opposed to passive, static content  Takes functionality together with the content  Allows complex personalization features  Allows better exploitation of the knowledge attached to content: ACE metadata layer  Can make use of complex content analysis  But…  Has to be executed in a sandbox  Risk of malicious code  Challenging in constrained devices

9 aceMedia structure Cross-Media Engine Wavelet based scalable video codec Content Delivery Tools Knowledge Infrastructure Content Engineering Content Analysis Content Management

10 aceMedia structure Knowledge Infrastructure Content Engineering Content Analysis Content Management Content Classification Natural Language Processing Low-level Features Region Labelling Person and Face Detection & Recognition Multimedia Reasoning

11 aceMedia structure Knowledge Infrastructure Content Engineering Content Analysis Content Management Personalized Browsing Intelligent Search and Retrieval Self-Organization Content Privacy Mgmt

12 OS Java VM aceFramework aceManager ACE Intelligence Exec. Env. aceNetwork Layer Configuration Storage aceMedia Application Other aceMedia applications over the same aceFramework … ACE Repository Software Architecture AMRegistry

13 Platforms  aceMedia system supports multiple platforms  Framework  Implemented in Java for portability  Linux on all platforms  JNI used where performance is necessary  OSGi for dynamically loadable modules  Content engineering  Scalable codecs to adapt to different networks and devices  JXTA network layer for content delivery  Content analysis  Remote invocation of modules to perform analysis on a different device  Content management  Self-governance to ensure access rights are enforced  Application  Multiple UIs designed to take into account different form factors, display sizes and input devices

14 User Interface  Prototyped on an A780 Creating a collection Browsing collections Browsing a collection

15 User Interface (2) Search word and image Edit annotations (detailed)

16 Conclusion  Entering the 4 th and final year of the project  Implemented software modules for content engineering, analysis and management  Software running on multiple platforms  Performed user studies on multiple platforms

17 Backup slides

18 ACE update - typical annotation process Manual Annotations Manual Annotations NLP processes manual annotations If rich enough, then automatic selection of domain ontology Visual Content Detector Content is classified: indoor / outdoor natural / man-made Person and Face detection and recognition Standing persons and faces detected. Learned faces recognized. Knowledge- Assisted Analysis Regions are labelled with concepts from the domain ontology Manual Annotations Manual Annotations Automatic Semantic Annotations Automatic Semantic Annotations Multimedia Reasoning Ambiguities removed. Regions merged. Final consistent semantic annotation Emp ty ACE aceMedia content analysis aceMedia content analysis