MOMIRAS Kick-Off Meeting, 28/7/ Center for Research and Technology Hellas / Information Technologies Institute ( MOMIRAS Project Overview Dr. Dimitrios Tzovaras CERTH/ITI

MOMIRAS Kick-Off Meeting, 28/7/ Summary of Project - MOMIRAS at a Glance  MOMIRAS aims to provide a highly novel programming environment that will exploit recent advances in mobile 3D sensing and will enable processes like 3D registration, object detection, optimized AR rendering and 3D content-bases search.  MOMIRAS will provide a complete set of tools that from one hand will improve current supported 3D features in Augmented Reality and on the other will further expand them by utilizing novel technologies (based on range image sensing) and algorithms (adaptive scene interactions, interaction simulation engine utilizing audio-visual cues for optimized immersive experience).  MOMIRAS aims to reinforce all key stakeholders in the AR value chain, such as Augmented Reality application developers, content and cultural heritage providers, industrial players e.g. from automotive and manufacturing domains in utilizing new solutions for better marketing and disseminating their assets and products to the end-user community.

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Objectives  Objective 1: To develop new AR algorithms that exploit depth sensing for efficient annotation and analysis of the interacting environment. 3D Scene Reconstruction by combining depth and visual cues Continuous and real-time analysis of the interacting environment Main Partners Involved: CERTH, RTC  Objective 2: Development of a multi-layer software architecture that enables robust object recognition and tracking and efficient object search and retrieval. Lightweight Processing components (mobile/embedded devices) High Level Processing components (cloud-based) Main Partners Involved: INTRACOM, CERTH

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Objectives  Objective 3: Advanced User interaction and scene manipulation. Handled by a simulation engine according to real world physics Basic manipulation functions such as add, remove and move of both real and virtual ones objects Main Partners Involved: UPAT, RTC  Objective 4: Head-coupled perspective augmented reality solutions. Realistic rendering of both real and synthetic objects fully exploiting depth information Support for different viewpoints including user’s head motion analysis Main Partners Involved: UPAT, CERTH

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Objectives  Objective 5: Image and 3D geometric in-painting for real world object manipulation. Real-world objects detection, registration and tracking Implementation of in-painting algorithms (2D and 3D) for supporting basic functionalities like virtual deletion of a real object Main Partners Involved: CERTH, RTC  Objective 6: Optimization of AR experience through processing and rendering of multimodal cues. Audio/visual environment capture and analysis Perform synthetic visual and audio rendering to improve existing AR rendering algorithms Main Partners Involved: UPAT, INTRACOM

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Objectives  Objective 7: Demonstration of MOMIRAS in challenging application scenarios. Cultural Heritage (CH) * oSolution for both personnel of cultural heritage sites and end-users (visitors) oAdvanced tour of a cultural collection Mobile Value Added Services (M-VAS)* oExpanded AR-enabled City Guide Service (part of the mobile VAS platform of Intracom) o Explore benefits for Social Networking, Smart Calendar and Advertising * Use Cases shall be early refined and agreed by Consortium partners (also to identify potential content providers)

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS – Gantt Chart (refined) Main Changes Performed in duration of WPs/Tasks All tasks ended after M15 in the original submission now end on M15 All reports and milestones up to M15 kept without changes whenever possible Total 10 Milestones and 20 Deliverables are anticipated in total by the Consortium MS1.1 MS2.1MS2.2 MS3.1 MS3.2 MS5.1 MS4.1 MS6.1 MS6.2 MS6.3

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Architecture WP5 WP2 WP3 WP4 Conceptual Architecture of MOMIRAS mapped to core development Work Packages WP2 Leader – RTC WP3 Leader – RTC WP4 Leader – UPAT WP5 Leader – CERTH/ITI

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS – Pert Diagram

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Anticipated Efforts (highlighted up to M6) DeliverableResponsibleDateNature/Level D1.1 “Service Definition and System Specification” ICOMMonth 5 (November 2014) Report D1.2 “System Design and Concept Architecture ” CERTH/ITIMonth 9 (March 2015) Report  WP1 MilestoneResponsibleDateComments MS1.1 “Availability of system specification and provisional system reference architecture” CERTH/ITIMonth 6 (December 2014) Initial specifications available using UML diagrams

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Anticipated Efforts (highlighted up to M6) DeliverableResponsibleDateNature/Level D2.1 “ Object detection and segmentation module ” RTCM12 (June 2015) Report + S/W D2.2 “Lightweight compact feature descriptors” CERTH/ITIMonth 12 (June 2015) Report + S/W D2.3 “ 3D Pose estimation, registration and 3D reconstruction” RTCMonth 12 (June 2015) Report + S/\W D2.4 “Client SDK infrastructure” RTCMonth 14 (August 2015) Report + S/\W  WP2 MilestoneResponsibleDateComments MS2.1 “ Initial version of the WP2 components available ” RTCM9algorithms available for testing MS2.2 “ Final version of the WP2 components available ” RTCM12All algorithms available

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Anticipated Efforts (highlighted up to M6) DeliverableResponsibleDateNature/Level D3.1 “ Interactions simulation engine ”RTCM14 (Aug. 2015) Report + S/W D3.2 “ Generation and interaction with augmented world data CERTH/ITIMonth 14 (Aug. 2015) Report + S/W D3.3 “ User interactions and manipulations module ” RTCMonth 14 (June 2015) Report + S/\W  WP3 MilestoneResponsibleDateComments MS3.1 “ First version of the simulation and interaction engine ready ” RTCM12 (June 15) Initial integrated version MS3.2 “ Availability of WP3 Results for Integration ” RTCM14 (Aug. 15) All modules tested properly

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Anticipated Efforts (highlighted up to M6) DeliverableResponsibleDateNature/Level D4.1 “ AR user aware rendering engine ” UPATM14 (Aug. 2015) Report + S/W D4.2 “ Algorithms to improve experience and content visualization “ UPATMonth 14 (Aug. 2015) Report + S/W D4.3 “ In painting module” RTCMonth 14 (Aug 2015) Report + S/\W  WP4 MilestoneResponsibleDateComments MS4.1 “ First version of the AR rendering engine ready ” UPATM10 (April 15)Including in- painting

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Anticipated Efforts (highlighted up to M6) DeliverableResponsibleDateNature/Level D5.1 “ Artefact import toolkit ”CERTH/ITIM10 (April 2015) Report + S/W D5.2 “ Feature extraction algorithms from 3D objects and the MOMIRAS 3D search and retrieval engine” CERTH/ITIMonth 14 (Aug. 2015) Report + S/W D5.3 “ WEB Service Infrastructure” ICOMMonth 14 (Aug 2015) Report + Prototype  WP5 MilestoneResponsibleDateComments MS5.1 “ Availability of the WEB Service Infrastructure for deployment ” ICOMM14 (Aug 15)AR Services available

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS Anticipated Efforts (highlighted up to M6) DeliverableResponsibleDateNature/Level D6.1 “Communication Plan including Dissemination and Exploitation” ICOMMonth 3 (Sept. 2014)Report D6.2 “Use case deployment plans and evaluation” CERTH/ITIMonth 15 (Sept. 2015) Report (prelim. version by M6?) D6.3 “Final SDK end-to-end platform”ICOM/RTCMonth 15 (Sept 2015) Report + Prototype D6.4 “Report on Achieving Dissemination and Exploitation objectives” ICOMM15 (Sept 2015)Report D6.5 “Business Model)ICOMM15 (Sept 2015)Report  WP6 MilestoneResponsibleDateComments MS6.1 “Availability of the Communication Plans ” ICOMMonth 3 (Sept. 2014) Web site, dissemination material, Event/Workshop? MS6.2 “Availability of the Integrated SDK platform” ICOM/RTCM15 (Sept. 15) All components working MS6.3 “Availability of the Business Model” ICOMM15 (Sept. 15) Business/Exploitation Model defined

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS – Main Tasks assigned to Partners INTRACOM  Project coordination, including administrative, financial and technical leadership.  Aligning the proposed system with expressed user needs (T1.1) towards production of technical specifications and conceptual system architecture (T1.2).  Efficient deployment of the results from WP2 on embedded platforms and mobile devices as well as means of exploiting cloud infrastructures for enhancing the capabilities and the performance of the MOMIRAS system (T3.1).  In WP4 Intracom will perform the development of the WEB service support infrastructure (T4.4).  Intracom will also provide a back-office support (T5.2) and will lead the exploitation and disemination activties (T5.5).

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS – Role of Partners UPAT  Leader of WP4 on AR optimization  Active involvement in WP2 and WP3 activities in the field of AR optimized rendering, head coupled perspective rendering, interaction and simulation engine. CERTH/ITI  Leader of the architecture definition activities (WP1) and of the cloud- based AR services (WP5)  R&D on novel algorithms for 3D object annotation, indexing and retrieval running on the cloud as services (WP4)  Development of low-level feature descriptors for 3D Object recognition and matching in resource-limited devices such as smart phones and tablets (Task 2.2)  Exploit the development of lightweight algorithms enabling compact feature extraction of 3D objects in non-controlled environments.

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS – Main Tasks assigned to Partners RTC Leader of WP2 : “AR client SDK” and WP3: “Advanced world interaction” Design and development of the SDK infrastructure and state of the are algorithms for object detection, tracking and segmentation, pose estimation and 3D reconstruction oLightweight and for use in android platforms Co-development with UPAT the simulation engine, and also the generation of the augmented world data, and the user interaction module. In WP4, RTC will develop the in-painting module. RTC will also work on the system specification and final integration and deployment

MOMIRAS Kick-Off Meeting, 28/7/ MOMIRAS – Expected Outcomes (per WP) WP2: Lightweight object descriptors for enabling 3D artifact search and retrieval Novel object detection, registration and reconstruction components residing in resource- limited devices (smart phones, etc) WP3: Delivery of innovative Interaction and simulation engines allowing user to interact with content (including virtual content) residing in the real environment WP4: AR rendering algorithms taking into account the environmental context and the motion of the observer 2D/3D in-painting algorithms to optimize the rendering of partial 3D content WP5: Unified artefact descriptors comprising of multi-layer information including semantics, quality indicators and multiple 3D extracted features Efficient and Effective 3D search engine running on a cloud infrastructure Open and Modular Reference Web service infrastructure to accommodate the MOMIRAS AR- enabled services. Technical Development & Innovation (WP2-WP4) Cloud Infrastructure & Services Deployment (WP5-WP6) AR SDK Design & Specs – WP1 MOMIRAS Validation & Assessment (WP6)