GRASP Management Meeting March 09 -2009 Munich

Agenda 10.00-11.00 EU reporting 11.00-12.00 Comments on the deliverables 12.00-13.00 Plan for integration 13.00-14.00 Lunch 14.00-15.00 Plan for review 15.00-17.00 Plan for integration 17.00-17.30 Wrap up

EU reporting A periodic report within 60 days of the end of each reporting period . The periodic report comprises: a)      An overview, including a publishable summary of the progress of work towards the objectives of the project, including achievements and attainment of any milestones and deliverables identified in Annex I. This report should include the differences between work expected to be carried out in accordance with Annex I and that actually carried out, b)      An explanation of the use of the resources, and c)    A Financial Statement from each beneficiary and each third party, if applicable, together with a summary financial report consolidating the claimed Community contribution of all the beneficiaries (and third parties) in an aggregate form, based on the information provided in Form C by each beneficiary. 

Parts of PPR Publishable summary: max 4 pages, should be publishable by the Commission: project objectives and work done; should be updated every reporting period Project objectives for the period: short overview of objectives as indicated in Annex1 Work progress and achievements: progress overview and relation to the state of the art; work packages progress with main results and deviations Deliverable and milestones tables

Parts of PPR Project management Management activities Problems and changes List of project meetings, venues Deviations from the milestones Cooperation and collaboration (also with other projects)

Parts of PPR Dissemination Website List of publications List of events: organized and planned for the next period List of exploitable results

What we promised?

What we promised? 1. Month 12: Definition and implementation of initial ontology: We will provide a description and a model of how selective visual processing is affected by object properties, task and object knowledge (D2). This will make basis for the definition of the initial ontology (D4). First prototype of human body detection and tracking will be developed (D3). A model of the integration of the individual modules will be given (D8) and will consider the integration of the control strategies (D5), the definitions of grasping primitives (D5) and modules for visual attention (D6). We will be able to demonstrate open-loop grasping and object manipulation strategies on several platforms using stereo based representations of object attributes. Planned prototypes and demonstrations: A first operational prototype of models for pre-reasoning and visual attention (D6). Demonstration on the humanoid head for monitoring of the environment for surprises (D7). Demonstration of the initial hand/body tracking system on the humanoid head (D3).

Comments on the deliverables

Plan for integration GRASP ’image’ Scenario(s) Who/What/When

Learning and abstraction Execution and Grounding Human data: Learning and abstraction WP1, WP2 LEARNING LOOP EMULATION WP2, WP4 Control cycle: Execution and Grounding World/Simulator Planning/Reasoning ”Mental model” WP2, WP7 generation Action Perception INTROSPECTION WP3, WP6, WP7 WP4, WP5 SURPRISE MISSION LOOP Task generation

Symbolic representation Task-level planning WP4, WP5 WP2, WP7 WP3, WP6, WP7 Off-line Perception: Vision and Haptics Memory system Symbolic representation Embodiment specific attributes Planning Execution and Prediction World/Simulator WP2, WP7, WP4, WP3? Reasoning Low-level or feature level WP2, WP3?, WP5? Human grasp: Learning and abstraction Sensorimotor experience WP1, WP2

Some issues Scenario in GRASP Libraries in GRASP Vision in GRASP Haptics in GRASP Hand models (robot and human) Objects in GRASP Knowledge representation in GRASP: objects, actions

Scenario year 1 Handling some the of 8 objects on the table (not in the basket) WP1: observe a human grasping one of these objects and provide the tracked 3D model of the hand and a classification of the type of grasp Note: A grasp is defined by: Grasp type Grasp starting point Approaching direction Hand orientation WP1 (Heiner): provide kinematics of the grasps, grasping points and covert and overt attention (see Daniel’s presentation this afternoon)

Scenario year 1 Handling some the 8 objects on the table (not in the basket) WP2: Discrete mapping of observed human grasp activities to one/two hand robots. Extract DMPs from the observed movement, Representations for integration WP3: Demonstrate the grasping cycle using the grasp types form WP2, object type and attributes from WP4 at the UJI platform WP4: Background/foreground segmentation in the case of textured objects, grasp points generation, pose of the object (6D) (bounding boxes), Identify the primitive shapes (3D model fitting) WP5: definition of the expectation model necessary for detecting surprise

Scenario year 1 Handling some the 8 objects on the table (not in the basket) WP6: Initial version of the simulator Integration of COLLADA and PAL to OpenRAVE Reproduction of what have been demonstrated in WP1, Mapping from WP2, location form WP4 on ARMAR in OpenRAVE WP7: Proposal for integration for the described scenario with focus on how to represent objects, actions (MMM, DMPs), input/output definitions, OpenRAVE/MCA Reproduction of grasping cylindrical textured objects on ARMAR using the 6D pose from WP4

Objects (1) Object representations via meshes in all workpackages Cylinder-like and box-like objects GRASP objects (8 items) Boxed salt (SFB 588), object ID 2 Cylindrical salt (SFB 588), object ID 3 Gauloises red Zwieback (SFB 588), object ID 11 Cups (Dani’s cups, i4280.JPG) Two different cups (two each to generate textured and no-textured) Complete representation of one object Meshes Stereo

Objects (2) Original monocular images (10 views) Darius Stereo images (5 views) for Markus, Lech Stereo information (Markus, Lech) Internal and external calibration (Depth map for the 5 stereo views ) Who will provide the meshes for these objects (UniKarl) Geometrical models of the objects are also needed

Input-output for scenario year 1 WP1 Input: Human experiments from WP1 (LMU, Daniel)  FORTH Output: Grasp type (1 of 4) and approaching vector (FORTH, Antonis, Georgios) (Human grasping library) WP2 Input: Human Experiments from WP1 (LMU, Daniel) Grasp Ontology, i.e. hierarchy of human hand postures, discrete mapping to Barrett and Karlsruhe hand; approach vector relation to the objects in the database (KTH, OB, Dani, Dan and Thomas) Representation of humans grasps using DMPs (Martin, Tamim)

Input-output for scenario year 1 WP3: Input: Results form KTH (Output WP2), 6D pose of grasp objects (WP4, Chavdar), object type (WP4, Chavdar) Output: grasping cycle on the UIJ platform (UJI, Javier) and (LUT, Janne) WP4: Input: Stereo-images from UJI plus calibration data (UJI, Antonio) Output: 2.5 point cloud + mesh (TUW, Lech and Mario); 6D object pose estimation (TUM, Chavdar, TUW, Markus and Lech); remote distributed computing) WP5: Input: Ontology hierarchy from WP2 (Maria, Dan and Thomas), object and scene representation from WP4 (TUM, Darius) Output: Stereo sequences of humans manipulation

Input-output for scenario year 1 WP6 Input: Results from KTH (Output WP2), 6D pose of grasp objects (WP4, Chavdar), object type (WP4, Chavdar) ARMAR controller OpenRAVE Plugin (UniKarl, Stefan, Markus) Output: Execution of observed movements (WP1) in the Simulation (OpenRAVE in its original version) on ARMAR (Demonstration of collision detection for the introspection detection, i.e. collision with the other 7 objects on the table, UniKarl, Tamim) WP7: Input: grasp ontology (KTH), action representation (UniKarl, KTH), objects representations (TUW, Markus), IO from all other WPs (see above) ; models of prediction (WP2, WP5, ????) Output: Demonstrate integration of OpenRAVE and MCA (UniKarl, Stefan, Tamim) and the predict-act-perceive cycle on ARMAR (UniKarl, Markus, Tamim).

Representations of object, action and surprise COLLADA file ID, category, shape, mesh, weight, material, inertia, CoM Grasp types Action Vocabulary of actions Reach (6D pose) Pre-shape (grasp type) Grasp (approach vector, hand orientation, grip forces) Lift (move in Cartesian space) Transport (move in Cartesian space) Place (move in Cartesian space, contact force) Release (open hand) Representations of object, action, …. and surprise Object-action Complexes (OAC) Embodiment specific and embodiment invariant

When/Who/What

Review meeting I Spain, June 9-10 (Tuesday-Wednesday) Reviewers Everybody arrives late on 7th (Sunday) or before 10 on 8th Final preparations on 8th Reviewers Yiannis Demiris: cognitive architectures, mapping/representations Peter Allen: grasping, simulation Carlos Balaguer: control June 9th: WP presentations June 10th: WP presentations, demosntrations, reviewers’ meeting

Plan for review Presentations Demos: Format and requirements will be sent out Demos:

Example

Input/Output definitions

Learning/Mission PREDICT- ACT – PERCEIVE Learning: Mission: Predict: Predict: simulator as the forward model Mission:

Detailed plan for collaboration

To do list Images: DEADLINE: April, 30th May, 20th – convergence …… Tamim/Antonio/Ville: WP6, WP7 -> Markus (System integration, achitecture, simulator) Dani/Darius/Tamim: WP2, WP5 -> Memory Markus, Darius, Ville: WP4, WP5, WP3 -> Surprise detection (fore learning and for mission) – 2 figures Antonis, Heiner, Dani, Tamim: Human grasping loop (motion primitives, actions, skills, task) DEADLINE: April, 30th May, 20th – convergence …… June