1. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program GRASP Emergence of Cognitive Grasping through Emulation, Introspection and Surprise Danica Kragic

2. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program GRASP  Challenge 2: Cognitive Systems, Interaction, Robotics  March 2008 – February 2012  Project officer: Mr Mariusz Baldyga  Coordinator: Danica Kragic (KTH)  Co-coordinator: Markus Vincze (TUW)  Administrator: Jeanna Ayoubi (KTH)  Project number: 215821

3. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Consortium University of Karlsruhe Karlsruhe, Germany T. Asfour, R. Dillmann Kungliga Tekniska Högskolan Stockholm, Sweden D. Kragic, P. Jensfelt Lappeenranta University of Technology, Lappeenranta, Finland V. Kyrki & H. Handroos Technische Universität München, Munich Germany D. Burschka, H. Deubel Technische Universität Wien, Vienna, Austria M. Vincze Universitat Jaume I, Castellón, Spain A. Morales, P. Sanz Otto Bock, Austria OB H. Dietl GRASP Foundation for Research and Technology - Hellas, Greece A. Argyros & M. Lourakis

4. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Introduction  GRASP is a project with a focused target  Objectives Theory of grasp modeling Self and context-awareness Curiosity and surprise driven behavior New grasping strategies Exploitation and evaluation for future prosthesis, industrial and service markets  Grasping and manipulation as a control problem have been studied since the beginning of robotics: HOWEVER- very little has been done in terms of cognitive aspects of grasping, implementation and evaluation of systems - this is what GRASP is about  Still a huge gap between visual servoing and grasping

5. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program GRASP: Concept and Objectives  “Robots handling individually or jointly....” GRASP proposes a generic predict-act-perceive paradigm for grasping and manipulation of objects, dealing with static and dynamic properties of objects, the environment and the task.  Cognitive approach: predict – act – perceive Generic principle of human brain  Paradigm open to development Unsupervised, fully autonomously, new objects  Clear application domain: grasping  Measurable benchmark: empty basket, fill pallet  Expected results: Grasping ontology Proof or disproof of paradigm

6. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Worpackages 1. Learning to Observe Human Grasping and Consequences of Grasping 2. Representations and Ontology for Learning and Abstraction of Grasping 3. Self-experience of Grasping and Multimodal Grounding 4. Perceiving Grasping Context and Interlinking Contextual Knowledge 5. Surprise: Detecting the Unexpected and Learning from it 6. Introspection and Prediction through Simulation 7. Cognitive Robotic Grasping - Integration and Applications

7. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Relationship between WPs

8. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Major Milestones

9. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Platforms, sensory feedback No new hardware development in GRASP

10. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program From simulation to real world and back WP1 What/Where/How WP7 Integration and Applications

11. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Research objectives  Goal: filter action relevant from action-irrelevant stimuli to enhance the manipulation performance; analyse, for a variety of increasingly complex sensorimotor tasks, the spatial and temporal properties of selective visual processing before and during manual reaching and grasping movements of human actors.  WP1: Kinematics and grasping forces, together with the eye gaze behaviour during the grasp will be measured/tracked.  WP2, WP7 – the definition of the ontologies requires definitions of action specific, selective visuo-spatial processes  WP3, WP4, WP6 – representations of control parameters, object and body (hardware) attributes  WP5 - eye-gaze positions: What to look for? What to disregard?

12. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Major Milestones – month 12 1 Spatio-temporal representation of human body and hands, mapping to different robot embodiments; initial evaluation on a humanoid platform. WP1, WP2, WP7 FORTH 2 Definition of initial ontology based on human studies; acquisition (perception and formalisation) of knowledge through hand-environment interaction. WP1, WP2, WP4, WP5 KTH 3 Initial design of the control and engine architectures; evaluation of the perception-action planning cycle in simulation. WP3, WP4, WP6, WP7 LUT

13. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Major Milestones – month 24 4 Analysis of action-specific visuo-spatial processing, vocabulary of human actions/interactions for perception of task relations and affordances. WP1, WP2, WP4, WP5 TUW 5 Implementation of high-level controllers including a global uncertainty model, integration and evaluation in the simulator and experimental platforms, grounding grasping primitives. WP3, WP6, WP7UniKarl 6 Integration and evaluation of human body and hand tracking on active robot heads, demonstration of a grasping cycle on the experimental platforms. WP1, WP2, WP4, WP7 TUM

14. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Major Milestones – month 36 7 Observing consequences of grasping; vocabulary of robot action/interactions and definition of a hierarchical structure of features. WP1, WP2, WP4, WP5 FORTH 8 Implementation of hybrid controllers for on-line adaptive primitive grounding; evaluation in the simulator and on experimental platforms. WP3, WP6, WP7 LUT 9 Integrating contextual representation in the ontology and development of the attention system with view planning; implementation on active head. WP1, WP2, WP5, WP7 TUM

15. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Major Milestones – month 48 10 Linking structure, affordances, actions and tasks; evaluation of representations defined by the ontology. WP2, WP4TUW 11 Integration and evaluation of scenarios on multiple experimental platforms, demonstration of cognitive capabilities of robots. AllUniKarl

16. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program Everything will be open source The simulator Hand models Tracking algorithms Vision algorithms Object and action representations Database with human data Software framework for testing and benchmarking

17. Emergence of Cognitive Grasping through Emulation, Introspection and Surprise GRASP EUl 7 th Framework Program THE END