Chapter 1. Cognitive Systems Introduction in Cognitive Systems, Christensen et al. Course: Robots Learning from Humans Park, Sae-Rom Lee, Woo-Jin Statistical Learning & Computational Finance Laboratory Industrial Engineering Seoul National University

Contents Introduction Objective of Project Motivating example Organization of the Research/Research Questions Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research Organization of the Book 2

CoSy project The assumption of the visionary FP6 “To construct physically instantiated … systems that can perceive, understand … and interact with their environment, and evolve in order to achieve human-like performance in activities requiring context- (situation and task) specific knowledge” Requirements Architectures, forms of representation, perceptual mechanisms, learning, planning, reasoning, motivation, action, and communication To validate science progress using test scenarios

Contents Introduction Objective of Project Motivating example Organization of the Research/Research Questions Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research Organization of the Book 4

Objective of Project Problem Most systems able to perform complex tasks that humans and other animals can perform easily, for instance robot manipulators, or intelligent advisers, have to be carefully crafted The way to forward Combining many different capabilities in a coherent manner -> 4-5 year child Generic capabilities

Steps to Success Achievable sub-goals Theory deliverables Implementation deliverables Theory deliverables The notion of an architecture combining components Reactive Deliberative Self-reflective, meta management Different learning processes Different varieties of communication and social interaction

Steps to Success Implementation Deliverables nature nurture vs Linguistic Visual Reasoning Planning Motor skills Integration

Contents Introduction Objective of Project Motivating example Organization of the Research/Research Questions Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research Organization of the Book 8

Motivating Example

Contents Introduction Objective of Project Motivating example Organization of the Research/Research Questions Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research Organization of the Book 10

Organization of the Research Research challenges Two scenarios for study of integrated systems Two major milestones Using intermodality and affordances for the acquisition of concepts, categories and language Introspection of models & representations; planning for autonomy – goal seeking architectures Represen- tations learning Perception- action modeling Communi- cation Planning & failure handling

Architecture Putting pieces together into a complex functional system Perception, action, reasoning and communicating

Representation The representation should Enable integration of representations of objects, scenes, actions, events, causal relations and affordances Allow incremental updating or sometimes correction Allow different types of learning (supervised, unsupervised, reinforcement) Allow integration of various modalities, of very different input signals Be suitable for recognition and categorization in the presence of a cluttered background and variable illumination Be scalable

Representation

Specific vs General Representations

Learning Modes of learning Tutor Driven A user (tutor) shows to the system an object or an action and explains to the cognitive system what he/she is showing or doing Tutor Supervised A cognitive system detects a new object, an action, event, affordance or a scene by itself and builds its representation in an unsupervised manner. Exploratory Updates the representation autonomously

Learning Example

Continuous Learning Representations employed allow the learning to be a continuous, open-ended, life-long process Continuously updated over time, adapting to the change in environment, new tasks, user reactions, user preferences, … Reliable continuous learning Representations have to be carefully chosen How new data is extracted and prepared

Perception-Action Modelling Abstract relation model General, non-task specific Observability of the world hand-constructed abstraction Probability relational representation Capture uncertainty in both action and observation Tractable for localization and path planning in continuous space Sensor-dependence Reinforcement learning Identifying features that are relevant to predicting the outcome on the task

Continuous Planning Difficulties Dynamic nature, partial observability Conditional planning Probabilistic planning 20

Continuous Planning Active Failure Diagnosis In most approaches it is typically assumed that the sensors and actuators of the robot are reliable in the sense that their input always corresponds to the expected input and that there is no malfunction of the sensors or actuators These approaches do not exploit the actuators of the robot to identify potential faults Once a fault has been identified, the high-level system is notified so that appropriate actions can be generated at the planning level. 21

Continuous Planning Collaborative Planning and Acting Cooperation is at the heart of the Cosy project Common language protocol Dialogue

Continuous Planning

Models of Action and Communication for Embodied Cognitive Agents Natural Language Integration of communication and action Recognition of intention, attention, and grounding/understanding Mixed-initiative Embodiment in an unknown environment

Models of Action and Communication for Embodied Cognitive Agents

Multi-Modal Recognition and Categorization Recognize Categorize Entry level categorization vs Recognition Recognition of objects Categorization Multi-cue

Scenario Driven Research System level Exploration/Mapping of Space Models of objects and concepts

Exploration / Mapping of space Where am I? How do I get to my destination? How do I detect that I have arrived at the destination? Perception and action Localization in the World Construction of a map of the environment Plan a sequence of actions

Affordances and Newer Approaches Space Object Robustness Wall, Door, Table

The World as an Outside Memory

Mapping of the Environment Encoding of position of objects/places Encoding of environmental topology Invariant to changes to perception system Invariant to changes in action system Facilitate spatial reasoning

Models for Object and Concepts Representation Continuous Learning Robustness Categorization Architecture Communication and Language

Contents Introduction Objective of Project Motivating example Organization of the Research/Research Questions Architecture Representations Learning Perception-Action Modelling Continuous Planning and Acting Models of Action and Communication Multi-Modal Recognition and Categrization Scenario Driven Research Organization of the Book 33

Organization Chapter 2 Architecture design, representation Chapter 3 perception - action Chapter 4 spatial maps Chapter 5 visual perception Chapter 6 planning recovery Chapter 7 adaptation & learning Chapter 8 Human-robot interaction Chapter 9 & 10 Demonstration

Thank you for Listening