1. How a Cooperative Behavior can emerge from a Robot Team Antonio D'Angelo Univ. of Udine Emanuele Menegatti Univ. of Padua Enrico Pagello Univ. of Padua

2. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Background and Motivations behavior-based robotics group of interacting autonomous robots biologically motivated using stigmergy exploiting both implicit and explicit communication deliberative versus reactive using dynamical role assignment

3. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Preceding Works Reactve Architecture for RoboCup Competition [IJCAI- 97] : implicit coordination was first devised Implicit Coordination in Multi-Agent Systems [DARS-98] : the idea of stigmergy was introduced Emergent Cooperative Behaviour [IAS- 98] : macroparameters and voidance Cooperative behaviors in multi-robot systems [RAS-29(1)] : some properties of voidance Emergent behaviors of a robot team performing cooperative tasks [AR-17(1)] : quality functions and roles

4. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Why Multi-Robot Systems (MRS) have been so successful ? In challenging application domains, MRS can often deal with tasks that are difficult, if not impossible, to be accomplished by an individual robot. A team of robots may provide redundancy and contribute cooperatively to solve the assigned task, or they may perform the assigned task in a more reliable, faster, or cheaper way beyond what is possible with single robots.

5. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. What a Cooperative Multi-Robot System is ? While Dudek [Autonomous Robots 1996] and Cao [Autonomous Robots 1997] gave a taxonomy, we prefer to identify several primary research areas Early research goes to Cellular Robotics by Fukuda, IECON 1987] and Cyclic Swarm by [Beni, Intelligent Control 1988] Multi-Robot Motion Planning by [Arai, IROS 1989] ACTRESS Architecture by [Asama, IROS 1989] Cooperative Robotics research field is so new that no topic can be considered mature

6. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Research roots for Cooperative Multi- Robot Systems Cooperative mobile robotics research began after the new behavior- based control paradigm Brooks 1986, Arkin 1990 Since behavior-based paradigm is rooted in biological inspirations, many researchers found it instructive to examine the social characteristics of insects and animals The most common application is using simple local control rules of various biological societies, like ants, bees, and birds, for similar behaviors in MRS MRS can flock, disperse, aggregate, forage, and follow trails, etc.

7. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. New and interesting research issues The dynamics of ecosystems has been applied to MRS to demonstrate Emergent Cooperation Cooperation in higher animals, such as wolf packs, has generated significant study in Predator-Prey Systems Pursuit policies relay expected capture times to the speed and intelligence of the evaders and the sensing capabilties of the pursuers Competition in MRS, such as in higher animals including humans, is being studied in domains such as multi-robot soccer.

8. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Inherently cooperative tasks [1] A particular challenging domain for MRS is the one whose tasks are inherently cooperative, that is, tasks in which the utility of the action of one robot depends upon teammates’ current actions Inherently cooperative tasks cannot decomposed into independent sub-tasks to be solved by a DARS Team success throughout task execution is measured by the combined actions of the robot team, rather than by individual actions

9. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Inherently cooperative tasks [2] More recently identified biological topics of relevance are: Imitation in higher animals to learn new behaviors Physical Interconnectivity by insects such as ants, to enable collective navigation over challenging terrains How to maintain Communication in a distributed animal society

10. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Communication versus Cooperation [1] Communication issue in MRS started since the inception of Distributed Autonomous Robots Systems (DARS) research. Distinctions between Implicit and Explicit Communication are usually made: Implicit communication occurs as a side-effect of other actions, or “through the world” Explicit communication is a specific act designed solely to convey information to other robots on the team.

11. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Communication versus Cooperation [2] Communication affects the performance of MRS in a variety of tasks even a small amount of information can lead to great benefit The challenge is to maintain a reliable communication even when connections between robots may change dynamically and unexpectedly setting up and maintaining distributed network

12. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Architecture and Task Planning [1] Research in DARS has focused on the development of architectures, task planning capabilities, and control addressing the issues of: action selection communication structure heterogeneity versus homogeneity of robots achieving coherence among team actions resolving conflicts, etc.

13. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Architecture and Task Planning [2] Each architecture focuses on providing a specific type of DARS capability: fault tolerance swarm control human design of mission plans role assignment, etc.

14. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Motion Coordination The Theoretical Background A popular topic in MRS is the Motion Coordination. It includes: multi-robot path planning formation generation and formation keeping, traffic control, etc. These issues are now well understood, although demonstration of these techniques in physical MRS (rather than in simulation) has been limited. Computational Complexity strongly limit applicability to dynamic real-time robot teams: the problem of moving multiple objects in a shared environment has been showed to be PSPACE-hard by Hopcroft, Schwartz & Sharir, 1984

15. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. General research issues introduced into RoboCup since the beginning RoboCup is organized in Leagues, each one having its own architectural constraints. Research issues are slightly different one to another. For simulated leagues: Teamwork among agents Agent modelling, from primitive skills to complex ones Multi-agent learning, for on-line and off-line learning of simple skills, as well as more complex strategy, etc.

16. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Issues in Real RoboCup Robots For real leagues: Real-time global or distributed perception from different sensing sources Individual mechanical skills of the physical robots Strategic navigation and actions, etc.

17. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. A Hybrid Architecture for MRS We suggest to use an Hybrid architecture where the Deliberative part and the Reactive part can take mutual advantages. We introduced Robot Schemas at the low level, as building blocks to grow-up complex behaviors from simple ones, according to Arbib and Arkin : Behaviors are chunks of basic knowledge of how to act and perceive. Each behavior is implemented with a schema composed by a motor schema, representing the physical activities a perceptual schema which includes the sensing

18. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. The Perceptual/Motor Schema At each level, the primitive control component is a behavior built by perceptual and motor schemas only. The lower reactive level uses only information coming from sensors, and feeds the motors with appropriate commands. It can elaborate on some perceptual patterns generated by other individual robots, both opponents and temmates.

19. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. The Layered Levels of Control By releasing a behavior, we fire an activation-inhibition mechanism, built on some given evaluation condition rule, at some level of abstraction. Simple Behaviors like defendArea, or carryBall, are implemented as motor schemas accessing directly the robot effectors. Basic Behaviors, like playDefensive, and chaseBall, are obtained by simply appending two perceptual schemas seeBall and haveBall. playDefensive : seeBall --> defendArea chaseBall : haveBall --> carryBall Since a primitive behavior results in appending just one perceptual schema to one motor schema, at the reactive level we obtain sensori- motor coordinations

20. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. An Abstract Architecture Compound behaviors appear only at higher level, when they may receive more structured information about the environment. Only the higher deliberative levels refer to cooperative capabilities that any robot could exhibit as a teammate, while a cooperative behavior is going to emerge.

21. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Software architecture: the ArtiFact implementation We designed a new hybrid deliberative reactive architecture. The classic deliberative paradigm (Sense-Reason-Act) has been evolved reinforcing reactive behaviors. A direct link between sense and act has been introduced to speed-up the reactive response of the robot Thus, deliberative conditions can be bypassed for certain inputs which need more reactive behaviors

22. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. A Functional Architecture [1]

23. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. A Functional Architecture [2] The architecture of each single robot shows An inner loop, for close feedback, An outer looop, for high level reasoning. To allow cooperation with teammates, two sensorial sources can input asynchronously both Environment constraints (the “Ruler”) Information about teammates (the “Teamplay”)‏

24. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. ADE Software Environment All software runs within the frame of ADE, a multi-thread distributed real-time environment working under Linux-OS ADE allows to create a set of processes structured as threads. Each thread can communicate, through message passing, with other threads of same process, and also with other processes running on other processors If a segmentation fault happens, it is possible to kill the thread that caused the error

25. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. On Role Allocation in RoboCup [1] Inspired by Stone and Veloso’s pioneering work, many teams employ role-based coordination, in which robots can take on different static roles within the team Although it would be possible to statically assign roles once forever, most teams switched to dynamic role allocation, by solving an iterated assignement problem, where the current allocation is re-evaluated periodically 10 times for each millisecond

26. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. On Role Allocation in RoboCup [2] Given N robots, N prioritized (weighted) single-robot roles, and some estimates of how well each robot can be expected to play each role, assign robots to roles so as to maximize the overall expected performance Gerkey and Mataric [Springer Book on RoboCup2004] showed that this technique is an instance of the canonical Greedy algorithm for Optimization theory

27. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Adopted solutions in RoboCup Teams RoboCup role allocation problem is similar to task allocation problem for MRS in order to cooperatively achieve the goal, where a time-extended role concept replace that of a transient task CS Friburg Team used a distributed role allocation mechanism in which two robots may exchange roles only if both want to do it, both moving to a higher-utility role for themselves. ART Team, as well as early, Artisti Veneti Team, ordered the roles in a descending priority, and then assigned each to the available robot with the highest utility.

28. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Utility Functions Multi-robot role allocation is a dynamic decision problem, that varies in time, according to the environmental changes, Utility concept rely on the fact that each individual robot can somehow internally estimate the value (i.e. the cost) of executing an action In RoboCup it is common to compute utility as the weighted sum of factors like distance to target, distance to ball, defence- offense coonfigurations, etc. The computation is affected by sensor noise, general uncertainties, and environmental changes Given the utility value Uij of each robot i for each role j, find the highest utility Uij, assign robot i to role j, and iterate

29. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Master/Supporter Roles Coordination Consider the coordination between two robots carrying the ball towards the opponent’s goal: We may indentify a Master Role and a Supporter Role Roles can be played at different responsibility levels: Can be >>> Assume >>> Acquire >>> Advocate Ball assignments depend on Ball Possesses HaveBall condition allows to discriminate which robot is really carrying the ball It is an Environment constraint acting as a kind of Macroparameter, evaluated by different teammates It allows to synchronize the activation of a new cooperation pattern

30. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Notifying Roles Roles can be switched provided a notification is exchanged between teammates A notification implies a communication between teammates on first-notified/first-advocated basis A notify(Role) rule is: Supporter (mate) -->> reply (role, mate)‏ Master (mate) --> request (role, mate)‏ Environment Rules require that a Master role must be advocated, whereas a Supporter role should be acquired. haveBall and notify(Role) are the two allowed asynchronous communication from outside for a single robot

31. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Implementing Clamping Behaviors [1] A role is switched from acquire to advocate, or from assume to acquire, provided a notification is made to its teammate Two complex Clamping Behaviors for Master and Supporter can be constructed from notify (x) and haveBall (z)‏ The Master robot shows a chase_ball behavior haveBall (me) & not haveBall (mate) -->> acquire (Master)‏ Acquire (Master) & Notify (Master) -->> advocate (Master)‏

32. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Implementing Clamping Behaviors [2] The Supporter robot shows an approach_ball behavior Not acquire (Master) & canBe (Supporter) -->> assume (Supporter)‏ Assume (Supporter) & Notify (Supporter) -->> acquire (Supporter)‏ The robot chasing the ball suggests a teammate to become supporter by advocating a master role, and forcing the other robot to acquire a supporter role by approaching the ball.

33. How a Cooperative Behavior can emerge from a Robot Team A. D'Angelo et. al. Conclusions How much deliberative process should be endowed in the distributed control of a robot team to play a soccer game cooperatively? To force coordination when the dynamics of the game is not quite fast, we have tried to drive a cooperative task by a dynamical role assignment, switching from the implicit team assessment, given by the evaluation of quality functions, to the explicit negotiation on the first notified/first advocated basis