1. The Hybrid Deliberative/Reactive Paradigm The City College of New York Department of Electrical Engineering Group Member: Jik Cheung Yongwen Zhu Yayi Hu Xuezhou Ma Junjun Li

2. Chapter Objectives Describe the hybrid paradigm in terms of SAP and sensing organization. Distinguish the responsibilities between the deliberative layer and reactive layer. List the basic components of a Hybrid architecture: sequencer agent, resource manager, cartographer, mission planner, performance monitoring and problem solving agent. Identify the difference between managerial, state hierarchy and model-oriented styles of Hybrid architectures. Be able to describe the use of state to define behaviors and deliberative responsibilities in state hierarchy styles of Hybrid architectures.

3. Overview However, the robot could not… Remember the state of the robot/world Plan optimal trajectories Make maps Monitor its own performance Select the best behaviors for a task Reactivity more art then science? Should planning be reintroduced? I.Reactive Paradigm is the major trend by the end of the 1980’s.

4. II.Deliberative Vs. Planning Not all of these activities involve Planning: Make maps Monitor its own performance Select the best behaviors for a task To differentiate this from path planning, the term deliberative was coined.

5. III.Hybrids How can slow planning be intergraded with fast reactivity? Five examples of architectures will be illustrated: AuRA, SFX, 3T, Saphira and TCA. First Opinion: The worst of both worlds! Reactive systems for unstructured worlds Hierarchical systems for knowledge-rich worlds Nowadays: The best of both worlds! Reactive functions for low level control Deliberation for higher level tasks

6. Hybrid Paradigm Organization: Plan, Sense-Act:

7. Motivation of Hybrids Cohesion (object oriented programming) Reactivity: Short time horizon (Present) No global knowledge Work with sensors and actuators Deliberation: Long time horizon (Pass, Future) Global knowledge Work with symbols Multi-tasking Deliberative functions execute in parallel with reactive functions.

8. Sensing Organization The Map (World Model) Can have its own sensors Can “eavesdrop” on other sensors Can act as “virtual” sensor World Map/ Knowledge Rep Behavior Sensor 3 Sensor 1 Sensor 2 Virtual sensor Behavior control only Feedback Planning only Eavesdrop

9. Skill Vs. Behaviors Not purely reflexive: Reflexive (response to stimulus) Innate (virtual sensor turns behavior on or off) “If power is low, charge” Learned Retain feedback to determine best behavior sequence to instantiate next time More complex emergent behaviors: Behavior sequences

10. Connotations of Global “Global” isn’t always truly global in Hybrids. Behavioral Management Planning which behaviors to use requires knowledge about current and future world state Performance monitoring Detecting task progress and sensor confliction require knowledge about the robot hardware and the overall goals. Nonetheless

11. Common Components Sequencer Generates a sequence of behaviors Resource Manager Allocates resources to behaviors Cartographer Creates, stores, maintains, accesses map information Mission Planner Interact with human and create a plan to achieve a goal Performance Monitor/problem solver Determines whether the robot is making progress toward its goal

12. Architecture Styles Managerial (division of responsibility as in business) AuRA SFX State Hierarchies (strictly by time scope) 3T Model-Oriented (Model serve as virtual sensors) Saphira TCA

13. Styles of hybrid architectures ● Managerial styles ● State hierarchies styles ● Model-oriented styles

14. Managerial Architectures Description -- top agents – high level planning ↓ subordinate agents – refine plan, gather resources ↓ lowest level agents ▲ AuRA Architectures ▲ SFX Architectures

15. ▲ Autonomous Robot Architecture (AuRA) It consists of five subsystems -- planner : responsible for mission and task planning -- cartographer : all map making, reading functions -- motor : motor schema -- sensor -- homeostatic control : modify the relationship between behaviors by changing the gain as a function of robot or other constraints

16. AuRA Architectural Layout

17. The table below summarizes AuRA in term of the common components and style of emergent behavior AuRA Summary Sequencer Agent Navigator, Pilot Resource Manager Motor Schema Manager Cartographer Mission Planner Performance Monitoring Agent Pilot, Navigator, Mission Planner Emergent Behavior Vector summation, spreading activation of behaviors, homeostatic control

18. ▲ Sensor Fusion Effects (SFX) description – It is an extension to AuRA. The extension was to add modules to specify how sensing and handling sensor failure.

19. Deliberative layers -- Mission planner : acts as a CEO giving a directions -- effector -- Task -- Sensor All of three of above determine the best allocation of effect, sensing resource and perceptual schema. -- Cartographer : map making, path planning

20. SFX (Sensor Fusion Effects) Behaviors (using direct perception, fusion) Sense Muscle Actuators Deliberative Layer Managers Sense Sensor Sense Receptive Field Choice of behaviors, resource allocation, motivation, context Focus of attention, recalibration Sensor Whiteboard Behavioral Whiteboard Deliberative Layer Reactive Layer Parameters to behaviors, sensor failures, task progress actions Superior Colliculus-like functions Cerebral Cortex-like functions Cartographer (model/map making) Recognition perception

21. Reactive layers All these layers reflect to ------- strategic behaviors and tactical behaviors Tactical behavior serves as filter on strategic commands to ensure to robot acts in a safe manner in as close accordance with the strategic intent as possible the interaction of strategic and tactical behaviors is still considered emergent behavior

22. Tactical Behaviors

23. The table below summarizes SFX in term of the common components and style of emergent behavior SFX Summary Sequencer Agent Task Manager Resource Manager Sensing and Task Manager Cartographer Mission Planner Performance Monitoring Agent Performance Monitor, Habitat Monitor Emergent Behavior Strategic behaviors grouped into abstract behaviors or scripts, then filtered by tactical behaviors

24. State-hierarchy Architectures (3 layers)

25. ▲ 3 – tiered (3T) Used for : planetary rovers underwater vehicles robot assistants for astronauts

26. Structure -- planner : setting goal and strategic plans -- sequencer : select a set of primetive behaviors develop a task network -- skill manager : in this layer the skills have associated events to verify explicitly that an action has had to correct effect

27. 3T Architecture

28. The table below summarizes 3T in term of the common components and style of emergent behavior 3T Sequencer Agent Sequencer Resource Manager Sequencer (Agenda) Cartographer Planner Mission Planner Planner Performance Monitoring Agent Planner Emergent Behavior Behaviors grouped into skills, skills grouped into task network

29. Model-oriented Architectures two of best-known model-oriented architecture ▲ Saphira architecture ▲ Task Control Architecture

30. ▲ Saphira Architecture -- PRS-Lite it is capable of taking natural language voice commands from humans and then operationalizing that into navigation tasks and perceptual recognition routines. -- virtual sensor -- navigation tasks manage the behaviors -- LPS (Local Perceptual Space) determine the planning and execution improve the quality of the robot ’ s overall behavior

31. Saphira Architecture

32. The table below summarizes Saphira in term of the common components and style of emergent behavior Saphira Sequencer Agent Topological planner, Navigation Tasks Resource Manager PRS-Lite Cartographer LPS Mission Planner PRS-Lite Performance Monitoring Agent PRS-Lite Emergent Behavior Behaviors fused with fuzzy logic

33. ▲ Task Control Architecture (TCA) -- Task Scheduling (Mission Planner) determine the goal and order of execution -- Path Planning (Cartographer) -- Navigation (Sequencer) to determine what the robot should be looking for, where it is, where it has been. -- Obstacle Avoidance To factor in not only obstacle but how to respond with a smooth trajectory for the robot ’ s current velocity.

34. TCA

35. The table below summarizes TCA in term of the common components and style of emergent behavior TCA Sequencer Agent Navigation Layer Resource Manager Navigation Layer Cartographer Path-Planning Layer Mission Planner Task Scheduling Layer Performance Monitoring Agent Navigation, Path-Planning, Task- Scheduling Emergent Behavior Filtering

36. Basic Important concept Paradigm Paradigm is both a way of looking at the world and an implied set of tools for solving problems. Sense, Plan, Act. Commonly accepted robotic primitives. Robotics have to go through these three, or at least two process to complete a mission. Local Processing and Global World Model Local: sensor data used in specific for each function. Global: all sensor data is processed to single model.

37. Hierarchical Paradigm What are the two main features? Robot operates in a top-down fashion. All sensor data tends to be gathered to one global world model. A single representation that planner can use to rout the action. SENSEPLAN ACT

38. Reactive Paradigm What are the two main features? Throw out planning all together. The inputs to an act are the direct output of a sensors. examine living example of intelligence. SENSE ACT

39. Hybrid Paradigm Features of Hybrid Deliberative/Reactive Paradigm It is reactive planning, Planning to subtask is done at one step. Deliberative planning take a long time comparing to the time of reactive execution Sensor data go directly to each behavior but is also available to the planner for construction of task-oriented global world model. Model-based Architecture focuses on the creation and maintenance of a global world model.

40. Hybrid Paradigm The basic models of Hybrid Paradigm Sequencer: generate a set of behaviors for subtasks. Resource manger: allocate resources to behavior Cartographer: for creating, storing, maintaining map or spatial information. Mission Planner: interact with man, construct a mission plan. Performance Monitoring: monitor the process of the executing, It’s self-awareness.

41. Hybrid Paradigm ACT SENSE Plan

42. Hybrid Paradigm Robot Primitive Input output PLAN Directives BEHAVIOR Information( sensed and cognitive ) Sensed data Actuator command

43. Other Hybrid Paradigm DARPA UGV Demo II and Demo III. Outdoor ground vehicle control and navigation. given a map and a set of directions find enemy location. Reach in automating highway vehicles by European Community ESPRIT agency and some United States agency Autonomous planetary rovers by NASA. Mapping planetary surface, planning path.

44. Advantages of Hybrid Architecture is highly modular Architecture is highly modular of the deliberative with object-oriented programming. Full knowledge of environment Software agents can use agent-specific abstractions to exploit the structure of an environment in order to fulfill their particular role in deliberation. Use of Global models Global models are only for symbolic functions and Planners( sequencers) often produce partial plans.

45. Advantages of Hybrid Execution is reactive. No frame problems. In the Hybrid Paradigm almost no the frame problems resulted by the Hierarchical. Self-consciousness. Ensure robustness by monitoring the performance of the robot and self-diagnosing, this is called self-consciousness.

46. Examples For Good of The Reactive Example1 we don’t need to turn all sensed data to global model to use in order to accuracy, convince, reliability, and saving time. Example 2 in Hierarchical Paradigm it is unwise in a lot of practical problems to block out the sensed data to Behaviors( Actuator).

47. II.f LED Sensor 1 Sensor 2 Sensor 3Pressure Sensor A/DD/A A/D CPU

48. Gas Sensor 1 Alarm CPU A/D D/A A/D

49. Interleaving Deliberation and Reactive Control For navigation Deliberation: Cartographer( planner) generates a complete optimal route, decompose the route to segments-waypoints. Reactive Control: Waypoint can be accomplished by behaviors. Top-down method Deliberative layers decompose the missions to finer steps. Reactive layers accomplish the first sub-goal.

50. Bottom-up method. Deliberative layers act as virtual sensors. The analyzed information as a sensed data input into behaviors( reactive layers)-Bottom-up Other functions of Deliberations In the deliberative layers, sequencer must know why a failure and know the need to change the behaviors and alert the human supervisor.-self- consciousness. Interleaving Deliberation and Reactive Control

51. Summary of AI Robotics

52. What is intelligent robots? What is the difference between AI and Engineering approaches to robotics? What is the difference between telepresence and semi-autonomous control? Ch.1: From Teleoperation to Autonomy

53. What is intelligent robots? Mechanical creatures that can function autonomously, which means it can sense, act, maybe even reason; doesn ’ t just do the same thing over and over like automation. The intelligent robots arose by the development of AI since the 1990 ’ s.

54. Teleoperation Teleoperation is that a human operator controls a robot from a distance. It is a ideal solution for controlling remotes because AI technology is nowhere near human levels of competence, especially in terms of perception and decision making. Cons: Cognitive fatigues; communications dropout; communications bandwidth; communications lag;

55. Add more intelligence to the early teleoperation Telepresence –providing sensory feedback to the point that teleoperator feels they are “ present ” in robot ’ s environment by adding more cameras. Semi-autonomous control –human is involved, but routine or “ safe ” portions of the task are handled autonomously by the robot –It is really a type of mixed-initiative

56. The Seven Areas of AI knowledge representation –How does the robot represent its world, task, and itself. understanding natural language –Natural language is usually challenging, it is not only talking about looking up words from a dictionary by understanding. Learning –A robot could be programmed by just watching a human ’ s behaviors.

57. The Seven Areas of AI planning and problem solving –The ability to plan actions and solve problems with those plans Inference –Inference is generating an answer when there is no complete information Search –Search means efficiently examining a knowledge representation of a problem to find the answer. Vision –The robot can simulate the effects of actions in its “ head ”

58. Robotics Paradigms What are robotic paradigms? –A paradigm is a philosophy or set of assumption and/or techniques which characterize an approach to a class of problems. There paradigms: –Hierarchical paradigm (Ch. 2) –Reactive paradigm (Ch. 4) –Hybrid paradigm (Ch. 7)

59. Ch. 2: Hierarchical paradigm The oldest paradigm, and was prevalent from 1967-1990. Under this paradigm, the robot senses the world, plans the next action, and then acts. PLANSENSEACT

60. Strips: means-ends analysis Strips is a variant of the general problem solver method, it uses an approach of means-ends analysis, where if the robot can ’ t accomplish the task in one “ movement ”, it picks a action which will reduce the difference between what the now state versus the goal state. To implement Strips, Designer must set up –World model representation –Difference table with operators, preconditions, add & delete lists –Difference evaluator

61. Strips: means-ends analysis Strips assumes closed world –Closed world: world model contains everything needed for robot (implication is that it doesn ’ t change) –Open world: world is dynamic and world model may not be complete Strips suffers from frame problem –Frame problem: representation grows too large to reasonably operate over

62. Representative Architecture An architecture is a method of implementing a paradigm, of embodying the principles in some concrete way. The two best known architectures are the Nested Hierarchical Controller (NHC) developed by Meystel and the NIST Realtime Control System (RCS) originally developed by Albus.

63. support for modularity: –decomposition by functionality niche targetability: –good, both have been used for apps like vehicle guidance, mining equipment ease of portability to other domains: –unclear, not sure if code could be reused — lots of rewriting on previous apps robustness: –RCA simulates plans in advance, but not sure what it would do with sensor or mechanical failures, etc. Evaluating the Two Architectures

64. Advantages and Disadvantages Advantages: –It provides an ordering of the relationship between sensing, planning, and acting. Disadvantages: –Planning: for every update cycle, robots had to do some type of planning. –Dependence on a global world model –Uncertainty: did the robots actually finish the action? We don’t know for sure.

65. Ch. 3: Biological Foundations of the Reactive Paradigm Why explore the biological sciences? What are the three levels in a computational theory? What are animal behaviors? Coordination of behaviors, perception, schema theory, and more…

66. Why do we need to explore the biological sciences? Animals and man provide existence proofs of different aspects of intelligence. The principles of animal intelligence are extremely important. –For examples: roboticists may overcome the closed world assumption that presented problems with shakey by observing the animals behaviors in an open world.

67. Marr ’ s Computational Theory The levels in the computation theory can be stated as: Level 1: What is the phenomena we’re trying to represent? Level 2: How it be represented as a process with inputs/outputs? Level 3: How is it implemented?

68. Animal Behaviors A behavior is a mapping of sensory inputs to a pattern of motor actions which then are used to finish a task Three catagories: –Reflexive stimulus-response, often abbreviated S-R –Reactive learned or “ muscle memory ” –Conscious deliberately stringing together

69. Coordination and Control of Behaviors There are four ways to acquire a behavior, which are: To be born with a behavior (innate) –Examples: Arctic terns. To be born with a sequence of innate behaviors. –Examples: mating cycle in digger wasps. To be born with behaviors that need some initialization (innate with memory). –Examples: bees, which are born with in hives. To learn a set of behaviors –Examples: Lions, who are nor born with any hunting behaviors.

70. How behaviors are coordinated and controlled -- innate releasing mechanisms (IRM) BEHAVIOR Sensory Input Pattern of Motor Actions Releaser The Releaser acts as a control signal to activate a behavior. If a behavior is not released, it does not respond to sensory inputs.

71. Perception Two functions of perception (can be the same percept) –Release a behavior –Guide a behavior Action-oriented perception (Neisser) –Planning is not needed to act –Perception is selective Cognitive Activity World Perception of Environment Samples, Finds Potential Actions Acts & Modifies World Directs what to look for

72. Schema Theory Schema theory provides a helpful way of casting some of the insights from above into an OOP format. is generic, equivalent to an object in OOP –schema specific knowledge (local data) –procedural knowledge (methods) schema intiantation is specific to a situation, equivalent to an instance in OOP a behavior is a schema, consists of –perceptual schema –motor schema

73. Ch. 3: Summary A behavior is the fundamental element of biological intelligence, and will server as the fundamental component of intelligence in most robot systems. Innate Releasing Mechanisms (IRM) are one model of how intelligence is organized. Perception in behaviors serves two roles, including a releaser for a behavior and a precept which guides the behavior. Schema theory is an object-oriented way of representing and thinking about behaviors.

74. Ch. 4: The Reactive Paradigm The Reactive Paradigm was a reaction to the Hierarchical Paradigm, and it was heavily used between 1988-1992. The fast execution time can be achieved by throwing away “Planning”. SENSEACT RELEASER behavior

75. Reactive Robots Most apps are programmed with this paradigm Biologically based: –Behaviors (independent processes), released by perceptual or internal events (state) –No world models or long term memory –Highly modular, generic –Overall behavior emerges SENSEACT RELEASER behavior

76. Hierarchical Organization is “ Horizontal ” Horizontal decomposition of tasks into the S, P, A organization of the Hierarchical Paradigm.

77. More Biological is “ Vertical ” The right figure shows that a vertical decomposition of tasks into an S-A orgrnization.

78. Architectures Historically, there are two main styles of creating a reactive system: –Subsumption architecture Layers of behavioral competence How to control relationships –Potential fields Concurrent behaviors How to navigate

79. Subsumption Architecture Subsumption has a loose definition of behavior as a tight coupling of sensing and acting. Higher layes may subsume and inhibit behaviors in lower layers. The design of layers and their behaviors is usually difficult. Behaviors are released by the presence of stimulus. Subsumption solves the frame problem by eliminating the need to model the world because the behaviors just simply respond to whatever stimulus is in the environment. Perception is largely direct, using affordances. Perception is ego-centric and distributed.

80. Potential Fields Potential field styles of behaviors always use vectors to represent behaviors and vector summation to combine vectors from different behaviors to produce an emergent behavior. Behaviors are defined as consisting of one or more of both motor and perceptual schemas and (or) behaviors. All behaviors operate concurrently and output vectors are summed. Behaviors may make varying contributions to the overall action of the robot, although they are treated equally. Perception is usually handled by direct perception or affordances. Perception can be shared by multiple behaviors.

81. Evaluation of Reactive Architectures Support for modularity –Both decompose the actions and perceptions. Subsumption favors a composition suited for a hardware implementation, whereas potential fields methods for a software-oriented system. Niche targetability –Both have hign targetabilities. Ease of portability to other domains –Subsumption depends on low layers heavily, while potential fields usually have no implicit reliance on a low layer. Robustness –Neither can be called genuinely robust.

82. Ch. 4: Summary The organization of the Reactive Poradigm is SENSE- ACT, No PLAN component. Under reactive paradigm, behaviors serve as the basic building blocks for robot actions. Reactive systems also exhibit good software engineering principles due to the programming by behavior approach. At last, two representative architectures are subsumption and potential fields. However, despite the differences in theory, these two systems appear to be largely equivalent practically.

83. The key points to understand what is main characters of AI robotics? OOP (Object-Oriented Programming) Model of sensing Hybrid deliberative/Reactive Paradigm Example of our homework#3 Future of Robot

84. What is OOP? Object-Oriented Concepts tap into this natural human tendency resulting in an easy to understand and use language. An automobile is a very good example of the Object- Oriented Concept. As humans, it is our natural tendency to think of an automobile as a single "thing", and not as a large group of several thousand small "things". Thinking of the automobile as a single "thing" helps us deal with the overwhelming complexity of the whole machine. We would say simple statements like; "Fill her up.“ or "How fast are we going?" or "I have a Blue car. "..... and everyone would understand how those statements apply to our car.

85. 1. Example for OOP Programming Using an automobile as an example of an Object, the following program shows an example of Object Oriented programming: BobsCar.Speed = 50 If BobsCar.Speed>CurrentRoad.SpeedLimit Then PoliceCar.Mode = Chase PoliceCar.Target = BobsCar PoliceCar.Speed = BobsCar.Speed + 10 End If Is it very simple and easy to understand? Here, please imagine that if we do not use OOP, what should our program look like?

86. 2. How behaviors can be implemented using OOP constructs such as classes? Recall from software engineering that an object consists of data and method, also called attributes and operations. And as noted before, schemas contain specific knowledge and local data structures and other schemas. So, a schema as a programming object will be a class. It’s defined as below:

87. 3. Example: move-to-go behavior 1) We put a robot in an empty arena with Coca-cola cans in random location and a blue recycling bin in a corner. 2) The behaviors needed is picking up a red can and moving to a blue bin. But we write a single generic behavior move_to_goal (color) to deal with both behaviors. 3)The behavior move_to_goal consist of a perceptual schema, which will be called extract-goal and a motor schema, which used an attractive field. extract-goal uses the affordance of color to extract where the goal is in the image, and then computer the angle to the center of the colored region and size of the region. The table below implies some important points about programming with behaviors:

88. Object Behavioral AnalogIdentifier DataPerceptgoal_angle goal_strength MethodPerceptual_schema Motor_schema extract_goal(goal_color) Pfield.attraction (goal_angle, goal_color) 4) The attraction motor schema takes that percept and is responsible for using it to turn the robot to center on the region and move forward. 5) Two schemas are both independent. The perceptual schema doesn’t know the existence of motor schema.

89. 1. Model of sensing environment Sensor Observation Or Image Perceptual Schema Motor Schema Robot Action Percept Sensor/transducer---------->Behavior------------->Action

90. 2. Behavioral Sensor Fusion Sensor FusionBehavior Perception in a reactive robot system has two roles: 1)to release a behavior 2)to support or guide the action of the behavior All sensing is behavior-specific, where behaviors map tap into the same sensors, but use the data independently of each other.

91. The Hybrid Deliberative /Reactive Paradigm 1. It can be thought as PLAN, then SENSE—ACT. 2. The SENCE—ACT portion is always done with reactive behaviors, where PLAN includes a broader range of intelligent activities. 3. Planning can be interviewed with execution. 4. Architecture usually encapsulate functionality into modules. The basic modules are: mission planner, behavior manager, performance monitor. 5. State-hierarchies divide deliberation and reaction by the state, available to the modules or agents operating that layer. Three states are: Past, Present, Future.

92. Example Plan (the Algorithm we use) ś=f (s(i));δ=g (Ψ(s), Ψd(s)); s(i+1)=h (s(i)); Xd=f1(s(i)); Yd=f2(s(i)) Sense (Virtual Vehicle) Xd(s), Yd(s), Ψ(s), Ψd(s) ACT (Actual Robot) X(s+1), Y(s+1), Ψ(s+1), Ψd(s+1) Do we use PLAN—SENSE—ACT concept? Modules concept? State-hierarchies? Planning can be interviewed with execution?

93. Future of Robot Enabling technologies Enabling technologies ranging from sensors to radio communications and navigation aids are all accelerating logarithmically. The ubiquitous acceptance of wireless LAN systems, the plunging costs of video cameras and processors, the availability of affordable laser navigation systems, and the ever-increasing accuracy and dropping cost of GPS navigation receivers are all combining to make autonomous robots potentially cheaper and ever more capable. At least as important, we now have enormous resources

94. in human experience. Countless software engineers and academics have spent endless hours developing concepts of modeling and control that are just as much part of the existing robotics toolbox as any sensor or processor. As a result, only the integration of these elements is required for new robotic configurations to burst onto the scene with blinding speed. Social forces The social issues already discussed are pushing customers to look for new solutions to performing many of the tasks that now require manual labor. These are tasks which autonomous robots can easily provide. Slowly but surely, a few venture capitalists (real ones) are beginning to make investments in companies like iRobot, and the industry is beginning to gain a little attention.

95. Thank you for your time!