(C) 2001, Ernest L. Hall, University of Cincinnati

Course objective To provide a broad understanding of the use of industrial robots And an experience in specifying, designing and presenting a new robot application in oral and written formats. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

SYLLABUS DATE TOPIC Sep Realistic and Safe Use of Robots Sep Applications of Industrial Robots Project Sep Economic Justification Excel Template Oct Robot Implementation Oct Arm Configurations Quiz 1 Take Home Oct Wrist Configurations Oct End Effectors and Tooling Oct Methods of Actuation Oct Non-servo Operation Oct Servo Controlled Robots Oct Cell Control, Hierarchical Design Oct Performance Measures Sample Report 1 - Welding Sample Report 2 - Painting Sample Report 3 - Soldering Sample Report 4 - Batch Manufacturing Sample Report 5 - Machine Loading Nov, Joint Control Programming Nov Path Control Programming Nov High Level Languages Nov Simulation and Programming, Review Nov Vision and Sensor Systems Nov Work Cell Interfacing; REPORT DUE WED Nov Intelligent Robot Cells Nov Flexible Manufacturing 21. FINAL ORAL EXAM 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Objective Determine the relationship between various robot applications and the methods of interfacing available on commercial robots or automated guided vehicles. Be able to understand the common interface standards and their application in the control of industrial robots used in manufacturing applications. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Intelligent robots in the news
September 5th 2001 Intelligent robot to help brain surgeons 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

The future for many complicated areas of neurosurgery looks brighter with the unveiling today, 5th September 2001, at Church House, London, of PathFinder the world’s first intelligent robot for image-guided surgery, developed by UK-based Armstrong Healthcare Limited. Modern medical scanners enable surgeons to identify areas in the brain where a surgical intervention is needed. The PathFinder robot provides surgeons with a way of guiding instruments very precisely to the chosen site. This means that even tiny structures deep inside the brain can be accessed reliably, with minimal damage to surrounding tissue. Examples of ways in which Pathfinder might be used in the future include the treatment of brain tumours, Parkinson’s disease and epilepsy. New techniques of stem cell replacement therapy are also ideally suited to precision placement by the PathFinder robot. The surgeon instructs the PathFinder robot by marking a target and an approach path on the patient’s scan. PathFinder carries a camera that automatically matches the scanner image to the position of the patient’s head on the operating table. The surgeon makes a tiny hole at the entry point in the skull, and PathFinder then very gently advances an instrument through the hole to the chosen target. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

PathFinder has been under development for several years by Armstrong, which has produced a number of prototype systems for specialist centres. The production version shown today for the first time is about to start a programme of clinical trials at the Queens Medical Centre Nottingham. Following extensive testing and regulatory approval, it will shortly be used to help surgeons there in live operations. Patrick Finlay, Managing Director of Armstrong, says “PathFinder is the first robot with the intelligence to map-read a patient’s skull from a scanner image. It is designed to provide the neurosurgeon with a precision positioning device which is safe and simple to use in increasingly complex procedures.” Paul Byrne, a consultant neurosurgeon at Queens Medical Centre says “This development is a step forward in surgery, and should make difficult operations easier to bear. It should improve the prospects of treatment for certain categories of patient, and I am looking forward to evaluating it.” 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Robust Robots at Lawrence Livermore National Laboratory
These capabilities may be applied to many other U.S. industries requiring the handling of nuclear materials or hazardous chemicals, or working in hazardous environments associated with spill cleanup or fire fighting.. This technology can also benefit the biotechnology industry by isolating microbiological materials and processes, and the semiconductor manufacturing industry by automating ultraclean rooms. Applications LLNL has developed applications typifying the operational conditions under which the intelligent controls and hardened robot will be employed, including special nuclear material processing, hazardous waste handling, characterization and sorting, and waste re trieval operations. Pyrochemical processes have been demonstrated in an automated glovebox processing system, including calcination, direct oxide reduction, electrorefining, molten salt extraction, and button breakout and sampling. Visionbased object recognition is being used for robot programming and control in an automatic waste characterization and sorting system. Additional production demonstrations have been performed to support weapons disassembly and equipment servicing. Thus far robotics and automation have been selectively applied to these areas using surrogate materials; however, developments are under way directed at processing special nuclear materials in the near future. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

FORFEITING THE FUTURE Bill Joy
ACCUSTOMED TO living with routine scientific break-throughs, we have yet tocome to terms with the fact that the most compelling new technologies - robotics, genetic engineering and nanotechnology - pose a different kind of threat than the technologies that came before. Specifically, robots, engineered organisms and nanobots share a dangerous amplifying factor: they can self-replicate. A bomb is blown up only once, but one altered gene can become many, and quickly get out of control. While replication in a computer or a computer network can be a nuisance, at worst it disables a machine or takes down a network or network service. But self-replication in the new technologies runs a much greater risk: a risk of substantial damage in the physical world. Each of these new technologies also offers untold promise: the vision of near immortality; genetic engineering that may soon provide treatments, if not outright cures, for most diseases; and nanotechnology and nanomedicine which can apparently address yet more ills. Together these technologies could significantly extend our average life-span and improve the quality of our lives. Yet, with each of these technologies, a sequence of small, individually sensible advances leads to an accumulation of great power, and, concomitantly, great danger. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

RAID workstation The RAID workstation is an assistive robotic appliance for individuals with manipulative impairments. Typical uses include the manipulation of books, paper documents, computer media, drinks, telephones, audio equipment and similar items. RAID provides increased independence in both office and domestic environments for work and leisure activities. It is controlled from a personal computer using a number of optional input devices. Features A track-mounted robot with purpose-designed grippers for handling books, paper documents, computer media, drinks, telephones and similar objects Intrinsically safe design using low power drives Storage racks accessible to the robot for filing books, diskettes and paper documents A computer-controlled lectern, allowing access to successive pages of books or paper documents Microsoft Windows-based control software An optional infra-red link, providing computer access using a wheelchair-mounted joystick A selection of optional peripherals, including a printer, a scanner, a stapler, a fax modem and high capacity disk storage 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Autonomous Intelligent Machines
Autonomous Machines are those which are capable of responding & reacting without human assistance. Artificial Intelligence techniques can be used to give such machines an ability to carry out sophisticated tasks unaided and learning from experience in a changing environment. Frequently, these machines are built by embedding computational processors into to them, without the frills of conventional computers (eg no keyboards and displays !). Robots, air & spacecraft, undersea exploration vehicles and even portable telephones can be typical of this type of system. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

The NASA Telerobotics Program addresses the three specific mission and application areas: on-orbit assembly and servicing, science payload tending, and planetary surface robotics. Within each of these areas, the program supports the development of robotic component technologies, development of complete robots, and implementation of complete robotic systems focussed on the specific manipulation and mobility aspects of the mission needs. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

RCS Architecture – Jim Albus
Task or mission planning World model Sensory level Action level 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Intelligent Control Architectures
8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

DARPA ITO Sponsored Research 1999 Project Summary Software-Enabled Control for Intelligent Uninhabited Aerial Vehicles (UAVS) Georgia Institute of Technology The approach is to use a hierarchical control architecture where mission planning and situation awareness are at the highest level, the flight control (including stability control augmentation) is at the lowest level, and a mid-level controller coordinates the transitions between modes and provides fault-tolerant reconfigurable control. The flexible integration of these dynamic control system components and their reconfiguration and customization during flight are enabled by an underlying Open Control Platform. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Innovative Control Technologies for Autonomous Highly Agile and Extreme-Performance Aerial Vehicles
Internal Failures External Threats Situation Awareness NEW IDEAS Intelligent control methods for mode transitioning and fault tolerance for extreme-performance maneuvers in Uninhabited Aerial Vehicles (UAVs) On-line customizable and interchangeable control modules for rapid response to mission, mode, or operational changes Open control platform for real-time distributed object computing and self-adaptive software High-Level Reactive Control Fault Diagnosis Fault Tolerance Mode Selection Open Control Platform Reconfigurable Control Real Time Sensor Processing Mode Switching IMPACT Improved mission effectiveness for UAVs (smoother operation, extreme-performance maneuverability, and robustness to failures) Rapid response to mission or operational changes through on-line reconfigurable control algorithms for UAVs Increased interoperability among distributed, heterogeneous components Reduced development costs due to open, plug and play, component-based software platform SCHEDULE FY1999 FY2000 FY2001 FY2002 First Generation OCP OCP integrated into R50 hardware/ software system. OCP demonstrated through simulation; OCP released and delivered. Seedling SEC algorithms incorporated into OCP. On-line Customization of Mid-Level (ML) Control Algorithms Seedling SEC algorithms developed. On-line adaptive ML SEC algorithms developed. Algorithms validated in simulation. Algorithms integrated into OCP. Flight-Test Support and Simulation Evaluation Evaluate SEC algorithms &OCP through flight testing. Hardware-in-the-loop simulation developed. Integrated Simulation and Flight Test Demonstration 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati FlightSim extended w/ R50 dynamics. Algorithms & OCP integrated into FlightSim. Flight test.

Intelligent Control Intelligent control systems are classically constituted by three basic levels: the organisation level which organises sequences of complex actions in a long term memory; the coordination level which coordinates decision making and learning in a short term memory to generate subtask sequences to execute simple commands on the basis of real time information of the world; the execution level which performs the continuous-time control of the system 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Intelligent Control Intelligent Control is presently a well established field within the discipline of Control Systems. It represents a generalization of the concept of control, to include autonomous anthropomorphic interactions of a machine with the environment. It has been successfully represented by a Technical Committee of the IEEE Control Systems Society, and the technical results of many researchers in the area are regularly reported in many of the annual conferences in the USA and around the world. The recent appearance in the technical literature of monographs on the subject matter is a strong indication that the area has entered a period of maturity. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

George Saridis However, it was only in 1971, that Saridis made the first attempts to analytically investigate control systems with cognitive capabilities, that could successfully interact with the environment, and Albus presented his Cerebellar Model Articulation Controller, resembling a human behavioral control system. Since then, Intelligent Control, was postulated by Saridis, as the process of autonomous decision making in structured or unstructured environments, based on the interaction of the disciplines of Artificial Intelligence, Operations Research, and Automatic Controls. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

King Sun Fu The name Intelligent Control was coined by K.S. Fu in 1971, when he was asked to define the area beyond Adaptive and Learning Control. The presentation was part of debate at the time, about the evolution of the Theory of Control Systems. Tracking the chronological development one may go back to the 1940's and 50's where Classical Control was formulated, using frequency domain techniques. In the 1960's, with the discovery of Pontryagin's Maximum Principle, Optimal Control Theoly flourished. Stochastic Optimal Control, was a by-product of this theory, introducing the concept of uncertainty in the design process. In the late 1960's, when structural uncertainties were accepted as part of the systems to be controlled, Adaptive Control was introduced as the methodology to manage systems of higher sophistication. They were using implicit or explicit system identification to provide optimal decision making for the best performance. In the meanwhile considerable progress was made in the behavioral sciences, regarding the collection and use of information about the environment for decision making by humans. It was then only natural to apply behavioral techniques to Control System Theory to improve the performance of a system operating in an uncertain environment. The approach was called Learning Control, and it utilized methods considered as predecessors of the modern Neural Net Theory. Self-Organizing Controls were formalized in the same period, to handle cases of autonomous management of uncertain processes in unfamiliar environments. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Increasing Precision with Decreasing Intelligence
Intelligent Controls, follow the above definition, in order to manage complex processes in uncertain environments in an anthropomorphic manner, by using cognitive engineering systems and the tremendous power of modern computer technology. Typical examples are modern Intelligent Robotic Systems. They are usually stratified in three levels with possible multiple substates in each level. Saridis has proposed an analytic formulation of such a system, based on a Principle of Increasing Precision with Decreasing Intelligence. Such a machine is structured in three levels; the Organization level, the Coordinationl level, and the Execution level. They follow a hierarchical order of decrease of machine intelligence with an increase of complexity, for most efficient operation. Neural Net, Petri Net, and Optimal Control technologies have been utilized in these three levels, with Entropy as the common measure of performance. Albus et al., developed NASREM, which represents another successful Hierarchical Intelligent Control System that uses a behavioral approach. Several other Intelligent Control methods that are also described in this volume, have various applications, especially to autonomous Robotic Systems. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Coordination and Control of Multi-Agent Systems
This research seeks to develop, implement, and evaluate algorithms for coordination and control of multi-agent organizations in the context of distributed knowledge networks consisting of stationary as well as mobile intelligent software agents designed to support the utilization of heterogeneous, distributed data and knowledge sources for automated data-driven knowledge acquisition and decision support. Modular and open-ended design of distributed knowledge networks implies that the resulting system consists of multiple more or less autonomous intelligent agents with different capabilities. Each agent is responsible for a particular, usually fairly narrowly defined function. Effective use of such agents in distributed problem solving (e.g., in computer-aided scientific discovery in bioinformatics) intrusion detection in distributed computing systems), require mechanisms for control and coordination the behavior of individual agents in a way that leads to the desired global behaviors. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Intelligent Machines and Systems Technology-NIST
The Intelligent Systems Division's research and development program focuses on addressing both immediate and long-term industry needs for: Open-systems architecture standard Intelligent controllers for manufacturing industry and Government applications Engineering methodologies and software tools for building intelligent systems Test methods and metrics for measuring the performance of intelligent control systems 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Hierarchical Navigation System for Autonomous Locomotion and Manipulation Robot Hierarchical Adaptive and Learning Architecture System (HALAS) with Error Recovery Characteristics. Modification of Map and Robust Landmark Detection Based on Fuzzy Template Matching with Plus and Minus Primitives. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Nagoya HALAS System Architecture

Overview of ONR UCAV Project
Exploration of Hybrid and Intelligent Control Architectures in Conjunction with Probabilistic Verification (ONR N ) Overview of ONR UCAV Project S. Shankar Sastry Electronics Research Laboratory University of California, Berkeley 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Problem: Design of Intelligent Control Architectures for Distributed Multi-Agent Systems An architecture design problem for a distributed system begins with specified safety and efficiency objectives for each of the system missions (surveillance, reconnaissance, combat, transport) and aims to characterize control, observation and communication. Mission decomposition among different agents Task decomposition for each agent Inter-agent and agent—mother ship coordination Continuous control and mode switching logic for each agent Fault management This research attempts to develop fundamental techniques, theoretical understanding and software tools for distributed intelligent control architectures with UCAV as an example. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Fundamental Issues for Multi-Agent Systems
Central control paradigm breaks down when dealing with distributed multi-agent systems Complexity of communication, real-time performance Risk of single point failure Completely decentralized control Has the potential to increase safety, reliability and speed of response But lacks optimality and presents difficulty in mission and task decomposition Real-world environments Complex, spatially extended, dynamic, stochastic and largely unknown We propose a hierarchical perception and control architecture Fusion of the central control paradigm with autonomous intelligent systems Hierarchical or modular design to manage complexity Inter-agent and agent–ship coordination to achieve global performance Robust, adaptive and fault tolerant hybrid control design and verification Vision-based control and navigation 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Autonomous Control of Uninhabited Combat Air Vehicles
UCAV missions Surveillance, reconnaissance, combat, transport Problem characteristics Each UCAV must switch between different modes of operation Take-off, landing, hover, terrain following, target tracking, etc. Normal and faulted operation Individual UCAVs must coordinate with each other and with the mothership For safe and efficient execution of system-level tasks: surveillance, combat For fault identification and reconfiguration Autonomous surveillance, navigation and target tracking requires feedback coupling between hierarchies of observation and control 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Research Objectives: Design and Evaluation of Intelligent Control Architectures for Multi-agent Systems such as UCAV’s Research Thrusts Intelligent control architectures for coordinating multi-agent systems Decentralization for safety, reliability and speed of response Centralization for optimality Minimal coordination design Verification and design tools for intelligent control architectures Hybrid system synthesis and verification (deterministic and probabilistic) Perception and action hierarchies for vision-based control and navigation Hierarchical aggregation, wide-area surveillance, low-level perception Experimental Testbed Control of multiple coordinated semi-autonomous DV8 helicopters 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Methods Methods Formal Methods Hybrid systems (continuous and discrete event systems) Modeling Verification Synthesis Probabilistic verification Vision-based control Semi-Formal Methods Architecture design for distributed autonomous multi-agent systems Hybrid simulation Structural and parametric learning Real-time code generation Modularity to manage: Complexity Scalability Expansion 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Thrust 1: Intelligent Control Architectures
Research Thrust 1: Intelligent Control Architectures Coordinated multi-agent system Missions for the overall system: surveillance, combat, transportation Limited centralized control Individual agents implement individually optimal (linear, nonlinear, robust, adaptive) controllers and coordinate with others to obtain global information, execute global plan for surveillance/combat, and avoid conflicts Mobile communication and coordination systems Time-driven for dynamic positioning and stability Event-driven for maneuverability and agility Research issues Intrinsic models Supervisory control of discrete event systems Hybrid system formalism 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Decentralized Observation, Communication and Control for Multi-Agent Systems Given a strategic objective and local observation What are the required information protocols with Human-centered system and other autonomous agents to command tactical control? Given a distributed control problem and the local observation at each site, what is the inter-site communication (minimal) or coordination protocols required to solve this problem? Given a cooperative mission What is the strategic objective (possibly dynamic) of each autonomous agent? How to distribute among the available agents a specified centralized control problem? 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Decentralized Observation and Communication for Discrete Event Systems
Agent Communication Channels A1 A2 A3 Plant (Lp) The agents have partial observation but can exchange messages. The plant has a set of unobservable distinguished events (failures). OBJECTIVE: Design the inter-agent communication scheme required to detect and isolate the distinguished events 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Synthesis of Inter-agent Communication for Decentralized Observation
Theorem 1 (Lp, áåpoi, åfiñiÎI) is decentrally diagnosable if there exists n Î N such that for all sf Î åf, usfv Î Lp, |v| £ n, implies (w Î Lp) Ù ( i, Påpoi (w) = Påpoi (usfv)) Þ (sf Î w). If any two sufficiently long plant traces look the same to all the agents, then either they have no failures or have all the same failures. Synthesis: The communicate all plant observations solution works. General drawback: Redundant information is communicated. L(f) may not be regular even though Lp is regular. Current focus: Minimal communication, protocol synthesis, trace abstraction Documentation: Draft paper available and sent to WODES’98 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Decentralized Control of Discrete Event Systems Problem Formulation
Each agent has a set of controllable events Controllable events are a subset of the set of observable events The next event is either an uncontrollable event from the plant, a controllable event enabled by an agent, or a message event scheduled by an agent A control objective is specified by a language Investigate the existence and synthesis of coordination protocols 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Communication and Control Synthesis for DES models
Advantages: Will synthesize symbolic, event-driven, inter-agent communication over a finite message set Very simple models permitting logical or combinatorial analysis and insights AHS example: Worked for most coordinating maneuvers other than stability properties for vehicle following Limitation: No formal way to capture continuous dynamics The semantics of an event is generally some alignment or safety conditions in velocity, position, and euler angles with respect to targets or other agents SOLUTION: Distributed control of hybrid systems 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Intelligent Control Architecture
UCAV Control Architecture Intelligent Control Architecture Mission Planning Resource Allocation Mission Control Strategic Objective Generating Trajectory Constraints Fault Management Strategic Layer Inter-UCAV Coordination Trajectory Constraints Flight Mode Switching Trajectory Planning Sensor Info on Targets, UCAV’s Tactical Layer Trajectory Replan Trajectory Tracking Set Point Control Regulation Layer Environmental Sensors Actuator Commands Tracking errors UCAV Dynamics 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Preliminary Control Architecture for Coordinating UCAVs
Regulation Layer (fully autonomous) Control of UCAV actuators in different modes: stabilization and tracking Tactical Layer (fully autonomous) Safe and efficient trajectory generation, mode switching Strategic Layer (semi-autonomous) Generating trajectory constraints and influencing the tasks of other agents using UCAV-UCAV and UCAV-ship coordination for efficient Navigation, surveillance, conflict avoidance Fault management Weapons configuration Mission Control Layer (centralized) Mission planning, resource allocation, mission optimization, mission emergency response, pilot interface 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Thrust 2: Verification and Design Tools
Research Thrust 2: Verification and Design Tools The conceptual underpinning for intelligent multi-agent systems is the ability to verify sensory-motor hierarchies perform as expected Difficulties with existing approaches: Model checking approaches (algorithms) grow rapidly in computational complexity Deductive approaches are ad-hoc We are developing hybrid control synthesis approaches that solve the problem of verification by deriving pre-verified hybrid system. These algorithms are based on game-theory, hence worst-case safety criterion We are in the process of relaxing them to probabilistic specifications. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Thrust 2: Verification and Design Tools
Hybrid Control Synthesis and Verification Approach The heart of the approach is not to verify that every run of the hybrid system satisfies certain safety or liveness parameters, rather to ensure critical properties are satisfied with a certain safety critical probability Design Mode Verification (switching laws) To avoid unstable or unsafe states caused by mode switching (takeoff, hover, land, etc.) Faulted Mode Verification (detection and handling) To maintain integrity and safety, and ensure gradual degraded performance Probabilistic Verification (worst case vs. the mean behavior) To soften the verification of hybrid systems by rapprochement between Markov and Bayesian decision networks 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Controller Synthesis for Hybrid Systems
The key problem in the design of multi-modal or multi-agent hybrid control systems is a synthesis procedure. Our approach to controller synthesis is in the spirit of controller synthesis for automata as well as continuous robust controller synthesis. It is based on the notion of a game theoretic approach to hybrid control design. Synthesis procedure involves solution of Hamilton Jacobi equations for computation of safe sets. The systems that we apply the procedure to may be proven to be at best semi-decidable, but approximation procedures apply. 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Thrust 3: Perception and Action Hierarchies
Research Thrust 3: Perception and Action Hierarchies Design a perception and action hierarchy centered around the vision sensor to support surveillance, observation, and control functions Hierarchical vision for planning at different levels of control hierarchy Strategic or situational 3D scene description, tactical target recognition, tracking, and assessment, and guiding motor actions Control around the vision sensor Visual servoing and tracking, landing on moving platforms 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

What Vision Can Do for Control
Global situation scene description and assessment Estimating the 3D geometry of the scene, object and target locations, behavior of the objects Allows looking ahead in planning, anticipation of future events Provides additional information for multi-agent interaction Tactical target recognition and tracking Using model-based recognition to identify targets and objects, estimating the motion of these objects Allows greater flexibility and accuracy in tactical missions Provides the focus of attention in situation planning 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Relation between Control and Vision
The control architecture needs The vision system provides Situation, 3D scene description Target recognition Continuous control Object tracking Motion detection, and optical flow Guided motor action Higher level Task decomposition for each agent Inter-agent, agent—mother ship coordination Lower level Higher level visual processing: precise, global information, computational intensive Lower level visual processing: local information, fast, higher ambiguity 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Key Issues in Vision and Control: Deliver the Right Information at the Right Time How to coordinate the planning stage with sensing stage The planner should adjust to the speed and uncertainty of the vision system The vision system should optimize its information flow from the lower level to the higher level, given the need of the planner How to adjust the focus of attention Selecting attention of visual processing in terms of the object locations, as well as level of abstraction Fine tuning lower-level vision-motor control loop A well-designed lower-level vision-motor control alleviates computation requirements of higher-level visual processing 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Approach for Hierarchical Vision Processing
Use grouping to extract a compact description of the scene from lower processing Reduces the computation complexity of higher-level reasoning, provides a basis for attention selection Information estimated from “big picture” of the scene is less likely to be affected by noise in the sensor Efficient computation algorithm which is able to capture the “big picture” of a scene has been developed General results reported in CVPR’97, results on motion reported in ICCV’98 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Approach for Hierarchical Vision Processing
Applying higher-level reasoning on the groups extracted Model-based object recognition Matching image groups to object models 3D scene geometry estimation Based on the motion correspondence found Tracking and behavior analysis of objects Applying Bayesian theory in selecting the right level of visual processing 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Approach for Lower-level Vision—Motor Control
Vision-guided motor control Use low-level image, motion flow information in formulating motor control law Tracking in the 3D coordinates Use optical flow equations to build a model of the scene in 3D space Look-ahead control law to allow for visual processing time Tracking in the image plane (2D) Track objects (such as the landing pad) in image frame Relate image measurement (such as image location of the pad, curvature of the lane marker) to motor control law 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Research Contributions
Fundamental Research Contributions Design of hybrid control synthesis and verification tools that can be used for a wide range of real-time embedded systems Design of vision and control hierarchies for surveillance and navigation Hierarchical vision for planning at different levels of control hierarchy Control around the vision sensor Our multi-agent control architecture can be used for many applications ONR applications UCAVs, simulated battlefield environment, distributed command and control, automatic target recognition, decision support aids for human-centered systems, intelligent telemedical system General engineering applications Distributed communication systems, distributed power systems, air traffic management systems, intelligent vehicle highway systems, automotive control 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Research Schedule FY 98 FY 99 FY 00 O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J Intelligent Control Architectures Multi-Agent Decentralized Observation System Preliminary UCAV Architecture Final UCAV Architecture Performance Evaluation of UCAV Architecture Verification Tools Hybrid Control Synthesis Methods Probabilistic Verification Theory Software Tools for Synthesis and Verification Perception Smart Aerobots 3D Simulation Label Recognition C++ QNX Real-Time Visual Situation Assessment Terrain Following Control Scheme Vision System for Autonomous Takeoff/Landing Integrated System for Target Recognition and Terrain Following for Multiple UCAVs Public Tests Cal Day Demo April 17 Robotic Helicopter Competition Aug 12-13, Richland, WA Cal Day Demo Robotic Helicopter Competition UCAV Architecture Demo 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Deliverables Task Duration Deliverables Intelligent Control Architectures Specification Tools 7/97 - 7/98 software, technical reports Design Tools 7/97 - 9/99 software, technical reports Architecture Evaluation Environment 7/97 - 7/00 software, technical reports UCAV Application 7/97 - 7/00 experiments, technical reports Verification Tools Design Mode Verification 7/97 -12/98 software, technical reports Faulted Mode Verification 7/97 - 9/99 software, technical reports Probabilistic Verification 9/97 - 9/99 technical reports Perception Surveillance 7/97 - 9/99 software, experiments Hierarchical Vision 7/97 - 7/00 software, technical reports Visual Servoing 9/97 - 7/00 experiments, technical reports 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Measures of Program Success
FY97-98 Design of preliminary UCAV architeture Design of hybrid control synthesis methods Design of multi-agent decentralized observation system FY98-99 Development of probabilistic verification theory Final UCAV architecture design Vision and control for terrain following, take-off and landing for single UCAV FY99-00 Performance evaluation of UCAV architecture Integration of vision and control for multiple coordinating UCAVs Final version of the software tools for Hybrid control synthesis and verification, and Decentralized observation and control Demonstration of UCAV architecture using the helicopter testbed 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

FY97-98 Accomplishments Controller synthesis for hybrid systems. Developed algorithms and computational procedures for designing verified hybrid controllers optimizing multiple objectives Multi-agent decentralized observation problem. Designed inter-agent communication scheme to detect and isolate distinguished events in system dynamics SmartAerobots. 3D virtual environment simulation. Visualization tool for control schemes and vision algorithms—built on top of a simulation based on mathematical models of helicopter dynamics Label recognition: prototype in Matlab, then in C++ (QNX real-time) 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Berkeley Team Name Role Tel Shankar Sastry Principal (510) Investigator (510) (510) Jitendra Malik Co-Principal (510) Investigator Datta Godbole Research (510) Engineer (510) John Lygeros Postdoc (510) Jianbao Shi Postdoc (510) Omid Shakernia Graduate Student (510) 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Teaming and Interdependency
Collaboration with Prof. Varaiya (Berekeley) in designing a hierarchical control architecture for coordinating UCAVs Collaboration with Prof. Russell (Berkeley) in developing probabilistic design and analysis tools Collaboration with Prof. Zadeh (Berkeley) on soft computing tools for control of UCAVs and mode transition methods for DV 8 developed using fuzzy control Collaboration with Prof. Speyer (UCLA) on fault detection and handling methods Collaboration with Prof. Morse (Yale) on vision-guided navigation Informal conversations with Prof. Anderson (ANU), Prof. Hyland (Michigan) and visit to Naval Post Graduate School Pending: more formal collaborations with Profs. Narendra, Morse 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Robot requirements need to be determined for the application
Payload and working range Arm and wrist configuration End-effector required Method of actuation Operation (servo or non-servo) Precision required Special features Commercial units available 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

Configurations fit applications
Cartesian Application – assembly and machine loading Configuration – PPP Percentage – 18 Advantage – equal resolution, simple kinematics Disadvantage – Poor space utilization, slow speed 8/30/2018 (C) 2001, Ernest L. Hall, University of Cincinnati

(C) 2001, Ernest L. Hall, University of Cincinnati

Similar presentations

Presentation on theme: "(C) 2001, Ernest L. Hall, University of Cincinnati"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

(C) 2001, Ernest L. Hall, University of Cincinnati

Similar presentations

Presentation on theme: "(C) 2001, Ernest L. Hall, University of Cincinnati"— Presentation transcript:

Similar presentations

About project

Feedback