An outline of the Grex system architecture and the tests at sea that will be performed in the Azores this summer Arvind A de Menezes Pereira Grad student at RESL, Viterbi School of Engineering, USC, Los Angeles and Summer Intern at ISR, IST, Lisbon

What is GREX anyway? Grex [Latin] (noun) crowd, herd, flock Source (Latin): Coordination and Control of Cooperating Heterogeneous Unmanned Systems in Uncertain Environments

GREX Partners ATLAS ELEKTRONIK, GmbHGermany Center of IMAR at Dept. of Oceanography and Fisheries at the University of the Azores Portugal InfremerFrance Innova S.p.A.Italy IST, ISRPortugal MC Marketing ConsultingGermany ORANGE ENERGY ConsultingPortugal SeeByte Ltd.United Kingdom Technical University IlmenauGermany Source :

GREX empowerment Quest for Hydrothermal vents Fish Data download Marine Habitat Mapping All vehicles depicted here are part of the GREX consortium. (Artists impression borrowed from GREX slides)

Sounds Interesting… But aren’t these vehicles VERY DIFFERENT??? That is exactly what we are trying to address with GREX. We are trying to synergize and enable strong team-driven capabilities among teams of robotic vehicles with different abilities & architectures. GREX will be a system that enables planning, communication, cooperative navigation, coordination and control of such vehicles from a seam-less “Team” perspective

What Hardware does a Vehicle usually have? In terms of Hardware: Power(usually batteries) Computer(s) & electronics Sensors (IMU, GPS, Doppler, Depth etc.) Actuators (thrusters, fins, rudders etc.) Communication equipment (Radios, Acoustic comms etc.)

What processes does a vehicle usually have? Navigation- Estimation of state Guidance- Commands to Autopilots Autopilots- Controllers for attitudes Controllers- Could involve all of the above as well as lower-level control Mission Executive – Executes vehicle mission Communications – Talks to the outside world Sensor Acquisition – Get data about the world Etc… etc…

Generalized Architecture for Unmanned Vehicles

So, can this “really” be done? Yes! - If we find ways to do: 1.Abstraction 2.Distributed behavior 3.Add safety checks 4.Develop Online Monitoring

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 1: Each vehicle uses its own computing platform/Operating System. How do we interface with it?

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 1: Each vehicle uses its own computing platform/Operating System. How do we interface with it? Solution : Find a way to convert between Team plans to Individual Vehicle Missions. We do not write software for each vehicle’s computer. Instead, GREX provides a small computer that runs software specific to team-related communications, control and navigation. Each vehicle’s GREX software knows how to interface with its vehicle computer which could be a Socket connection over TCP/IP or even a serial port.

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 2: How do we convert from Team Mission Plans to each Vehicle’s Individual Mission Plans?

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 2: How do we convert from Team Mission Plans to each Vehicle’s Individual Mission Plans? Solution : Define MVPs or Multi-Vehicle Primitives for higher level Team Plans and SVPs or Single-Vehicle Primitives for individual vehicle missions Each vehicle manufacturer translates from SVPs to their own internal mission representation

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 3: What if a vehicle does not have a capability?

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 3: What if a vehicle does not have a capability? Solution : GREX provides interfaces that can implement these capabilities if the vehicle is physically capable of doing them. Eg. path-following via autopilot commands can be provided. If the vehicle cannot physically perform such a maneuver it will REPORT its inability to do so and it will not take part in this part of the team mission.

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 4: Why use distributed Control?

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 4: Why use distributed Control? Solution : In centralized control failure in communication with the centralized system can paralyze functionality of the affected part. Scenarios in GREX may involve poor global communication although there may be good local communication which is suitable for achieving mission execution. Hence a distributed scheme makes much more sense!

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 5: What about safety?

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 5: What about safety? Solution: Checks are provided at almost every level. Each Vehicle’s interface module can choose if it wants to execute any instruction. Communication telegrams ensure everything is running according to plan. Abort conditions are implemented even at team level. Individual Vehicle aborts are left untouched for additional safety.

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 6: This does not look like it can react to anything new!

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring Can this “be” done? Problem 6: This does not look like it can react to anything new! Solution: GREX can be reactive! There are provisions for jumping to a different section of mission files, changing parameters such as time- outs or speed settings and even passing autopilot commands directly to vehicles if the vehicle provider so allows.

Yes! - If we find ways to do: 1.Abstraction 2.Distributed Behavior 3.Add safety checks 4.Develop Online Monitoring 5.and so on and so forth… Can this “be” done?

a GREX-enabled Vehicle

GREX-enabled Vehicles

GREX Team-Oriented Mission Planning Concept Mission Planning Software SeeTrack (by SeeByte) MVP-Pool GREX Meta-language Team Level GREX Meta-language Vehicle Level Languages of Real Vehicles GREX Team Mission Plan Mission Planning by Operator Mission Plan for Vehicle 1 Mission Plan for Vehicle 2 Mission Plan for Vehicle n … GREX Interface Module Veh.1 GREX Interface Module Veh.2 GREX Interface Module Veh.n to the vehicles … Mission Plan for Vehicle 1 Mission Plan for Vehicle 2 Mission Plan for Vehicle n Translation by Mission Planning Software Rules, Vocabulary Mission Plan for Vehicle n GREX Interface Module Veh.n Mission Plan for Vehicle n new MVPs Slide from Glotzbach, COMPIT´08

GREX Console Running SeeTrack Translation Process 2. start Vehicle Console TMP SVMP 4. Check Formal Language Verification SVMP 3. Translation SeeTrack SVMP TMP 1. create Operator 6. Translation into Real Vehicle Language by Vehicle Provider RVMP 7. Check Already existing checking process RVMP SVMP 5. Transfer via Network Link Vehicle Existing HWGREX HW TMPSVMPRVMP 8. T r a n s f e r v i a N e t w o r k L i n k Slide from Glotzbach, COMPIT´08

Mission Management Manoeuvre Management Auto Pilot TeamHandler Mission Management Manoeuvre Management Auto Pilot Vehicle 3 Mission Management Manoeuvre Management Auto Pilot Vehicle 2 Mission Management Manoeuvre Management Auto Pilot Vehicle 1 Team Handler modules Manoeuvre Management:  Mission Modification (Obstacle Avoidance)  Mission Adjustment (Typical Primitive Execution) Mission Management:  Mission Reorganisation (Handling of special situations) Auto Pilot (Team):  Mission Coordination (keeping of formation, inter vehicle collision avoidance) Slide from Glotzbach, COMPIT´08

Tests this Summer in the Azores Coordinated Path Following (July 25) GoToFormation maneuver (July 26) Coordinated Target Tracking (July 27) Azores Photos by Isabelmar (Flickr Creative Commons License) Participating Vehicles Aguas Vivas, ASV Delfim X and AUV Seabee (H2-IL mode)

Simple Path Following Desired Path is pre-defined Delfim X follows a mission consisting of pre- defined #ARC and #POINT (line) SVPs

Coordinated Path Following Path already provided Vehicles coordinate with leader vehicle to do coordinated path-following

GoToFormation MVP implementation to take vehicles to a starting formation Will use path/trajectory planning algorithms that do spatio- temporal de- confliction Obstacle avoidance is also required

Simple Target Tracking Agua Viva moves around while sending Delfim X its GPS locations Delfim X estimates path followed in GREX SVMP format and follows this path

Coordinated Target Tracking Delfim-X GREX- module’s estimator learns Agua Viva path and does a coordinated path- following with Seabee AUV using GREX MVPs

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Parameters : Epsilon

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Parameters :  Formation Parameters  Movement Parameters  Epsilon (Work in Progress)

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Parameters :  Primitive Constraints  Formation Parameters  Epsilon  Value Threshold  Additional Sensor String  Chase Leader

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Parameters :  Formation Parameters  Movement Parameters  Epsilon

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Important Single-Vehicle Primitives (SVPs):  #INIT  #POINT  #ARC  #REPORT Parameters :  Placeholder waypoint position  Min-depth, Max-depth, Sensor initializations  Epsilon

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Important Single-Vehicle Primitives (SVPs):  #INIT  #POINT  #ARC  #REPORT #POINT is used to specify waypoints and straight lines Parameters :  Lat/Long  Depth/Altitude, Velocity  WaypointType  WaitTime  MinDepth, MaxDepth  Sensor Configurations  Epsilon

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Important Single-Vehicle Primitives (SVPs):  #INIT  #POINT  #ARC  #REPORT #ARC is used to specify curves and the behavior along it Parameters :  Start Lat/Lon, End Lat/Lon, Center Lat/Lon  m/n mid-point Lat/Lons  Direction  Epsilon

Eg. Coordinated Target Tracking Important Multi-Vehicle Primitives (MVPs):  M_Init  M_GotoFormation  M_CoordinatedTargetTracking  M_Final Important Single-Vehicle Primitives (SVPs):  #INIT  #POINT  #ARC  #REPORT #REPORT is used to make the vehicle report or send messages to the main GREX computer or other vehicles. These messages will be in the form of Telegrams and their exact contents are decided by the TeamHandler

Obviously this is NOT enough… To do Target Tracking, we also need: Path-estimation Path-following Coordination Inter-Vehicle Communication Intra-Vehicle Communication

How do vehicles talk to each other? They do so by two ways: Data Telegrams (pre-defined data structs) General Telegrams It is up to each client to decide if it wants to send ‘synchronously’ or ‘asynchronously’

Some important data-structures THAVehicleData(THA request) VehicleCount NavTrackData(THA->TNAV) TNAVVehicleData(init for Navigation) VehicleNavData (Nav data other vehicle) TeamNavData(Nav data whole team) RangeData

Important Telegrams REQUEST_DATA START_MISSION, STOP_AND_KEEP_POSITION CHANGE_PARAMETER_SVMP CHANGE_SUBMANOEUVRE_SVMP JUMP_AFTERWARDS_TO_PRIMITIVE_SVMP VEHICLE_COUNT, NAV_TRACK_DATA, TNAV_VEHICLE_DATA…

IPC Communications

A brief description of the modules… The main GREX computer generates SVMPs from the TMP COM module is responsible for Inter-Vehicle communication The Team Handler on each GREX computer knows it is doing “Coordinated Target Tracking” or other tasks involving multiple vehicles We have a client-server architecture for IPC between GREX modules The GIM (GREX Interface Module) communicates information with the vehicle and also translates SVMPs to RVMPs (Real Vehicle Mission Plans) Team Navigation handles team-localization through info obtained through the COM client and GIM

General Overview for Coordinated Target Tracking Collect tracking-data from Comm module Filter, smooth and estimate arc and straight line reference paths Dispatch one arc/line at a time via the radio to other Team Handlers by using the CHANGE_PARAMETER_OF_SUBMANEOUVRE_SV MP and CHANGE_PARAMETER_OF_PRIMITIVE_SVMP telegrams to vehicle Team handlers Vehicle Team Handler may change mission parameters or change velocity/autopilot values through GIM Online commands and data are passed through Telegrams

Recap of GREX system 1.Central Planning for teams 2.Hardware/OS Abstraction 3.Provides for information and command flow across multiple vehicles 4.Central + Distributed Mission planning 5.Capable of complex coordination 6.Safety checks implemented at several levels and transmitted via status/emergency telegrams

Thank you…!