Monitoring, Modelling, and Predicting with Real-Time Control Dr Ian Oppermann Director, CSIRO ICT Centre

Understanding the world … in real time CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control It is so natural for us. Nothing is “natural” in Robotics How do we do it in “real time”? We focus on four components. Sensing Perception, Control Actuation

INS (GPS, IMU) Lidar / Radar Video / IR Temperature Pressure Sensing Current State Localization Mapping Object Tracking Change Detection Perception Path Planning Obstacle avoidance Relative (reactive) Absolute Control Hydraulic/Servos Locomotion (Wheels/Tracks) Manipulation (Arm/Gripper) Actuation Understanding the world : A Robot’s Viewpoint CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control Real time operations performed, from sensing to actuation, within the timeframe required for effective operation in a dynamic environment

Globally Consistent Real-time Perception LocalizationMapping Moving Object Tracking Change Detection Data Association Robust Optimization Sensing (Lidar. Radar, Stereo) Sensing (Lidar. Radar, Stereo) Semantics (Path, Obstacles) Semantics (Path, Obstacles) SLAM Long Term Mapping The difference between MOT and CD is in the time scale and whether the object is transitory or permanent The difference between Localization and MOT is the difference between recording the pose of ourselves versus recording the pose of somebody else Globally Consistent Model (map, trajectories etc.) CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Mobile Mapping: What’s available now? Big, Bulky and Expensive Rooftop-mounted sensors 2D lidars with high-end GPS/INS Cost: $ X00,000 Limited Environments Reliance on GPS which is challenged in urban canyons, underground, near large infrastructure, forests, mines Not Real Time Processing straightforward if accurate position is known at all times but NOT REAL TIME CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

What problems to address? Cost move from expensive hardware to software Accuracy improve location estimation loop closure (knowing where you have been) A dynamic environment Long term versus short term changes in environment Speed Real time means being able to sense, aggregate, decide or re-plan in the time frames affected by limits of safety, fuel, task completion constraints

CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control Mapping and Location Improving accuracy, reducing cost 3D Map created with spinning 2D LIDAR on Bobcat With Scan Matching (without any additional sensors) Data Association Robust Optimization Data Association Robust Optimization No GPS No odometry No encoders No IMU (inertial measurement Unit)

CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control Loop Closure – Improving accuracy With Place Recognition (loop closure) 2D LIDAR on moving 4WD Data Association Robust Optimization Data Association Robust Optimization

CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control Loop Closure Reliability and Accuracy over long times and long distances 2 Lasers No GPS No odometry No encoders No IMU No Calibration

Object Tracking Long term Tracking Under different conditionsPassive actuation CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Mapping Jenolan Caves CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Mapping Jenolan Caves Sample Point Cloud CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Mapping Jenolan Caves Watertight Surface Generation CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Mapping Jenolan Caves Chifley Cave Surface Model CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Mapping Jenolan Caves Registered Point Cloud CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Mapping Jenolan Caves Point Cloud Overlay CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example: Bringing it all together CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Example : Airborne Terrain Mapping and Static Object Avoidance CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

The Future? Connecting the Virtual to the Real (Robot) Mixed-Reality Tele-Robotic Robot Tele-Operation Machine Autonomy Shared Autonomy Manual User Interface Intelligent Behavior Extent of Knowledge Communications Latency Global Local Reactive Proactive Augmented Reality Augmented Virtuality Supervisory Assistive Real Virtual CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

The Future? Convergence of Technology Realistic Simulation of Environment. Realistic Visualization of Environment Realistic Interaction with Environment Planning CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Simulation and Computation Spectrum Off-line Analysis Multiple Scenarios CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

DataImportanceLatencyBandwidth Safety (automation)FailsafeLow Control & StateCriticalMed SensingDesirableVariableHigh Peer-to-Peer MBWA Peer-to-Peer WiMAX Backhaul Network WiFi/MiMo Mesh Access Network Whole of system view CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

The Challenge: Unified User Interface Command & Control Tele - RoboticSCADA Whole of Mine Planning by Paul BourkePaul Bourke CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Summary Automation and Teleoperation in Realtime Environments require: Knowing state of environment [Mapping] Knowing where you are [Localization] SLAM provides: Localization in GPS denied areas Mapping over large scales Independent real-time sensor Integration with other sensors (GPS,IMU etc) Future Realtime Automation will require: Pervasive Tracking SLAM-MOT (SLAM with Moving Object Tracking) Collaborative / Distributed real time mapping Deal with uncertainty, latency, trust and different sensing modalities Life long mapping and long term map management Scalable (semantic mapping) Maps get better with age, rather than “blurry CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

CSIRO – more than 80 years of achievements CSIRO. Monitoring, Modelling, and Predicting with Real-Time Control

Contact Us Phone: or Web: Thank you With Thanks to Dr Rob Zlot Dr Mike Bosse Dr Elliot Duff Dr Jonathan Roberts For further information: Dr Ian Oppermann Director, CSIRO ICT Centre Web: