Intelligent off-road vehicles Martin Servin, Department of Physics,

Outline Background to the field Overview IFOR Autonomous navigation Crane automation Simulator based design Feel free to ask questions and make comments and proposals!

A sample of technological gems… Mars rover – extreme teleoperation Deep Blue – reasoning computer DARPA Grand Challenge – competition with autonomous vehicles QRIO – balancing robot Parthenon – virtual 3D reconstruction HCI – retinal display

The off-road challange Demand for new technology Increased productivity Increased safety and work environment Environmental sustainability The forestry challange Complex work processes to automate – no computer beats the human in running a harvester Rough environment with big variations – sensor vision in forest, robust and sustainable system handling vibrations, moist and dirt Vision from forestry industry “2025 – Ingen man i maskinen, ingen hand på spakarna “

An initiavive for R&D for intelligent off-road technology Initiated by the industry in 2001 Collaboration between academia and industry = a forum for R&D and a collection of projects focused at IFOR technology What is IFOR ? Academia: Umeå University Swedish University of Agricultural Sciences Skogforsk Industry: Komatsu Forest Holmen Skog Sveaskog BAE Systems Hägglunds LKAB + network of other research centers and companies

Technology vision Improved work environment Increased productivity and cut costs Increased safety Reduced environmental impact Technology:Goals: Control algorithms and modeling Interaction – man, machine and environment Sensor vision Localization and map building Automation of routine work processes Crane tip control Unmanned vehicles

Activities and projects Autonomous navigation Dr Thomas Hellström 1 PhD students Computing Science Department Smart Crane Control Prof Anton Shiriaev 1 FoAss, 1 PostDoc, 3 PhD Department of Applied Physics and Electronics Vehicle simulators Dr Martin Servin In collaboration with VRlab at UmU Miscellanious - Seminars and workshops - Experiments and pre-studies - Student projects Equipment Forest machines – valmet forwarder and harvester Full sized in-door hydraulic crane Portable prototyping hardware for feedback control Sensors (dgps, laser radar, hydraulic pressure, stereo camera,…) Simulator systems Funding > 25 MSEK since 2001 Kempe Foundation, Sveaskog, Vinnova, Komatsu Forest, Sparbanksstiftelsen Norrland, Umeå University, LKAB, BAE Systems Hägglunds Other actors SLU Skogforsk Applied Mathematics – Prof Mats G Larsson Design Institute UCIT / ProcessIT Innovations

Autonomous navigation Dr Thomas Hellström 1 PhD students - unmanned transportation of logs - localization, path tracking and path planning - RTK-DPGS with cm accuracy - laser scanners, radars,... - machine learning and sensor fusion - first prototype demonstrated in Dec ”Simulator in the loop” Autumn 2008 we are running the student DBT-projects: - Sensor vision and remote operation - Simulation of terrain vehicle with autonomous abilities

Smart Crane Control Prof Anton Shiriaev – Control System Theory 1 FoAss, 1 PostDoc, 3 PhD - motion planning, motion control for mechanical systems - feedback design for hydraulically actuated cranes - crane tip control - optimized motions – speed and stability - semi-automation, e.g. automatich loading - VR-enabled remote operation - portable prototyping hardware for feedback control Recent results: - motion faster and more stable than human operator – Valmet forwarder - demonstrated automatic loading in lab Grant from “Stiftelsen för strategisk forskning” for crane control using only hydraulic measurements at Komatsu Forest 1 industrial PhD have been granted (?) - Komatsu Forest and Umeå University splitting the costs – Semi-autonomous harvester control system

 Fast crane motion.avi Motion faster and more stable than human operator is possible!

 Virtual Environment Teleoperation Click control.avi Detection of rotating log.avi

Visual Simulation of Machine Concepts for Forest Biomass Harvesting Martin Servin, A. Backman, K. Bodin - Umeå University, Sweden U. Bergsten, D. Bergström, T. Nordfjell, I. Wästerlund - Swedish University of Agricultural Sciences, Sweden B. Löfgren - Skogforsk (the Forestry Research Institute of Sweden) VRIC 2008 – 10th International Conference on Virtual Reality (Laval Virtual)

Outline Simulator-based design Forest biomass harvesting –concept machine and work method Experiments in simulator environment –system and procedure –purpose: find optimal harvesting technique and machine design Training simulator technology – also for concieving new machines concepts and work methods

Simulator-based design (SBD) Simulation tools are converging – R&D process impoves – cross-disciplinary participation Extension of virtual prototyping and simulation to include human-in-the-loop Fast and sheap Simulators – complex yet controllable environments Figure from T. Alm ”Simulator-based design” (2007). End customer Manufacturer Designer Researcher Engineer Simulator training

Application of SBD to: Forest biomass harvesting Increasing demand for forest biomass Early harvesting/thinning is becoming profitable Large volumes and areas, small income per unit, energy consumption Crucial to use optimized technology – economically and environmentally sustainable Uncertain on what solution to choose for thinning Virtual and real prototypes are important!

New harvesting methods in dense forest stands Early harvesting = thinning + biomass harvesting - single-tree harvesting - multi-tree harvesting - geometric area based felling strip roads 3 m wide every m corridors 1x10 m 10 trees, 6 m, 50 kg collect in piles of 50 trees

Machine concept for harvesting in dense forest stands Size: 4x2 m, 2.5 ton, 8m reach Mobility: indv 4W on pendulum arms Harvester head: multi-tree vs blade Control and HMI: boom-tip control, semi-autonomous, teleoperation (direct or VE), laser scanner & stereo camera, dynamic 3D maps from sattelite and AUV

Experiments in simulator environment - system and procedure System components software: Colosseum3D (OSG, Vortex – AgX Multiphysics, lua,…) hardware: full simulator environment (screen projection, authentic chair and joysticks, motion platform) or portable case, convential multicore PC models: data from real forest stands in 3D terrain, vehicle = 20 rigid bodies coupled by kinemtaic constraints (wheel suspension, crane joints,…) vehicle automation and HMI module: vehicle control, automation, sensor, 3D-map engine and HMI interface The application requires advanced real-time physics: terramechanics, stacking, hydraulics,…

Experiments in simulator environment - system and procedure Experiment procedure Task: do harvest thinning in a given dense forest stand Variations: - forest stand (distribution, species, topology) - harvester head (single, multi, sword) - vehicle (existing machines, new proposals) - automation and HMI (manual, semi-automatic, fully auto) - operator Measurements: - time per biomass unit in kg (strip road, corridor, tree, move to pile, positioning, transport) - energy consumption - work environment Optimize: find optimal mechine design and work method – data from simulator experiments used in logistics computation