Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 An Application of Reinforcement Learning to Aerobatic Helicopter Greg McChesney Texas Tech University Apr 08, 2009 CS5331: Autonomous.

Published byScarlett Watts Modified over 9 years ago

Similar presentations

Presentation on theme: "1 An Application of Reinforcement Learning to Aerobatic Helicopter Greg McChesney Texas Tech University Apr 08, 2009 CS5331: Autonomous."— Presentation transcript:

1 1 An Application of Reinforcement Learning to Aerobatic Helicopter Greg McChesney Texas Tech University Greg.mcchesney@ttu.edu Apr 08, 2009 CS5331: Autonomous Mobile Robots

2 Overview  Creating a robot that can fly autonomously  Software developed at Stanford as part of their AI lab  This paper is slightly outdated as many new maneuvers have been created. Apr 08, 2009 CS5331: Autonomous Mobile Robots2

3 Learning Approach  Apprenticeship Collect data from human trying maneuver (multiple times) Learn a model from the data Find controller than can simulate based on model Test on helicopter (pray it doesn’t crash) Apr 08, 2009 CS5331: Autonomous Mobile Robots3

4 Helicopters State  Position  Velocity  Angular Velocity  Controlled with 4 dimensions Cyclic pitch Tail rotor  Take gravity out when calculating the model Apr 08, 2009 CS5331: Autonomous Mobile Robots4

5 Controller Design  Use a Markov decision process  Sextuple (S,A,T,H,s(0),R) S-set of states A-set of actions (inputs) T-dynamic model-set of probability distributions for the next state H-horizon or number of time steps of interest s(0)-initial state R-reward function Apr 08, 2009 CS5331: Autonomous Mobile Robots5

6 Differential Dynamic Programming(DDP)  Compute the linear approximation  Compute the optimal solution to the linear quadratic regulator Must take into account error state Cost for change in input-needed in real testing Apr 08, 2009 CS5331: Autonomous Mobile Robots6

7 DDP-Continued  2 phases DDP to find open loop input sequence Use DDP again refining the inputs as a deviation from the nominal open-loop input sequence  Integral control-take into account wind and errors in the model Apr 08, 2009 CS5331: Autonomous Mobile Robots7

8 Rewards  24 features  Used inverse reinforcement learning  Rewards from inverse reinforcement usually did not produce correct result  Took inverse results and manually tuned them to get good results Apr 08, 2009 CS5331: Autonomous Mobile Robots8

9 Helicopter  Xcell Tempest  54” long  19” high  13 lbs  Two-stroke engine  Orientation sensors  GPS-doesn’t work during flips Apr 08, 2009 CS5331: Autonomous Mobile Robots9

10 Apr 08, 2009 CS5331: Autonomous Mobile Robots10

11 Flip Apr 08, 2009 CS5331: Autonomous Mobile Robots11

12 Roll Apr 08, 2009 CS5331: Autonomous Mobile Robots12

13 Tail-In Funnel Apr 08, 2009 CS5331: Autonomous Mobile Robots13

14 Nose-In Funnel Apr 08, 2009 CS5331: Autonomous Mobile Robots14

15 Questions  Motivations/Who pays for it I can see applications in the defense sector DARPA  Could more maneuvers be done just by changing some parameters? Probably not because the filter is learned based on a model so you would need to create a new model Apr 08, 2009 CS5331: Autonomous Mobile Robots15

16 More Questions  What's the relationship between reinforcement learning and MDP? Not Sure  Could a helicopter like this operate in the West Texas wind storms? Apr 08, 2009 CS5331: Autonomous Mobile Robots16

17 Fun Stuff  Videos: http://heli.stanford.edu/ http://www.youtube.com/watch?v=VC dxqn0fcnE http://www.youtube.com/watch?v=VC dxqn0fcnE  Helicopter http://www.miniatureaircraftusa.com/h elicopterkits/1025_Spectra_G/1025_kit _main.asp http://www.miniatureaircraftusa.com/h elicopterkits/1025_Spectra_G/1025_kit _main.asp Apr 08, 2009 CS5331: Autonomous Mobile Robots17

Download ppt "1 An Application of Reinforcement Learning to Aerobatic Helicopter Greg McChesney Texas Tech University Apr 08, 2009 CS5331: Autonomous."

Similar presentations

Introductory Control Theory I400/B659: Intelligent robotics Kris Hauser.

Introductory Control Theory I400/B659: Intelligent robotics Kris Hauser.
David Rosen Goals  Overview of some of the big ideas in autonomous systems  Theme: Dynamical and stochastic systems lie at the intersection of mathematics.

David Rosen Goals  Overview of some of the big ideas in autonomous systems  Theme: Dynamical and stochastic systems lie at the intersection of mathematics.
Design of Experiments Lecture I

Design of Experiments Lecture I
Markov Decision Process

Markov Decision Process
Least squares CS1114

Least squares CS1114
1. Algorithms for Inverse Reinforcement Learning 2

1. Algorithms for Inverse Reinforcement Learning 2
NONLINEAR BACKSTEPPING CONTROL WITH OBSERVER DESIGN FOR A 4 ROTORS HELICOPTER L. Mederreg, F. Diaz and N. K. M’sirdi LRV Laboratoire de Robotique de Versailles,

NONLINEAR BACKSTEPPING CONTROL WITH OBSERVER DESIGN FOR A 4 ROTORS HELICOPTER L. Mederreg, F. Diaz and N. K. M’sirdi LRV Laboratoire de Robotique de Versailles,
Adam Coates, Pieter Abbeel, and Andrew Y. Ng Stanford University ICML 2008 Learning for Control from Multiple Demonstrations TexPoint fonts used in EMF.

Adam Coates, Pieter Abbeel, and Andrew Y. Ng Stanford University ICML 2008 Learning for Control from Multiple Demonstrations TexPoint fonts used in EMF.
Learning Parameterized Maneuvers for Autonomous Helicopter Flight Jie Tang, Arjun Singh, Nimbus Goehausen, Pieter Abbeel UC Berkeley.

Learning Parameterized Maneuvers for Autonomous Helicopter Flight Jie Tang, Arjun Singh, Nimbus Goehausen, Pieter Abbeel UC Berkeley.
Robust control Saba Rezvanian Fall-Winter 88.

Robust control Saba Rezvanian Fall-Winter 88.
Probabilistic Robotics Course Presentation Outline 1. Introduction 2. The Bayes Filter 3. Non Parametric Filters 4. Gausian Filters 5. EKF Map Based Localization.

Probabilistic Robotics Course Presentation Outline 1. Introduction 2. The Bayes Filter 3. Non Parametric Filters 4. Gausian Filters 5. EKF Map Based Localization.
Nonlinear Optimization for Optimal Control

Nonlinear Optimization for Optimal Control
What Are Partially Observable Markov Decision Processes and Why Might You Care? Bob Wall CS 536.

What Are Partially Observable Markov Decision Processes and Why Might You Care? Bob Wall CS 536.
Markov Decision Processes

Markov Decision Processes
Apprenticeship learning for robotic control Pieter Abbeel Stanford University Joint work with Andrew Y. Ng, Adam Coates, Morgan Quigley.

Apprenticeship learning for robotic control Pieter Abbeel Stanford University Joint work with Andrew Y. Ng, Adam Coates, Morgan Quigley.
Using Inaccurate Models in Reinforcement Learning Pieter Abbeel, Morgan Quigley and Andrew Y. Ng Stanford University.

Using Inaccurate Models in Reinforcement Learning Pieter Abbeel, Morgan Quigley and Andrew Y. Ng Stanford University.
Introduction to Kalman Filter and SLAM Ting-Wei Hsu 08/10/30.

Introduction to Kalman Filter and SLAM Ting-Wei Hsu 08/10/30.
Apprenticeship Learning by Inverse Reinforcement Learning Pieter Abbeel Andrew Y. Ng Stanford University.

Apprenticeship Learning by Inverse Reinforcement Learning Pieter Abbeel Andrew Y. Ng Stanford University.
An Application of Reinforcement Learning to Autonomous Helicopter Flight Pieter Abbeel, Adam Coates, Morgan Quigley and Andrew Y. Ng Stanford University.

An Application of Reinforcement Learning to Autonomous Helicopter Flight Pieter Abbeel, Adam Coates, Morgan Quigley and Andrew Y. Ng Stanford University.
Apprenticeship Learning for the Dynamics Model Overview  Challenges in reinforcement learning for complex physical systems such as helicopters:  Data.

Apprenticeship Learning for the Dynamics Model Overview  Challenges in reinforcement learning for complex physical systems such as helicopters:  Data.

Similar presentations

Ads by Google