1. On Experimental Research in Sampling-based Motion Planning Roland Geraerts Workshop on Benchmarks in Robotics Research IROS 2006

2. Probabilistic Roadmap Method Construction (G =V,E ) Loop c  a free sample add c to the vertices V N c  a set of nodes for all c’ in N c in increasing distance if c’ and c are not connected in G then if local path between c and c’ exists then add the edge c’c to E Forbidden space Free space Sample c Colliding path c c’ c Local path c’ c c

3. Probabilistic Roadmap Method Construction (G =V,E ) Loop c  a free sample add c to the vertices V N c  a set of nodes for all c’ in N c in increasing distance if c’ and c are not connected in G then if local path between c and c’ exists then add the edge c’c to E Query connect sample s and g to roadmap Dijkstra’s shortest path Forbidden space Free space Sample Start / goal Local path Shortest path

4. Methods General setup –SAMPLE Implemented in C ++ using VS. NET 2003 Easy API to add techniques GUI : easily set up experiments Repeatability: load/save an experiment Easily comparing different techniques Easily examining parameter of a technique Automatically collect/process data of experiment –Demo

5. Methods Test problems –Conclusions were often too general due to limited set of problems –Also choose worst-case problems

6. Methods Interchangeability –Libraries taking take of common functionality Collision checking, visualization Callisto: http://www.cs.uu.nl/dennis/callisto/callisto.html [Nieuwenhuisen] Graph utilities Atlas: http://www.cs.uu.nl/dennis/atlas/atlas.html [Nieuwenhuisen] Nearest neighbor MPNN : http://msl.cs.uiuc.edu/~yershova/mpnn/mpnn.htm [Yershova, Lavalle] Deterministic sampling methods http://msl.cs.uiuc.edu/~yershova/so3sampling/so3sampling.htm [Yershova] Rotation in 3D http://www.kuffner.org/james/software [Kuffner]

7. Methods Interchangeability –Source code of motion planning framework Motion planning kit MPK : http://ai.stanford.edu/~mitul/mpk [Latombe] Move3D http://www.laas.fr/~nic/Move3D [Siméon] Motion strategy library MSL : http://msl.cs.uiuc.edu/msl [Lavalle] –Unfortunately, code is often not up-to-date

8. Methods Interchangeability –Sources Geometry of environment/robot: VRML Problem descriptions: XML –Advantages of using existing languages Well documented Parsers/type checkers are available for all platforms Existing programs for creating/editing the files

9. Methods Interchangeability –Sources of geometry files and benchmarks http://www.give-lab.cs.uu.nl/movie/moviemodels [ MOVIE ] http://faculty.cs.tamu.edu/amato/dsmft/benchmarks [Amato] http://mpb.ce.unipr.it/ [Reggiani] –Problems should be put online when article is published

10. Results Evaluation of solution –Compare new technique with existing ones Pitfall: parameter tuning only for the new technique –Compare against optimal solution Often only known for trivial cases Approximate optimal solution by many runs –User studies

11. Results Statistics –Large variances in running times Complicates statistical analysis Makes analysis unreliable Is undesirable from a user’s point of view –Perform large number of runs –Provide more statistical info, e.g. box plots –Deterministic versus randomized techniques Deterministic techniques can respond sensitively to small changes in the problem setting

12. Conclusion Automate conducting experiments as much as possible Choose test problems carefully Source code, software components and problem data should be made available Use standard file formats ( VRML, XML ) Provide an extensive statistical analysis