On Experimental Research in Sampling-based Motion Planning Roland Geraerts Workshop on Benchmarks in Robotics Research IROS 2006

Probabilistic Roadmap Method Construction (G =V,E ) Loop c  a free sample add c to the vertices V N c  a set of nodes for all c’ in N c in increasing distance if c’ and c are not connected in G then if local path between c and c’ exists then add the edge c’c to E Forbidden space Free space Sample c Colliding path c c’ c Local path c’ c c

Probabilistic Roadmap Method Construction (G =V,E ) Loop c  a free sample add c to the vertices V N c  a set of nodes for all c’ in N c in increasing distance if c’ and c are not connected in G then if local path between c and c’ exists then add the edge c’c to E Query connect sample s and g to roadmap Dijkstra’s shortest path Forbidden space Free space Sample Start / goal Local path Shortest path

Methods General setup –SAMPLE Implemented in C ++ using VS. NET 2003 Easy API to add techniques GUI : easily set up experiments Repeatability: load/save an experiment Easily comparing different techniques Easily examining parameter of a technique Automatically collect/process data of experiment –Demo

Methods Test problems –Conclusions were often too general due to limited set of problems –Also choose worst-case problems

Methods Interchangeability –Libraries taking take of common functionality Collision checking, visualization Callisto: [Nieuwenhuisen] Graph utilities Atlas: [Nieuwenhuisen] Nearest neighbor MPNN : [Yershova, Lavalle] Deterministic sampling methods [Yershova] Rotation in 3D [Kuffner]

Methods Interchangeability –Source code of motion planning framework Motion planning kit MPK : [Latombe] Move3D [Siméon] Motion strategy library MSL : [Lavalle] –Unfortunately, code is often not up-to-date

Methods Interchangeability –Sources Geometry of environment/robot: VRML Problem descriptions: XML –Advantages of using existing languages Well documented Parsers/type checkers are available for all platforms Existing programs for creating/editing the files

Methods Interchangeability –Sources of geometry files and benchmarks [ MOVIE ] [Amato] [Reggiani] –Problems should be put online when article is published

Results Evaluation of solution –Compare new technique with existing ones Pitfall: parameter tuning only for the new technique –Compare against optimal solution Often only known for trivial cases Approximate optimal solution by many runs –User studies

Results Statistics –Large variances in running times Complicates statistical analysis Makes analysis unreliable Is undesirable from a user’s point of view –Perform large number of runs –Provide more statistical info, e.g. box plots –Deterministic versus randomized techniques Deterministic techniques can respond sensitively to small changes in the problem setting

Conclusion Automate conducting experiments as much as possible Choose test problems carefully Source code, software components and problem data should be made available Use standard file formats ( VRML, XML ) Provide an extensive statistical analysis