1. COMP 417 – Jan 12 th, 2006 Guest Lecturer: David Meger Topic: Camera Networks for Robot Localization

2. Introduction Who am I? Overview, Camera Networks for Robot Localization What Where Why How (technical stuff)

3. Introduction - Hardware

4. Intro - What Previously: Localization is a key task for a robot. It’s typically achieved using the robot’s sensors and a map. Can “the environment” help with this?

5. Typical Robot Localization

6. Sensor Networks

8. Intro - Where In cases where there is sensing already in the environment, we can invert the direction of sensing. Where is this true? Buildings with security systems Public transportation areas (metro) More and more large cities (scary but true)

9. Intro – Why Advantages: In many cases sensors already exist Many robots operating in the same place, can all share the same sensors Computation can be done at a powerful central computer, saves robot computation Interesting research problem

10. Intro – How As the robot appears in images, we can use 3-D vision techniques to determine its position relative to the cameras What do we need to know about the cameras to make this work? Can we assume we know where the cameras are? Can we assume we know the camera properties?

11. Problem Can we use images from arbitrary cameras placed in unknown positions in the environment to help a robot navigate?

12. Proposed Method 1. Detect the robot 2. Measure the relative positions 3. Place the camera in the map 4. Move robot to the next camera 5. Repeat

13. Detection – An algorithm to detect these robots?

14. Detection (cont’d) Computer Vision techniques attempt detection of (moving) objects Background subtraction or image differencing Image templates Color matching Feature matching A robust algorithm for arbitrary robots is likely beyond current methods

15. Detection – Our Method

16. ARTag Markers

17. Proposed Method Detect the robot 2. Measure the relative positions 3. Place the camera in the map 4. Move robot to the next camera 5. Repeat

18. Position Measurement Question: Can we determine the 3-D position of an object relative to the camera from examining 2-D images? Hint: start from the introduction to Computer Vision from last time

19. Pinhole Camera Model

20. Camera Calibration An image depends on BOTH scene geometry and camera properties For example, zooming in and out and moving the object closer and farther have essentially the same effect Calibration means determining relevant camera properties (e.g. focal length f)

21. Projective Calibration Equations

22. Coordinate Transformation

23. Calibration Equations Matrix AT is a 3x4 and fully describes the geometry of image formation Given known object points M, and image points m, it is possible to solve for both A and T How many points are needed?

24. Calibration Targets

25. 3-Plane ARTag Target

26. Position Measurement Conclusion With enough image points whose 3-D location are known, measurement of coordinate transformation T is possible The process is more complicated than traditional sensing, but luckily, we only need to do it once per camera

27. Proposed Method Detect the robot Measure the relative positions 3. Place the camera in the map 4. Move robot to the next camera 5. Repeat

28. Mapping Camera Locations Given the robot’s position, a measurement of the relative position of the camera allows us to place it in our map Question: What affects the accuracy of this type of relative measurement?

29. Proposed Method Detect the robot Measure the relative positions Place the camera in the map 4. Move robot to the next camera 5. Repeat

30. Robot Motion A robot moves by using electric motors to turn its wheels. There are numerous strategies here in each of the important aspects: Physical Design Control algorithms Programming Interface High-level software architecture

31. Nomad Scout

32. Differential Drive Kinematics

33. Odometry Position Readings

34. Robot Motion - Specifics Robot control accomplished by using an in-house application – Robodaemon Allows “point and shoot” motion, not continuous control Graphical and programmatic interface to query robot odometry, send motion commands, collect sensor data

35. Proposed Method Detect the robot Measure the relative positions Place the camera in the map Move robot to the next camera Repeat Are we done?

36. Challenges In general, it’s impossible to know the robot or camera positions exactly. All measurements have error What should the robot do if the cameras can’t see the whole environment? I didn’t say anything about how the robot should decide where to go next More?

37. Mapping with Uncertainty Given exact knowledge of the robot’s position, mapping is possible Given a pre-built map, localization is possible What if neither are present? Is it realistic to assume they will be? If so, when?

38. Uncertainty in Robot Position In general, kinematics equations do not exactly predict robot locations Sources of error Wheel slippage Encoder quantization Manufacturing artifacts Uneven and terrain Rough/slippery/wet terrain

39. Typical Odometry Error

40. Simultaneous Localization and Mapping (SLAM) When both the robot and map features are uncertain, both must be estimated Progress can be made by viewing measurements as probability densities instead of precise quantities

41. SLAM Progress

42. SLAM (cont’d) A quantity of the work in robotics in the last 5-10 years has involved localization and SLAM, results are now very pleasing indoors with good sensing These methods apply to our system More on this later in the course, or after class today if you’re interested

43. Motion Planning The mapping framework described is dependant on the robot’s motion: The robot must pass in front of a camera in order to collect any images Numerous points are needed for each camera to perform calibration SLAM accuracy affected by order of camera visitation

44. Local and Global Planning Local: how should the robot move while in front of one camera, to collect the set of calibration images? Global: in which order should the cameras be visited?

45. Local Planning Modern calibration algorithms are quite good at estimating from noisy data, but there are some geometric considerations Field of view Detection accuracy Singularities in calibration equations

46. Local Planning We must avoid configurations where all points collected lie in a linear sub-space of R 3 For example, a set of images of a single plane moved only through translation, gives all co-planar points

47. Projective Calibration Equations

48. Global Planning Camera positions estimated by relative measurements from the robot This information is only as accurate as our knowledge about the robot “Re-localizing” is our only way to reduce error

49. Distance / Accuracy Tradeoff Returning to well-known cameras helps our position estimates but causes the robot to travel farther than necessary An intelligent strategy is needed to manage this tradeoff Some partial results so far, this is work in progress

50. Review Using sensors in the environment, we can localize a robot In order to use previously un-calibrated and unmapped cameras, a robot can carry out exploration, and SLAM This must only be done once, and then accurate localization is possible

51. Future Work Better motion planning strategies globally Integrate other sensing (especially if the cameras have blind spots) Lose the targets? Other types of ubiquitous sensing (wireless, motion detection, etc)