for Vision-Based Navigation Landmark Selection for Vision-Based Navigation Pablo L. Sala Joint work with Robert Sim, Ali Shokoufandeh and Sven Dickinson To be presented in IROS 2004 September 17th, 2004

Robot Navigation Where am I? Where am I going? How do I get there? [Leonard and Durrant-Whyte] Where am I? Where am I going? How do I get there?

Robot Navigation Where am I? Localization Where am I going? [Leonard and Durrant-Whyte] Where am I? Localization Where am I going? How do I get there?

Robot Navigation Where am I? Localization [Leonard and Durrant-Whyte] Where am I? Localization Where am I going? Goal Identification How do I get there?

Robot Navigation Where am I? Localization [Leonard and Durrant-Whyte] Where am I? Localization Where am I going? Goal Identification How do I get there? Path-planning

Robot Navigation Where am I? Localization [Leonard and Durrant-Whyte] Where am I? Localization Where am I going? Goal Identification How do I get there? Path-planning

Landmark-Based Navigation What makes a good landmark?

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?)

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility How do we select good landmarks?

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility How do we select good landmarks? Manually

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility How do we select good landmarks? Manually Automatically

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility How do we select good landmarks? Manually Automatically… but how?

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility How do we select good landmarks? Manually Automatically Store every landmark visible at each location (costly!)

Landmark-Based Navigation What makes a good landmark? Distinctiveness (does it tell me where I am?) Wide Visibility How do we select good landmarks? Manually Automatically Store every landmark visible at each location (costly!) Find smallest subset of landmarks that supports reliable navigation (optimal!)

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Collection of images acquired at known discrete points in pose space. Pose recorded and image features extracted and stored in database.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Collection of images acquired at known discrete points in pose space. Pose recorded and image features extracted and stored in database.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Collection of images acquired at known discrete points in pose space. Pose recorded and image features extracted and stored in database.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Collection of images acquired at known discrete points in pose space. Pose recorded and image features extracted and stored in database.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Collection of images acquired at known discrete points in pose space. Pose recorded and image features extracted and stored in database.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Four features are needed in this set.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Four features are needed in this set. Only two features needed. Our goal is to find this decomposition.

View-Based Robot Navigation Off-line Exploration Landmark Database Construction On-line Localization Current pose is estimated using the locations of a small number of features in the current image, matched against their locations in two model views.

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

View-Based Robot Navigation

Intuitive Problem Formulation

Outline Problem Formulation Complexity Heuristic Methods Results on Synthetic and Real Images Conclusions

A Graph Theoretic Formulation Problem Definition: The -Minimum Overlapping Region Decomposition Problem (-MOVRDP) for a world instance <G=(V,E), F, {v} vV> consists of finding a minimum size -overlapping decomposition D = {R1, …, Rd} of V into regions such that:

A Graph Theoretic Formulation Problem Definition: The -Minimum Overlapping Region Decomposition Problem (-MOVRDP) for a world instance <G=(V,E), F, {v} vV> consists of finding a minimum size -overlapping decomposition D = {R1, …, Rd} of V into regions such that:   Theorem 1: A -MOVRDP can be reduced to an equivalent 0-MOVRDP, and the solution to this latter problem can be extended to a solution for the original problem.

A Graph Theoretic Formulation Problem Definition: The -Minimum Overlapping Region Decomposition Problem (-MOVRDP) for a world instance <G=(V,E), F, {v} vV> consists of finding a minimum size -overlapping decomposition D = {R1, …, Rd} of V into regions such that:   Theorem 1: A -MOVRDP can be reduced to an equivalent 0-MOVRDP, and the solution to this latter problem can be extended to a solution for the original problem. Theorem 2: The decision problem <0-MOVRDP, d> is NP-complete. (Proof by reduction from the Minimum Set Cover Problem.)

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms.

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region:

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 25

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 25

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 19

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 19

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 19

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 19

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 17

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 17

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 14

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 14

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 11

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 11

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 9

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 8

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 8

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 6

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 4

Heuristic Methods for 0-MOVRDP 0-MOVRDP is intractable. Optimal decomposition not needed in practice. We developed and tested six greedy approximation algorithms. Algorithm A.x: O(|V|2|F|) k = 4 Features commonly visible in region: 4

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region:

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 1

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 1

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 1

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 1

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 1

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 1

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 2

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 2

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 2

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 2

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 2

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 3

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 4

Heuristic Methods for 0-MOVRDP Algorithms B.x and C: O(k|V|2|F|) k = 5 Features commonly visible in region: 5

Results Simulated Data

Results Simulated Data (cont.)

Results Simulated Data (cont.) World Settings Two types of Worlds: Irregular (Irreg) and Rectangular (Rect). average diameter: 40m. pose space sampled at 50 cm intervals. average number of sides: 6. average number of obstacles: 7.

Results Simulated Data (cont.) World Settings Two types of Worlds: Irregular (Irreg) and Rectangular (Rect). average diameter: 40m. pose space sampled at 50 cm intervals. average number of sides: 6. average number of obstacles: 7. Two types of Features: Short-Range and Long-Range. visibility range N(0.65, 0.2) to N(12.5, 1) m, and angular range N(25, 3) degrees. Visibility range N(0.65, 0.2) to N(17.5, 2) m, and angular range N(45, 4) degrees.

Results (cont.) Simulated Data (cont.) 300 randomly generated worlds Runtime of few seconds Avg. size of regions depends on stability of features in pose space. Number of regions increases as avg. size of regions decreases. Alg. B.2 achieved the best results. Algorithms  A.1 A.2 A.3 B.1 B.2 C Setting World Feature 1 Rect Short-Range 2 Long-Range 3 Irreg 4

Results (cont.) Real Data We applied the best-performing algorithm (B.2) to real feature visibility data. 0 90 180 270

Results (cont.) Real Data Experiment 1 Data collected in 2m  2m area. Sampled at 20 cm intervals. Total of 46 visible features. Camera at a fixed orientation (looking forward). Features were extracted using the Kanade-Lucas-Tomasi operator. Parameters used:  = 0, k = 4.

Results (cont.) Real Data Experiment 1 Data collected in 2m  2m area. Sampled at 20 cm intervals. Total of 46 visible features. Camera at a fixed orientation (looking forward). Features were extracted using the Kanade-Lucas-Tomasi operator. Parameters used:  = 0, k = 4.

Results (cont.) Experiment 2 Data collected in 6m  3m area. Sampled at 25 cm intervals. Total of 897 visible features. Camera at 0, 90, 180, and 270 degree orientations. Lowe’s SIFT features.

Typical Feature Visibility Regions Results (cont.) Experiment 2 Data collected in 6m  3m area. Sampled at 25 cm intervals. Total of 897 visible features. Camera at 0, 90, 180, and 270 degree orientations. Lowe’s SIFT features. Typical Feature Visibility Regions

Results (cont.) Experiment 2 (cont.) k=4, =0

Results (cont.) Experiment 2 (cont.) k=4, =1

Results (cont.) Experiment 2 (cont.) k=10, =0

Results (cont.) Experiment 2 (cont.) k=10, =1

Conclusions We have introduced a novel graph theoretic formulation of the landmark acquisition problem, and have established its intractability. We have explored a number of greedy approximation algorithms, systematically testing them on synthetic worlds and demonstrating them on two real worlds. The resulting decompositions find large regions in the world in which a small number of features can be tracked to support efficient on-line localization. The formulation and solution are general, and can accommodate other classes of image features.

Future Work Integrate the image collection phase with the region decomposition stage to yield an on-line process for simultaneous exploration and localization (SLAMB). Path planning through decomposition space, minimizing the number of region transitions in a path. Detect and cope with environmental change. Compute the performance guarantee of our heuristic methods and provide tight upper bounds on the quality of our solution compared to the optimal. Use feature tracking during the image collection stage to achieve larger areas of visibility for each feature. (Maintain equivalence classes of features in the DB.)