1. Astrobiology Science and Technology for Exploring Planets (ASTEP) Mid-Year Review August 4, 2004 Robust Autonomous Instrument Placement for Rovers (JPL: Visual Target Tracking) Issa A.D. Nesnas (PI) Max Bajracharya Richard Madison Section 348 Jet Propulsion Laboratory

2. FY04 Mid-Year Review – August 4, 2004 - 2 Robust Autonomous Instrument Placement for Rover Objectives: Demonstrate using a single command cycle autonomous placement of a science instrument on a target designated from ten rover lengths away JPL: Develop and demonstrate algorithms to visually track a target designated from ten rover lengths away in realistic, Mars-like, terrain conditions FY03-FY05 Milestones: FY03 S oftware prototype of 2D/3D target feature tracker FY04 In JPL Mars Yard demonstrate tracker on Rocky 8 rover using a fixed mast similar to K9 rover at ARC FY05 Tune tracker performance based on feedback for an independent validation task. Deliver to ARC Funding Profile ($K): Task Manager: Issa A.D. Nesnas(818) 354-9709 nesnas@jpl.nasa.gov Participating Organizations: Jet Propulsion Laboratory Ames Research Center (PI: Liam Pedersen) Facilities: Rocky 8 and CLARAty software JPL Mars Yard, CLARAty Test bed Motion Correlator Motion Correlator Stereo Correlator Stereo Correlator K9 Rocky 8

3. FY04 Mid-Year Review – August 4, 2004 - 3 Problem Statement Problem: –Track target designated from 10 m away (10 rover lengths) –Maintain lock on target while rover navigates across rough terrain –Keep target within one pixel accuracy (1 cm final placement) Key Challenges –Continuous, high-frame-rate visual tracking not possible due to slow acquisition and limited/shared computational resources –Both image and 3D reconstructed model change significantly Target grows as rover moves toward it Sudden changes in FOV due to tilt or drop off rocks –Must operate with obstacle avoidance and pose estimators

4. FY04 Mid-Year Review – August 4, 2004 - 4 Key Challenges (a) Target (b) Designated Target Target Tracking time = t2 (avoiding an obstacle) time = t1 1 st Frame 37 th Frame after 10 m

5. FY04 Mid-Year Review – August 4, 2004 - 5 Mission Relevance and State-of-the-Art Mission Relevance –Key component technology for single-sol instrument placement –Relevant to MSL and future rover missions –Enabling technology that will reduce sols (3 sols for MER to 1 sol for MSL) State-of-the-art –MER baseline – no designated target tracking or single-sol instrument placement –MER non-baseline – visual odometry for improved pose estimation –Previous and current related research efforts: ARC single-cycle instrument placement (ASTEP) (FY02-FY05) FIDO instrument placement using homography transforms (FY00) Planetary Dexterous Manipulation - sample acquisition & instrument placement (FY95-99) ARC visual servoing on Marsokhod (FY96-97) –Military tracking relevant but not directly applicable Can safely assume high-frame rates Assumes some knowledge of tracked target – human-made objects –Most trackers require small motion between frames or a well estimated motion –Visual odometry alone accumulates errors that exceed 1 cm tracking error budget for a 10 m traverse –Our task: Tracks natural terrain features not known a priori Uses visual odometry for pose estimation Combines 2D and 3D information to improve accuracy Gracefully degrades in the absence of good stereo

6. FY04 Mid-Year Review – August 4, 2004 - 6 Technical Approach Use visual information to maintain lock on target as rover moves Combine 2D imaging techniques with 3D stereovision information Bound error by affine matching to originally selected feature Increase accuracy by tracking from different FOV cameras Develop precise mast kinematics to control the gaze of the cameras Seed algorithm with good pose estimates (visual odometry) Integrate software into CLARAty infrastructure Adapt to Rocky 8 rover Test in JPL Mars Yard Deliver software (through CLARAty) to MTP validation task Tune algorithm based on feedback from MTP validation task

7. FY04 Mid-Year Review – August 4, 2004 - 7 Milestones and Deliverables FY03: –Software prototype of a combined target feature tracker that uses both 2D and 3D visual information to keep a user selected feature tracked FY04: –Adapted version of the combined feature tracker operating the Rocky 8 rover with a fixed mast, similar to K9 rover at ARC, and tested in the JPL Mars Yard –Deliverable: Progress report to program office/sponsor FY05: –Tuned tracker parameters and improved tracking performance based on feedback for an independent validation task. Clearances for all software for ARC access. Support ARC for integration on K9 rover –Deliverable: The “combined 2D/3D visual tracker” software integrated into the CLARAty environment. To be delivered to : ARC - PI; Liam Pedersen Necessary clearances for the PI to access the technology component listed in item 1 and all its dependent component technologies (contingency: all dependent components are also cleared by JPL and Caltech IP offices). To be delivered to : ARC - PI; Liam Pedersen

8. FY04 Mid-Year Review – August 4, 2004 - 8 FY03 - FY04 Accomplishments

9. FY04 Mid-Year Review – August 4, 2004 - 9 Rover Platforms Intel x86 Ames JPL Rocky 8 VxWorks K9 Intel x86 Linux

10. FY04 Mid-Year Review – August 4, 2004 - 10 FY03 Experimental Setup Rocky 8 mast head Rocky 8 Rover with mast head Vision System Used for Tracking Camera Name Placed On BaselineLensFOVCCD Resolution Pixel size (μm) NavigationMast19 cm4 mm 60  640x4809.9 PanoramicMast23 cm16 mm 17  1024x7684.65 HazardBody8.6 cm2.8 mm 90  640x4809.9

11. FY04 Mid-Year Review – August 4, 2004 - 11 FY04 Experimental Setup Camera Name Placed On BaselineLensFOVCCD Resolution Pixel size (μm) NavigationMast20 cm6 mm 45  1024x7684.65 PanoramicMast30 cm16 mm 17  1024x7684.65 HazardBody8.6 cm2.8 mm 90  640x4809.9 Vision System Used for Tracking Fixed Rocky 8 Mast

12. FY04 Mid-Year Review – August 4, 2004 - 12 Integrated 2D/3D Tracker Til t Pan Wheel Odometry Estimator Visual Odometry (Hazard Cameras) Normalized Cross-correlation (NCC) 2D Affine Tracking Multiple Pyramid Levels (start w/ smallest template) Stereovision Target Point 2D image coordinat e NCC Template Locomotor command from navigator Create Affine (KLT) Templates of various sizes (not updated) Verify 2D Location If fails use predicted Pan/Tilt angles One time operation Verify 2D Location If fails use NCC result Verify 3D Location If fails use predicted 3D If fails use wheel odometry Odometry Pose Estimate Δ Rover Pose Non-autonomous step Image / sub image input Coordinate/Transform input First Left Image 2D/3D Tracked Target Predict New Target Location Drive a Step Towards Target Acquire mast stereo Images Left Image Right Image Mast Pointing Kinematics (Pan/Tilt) Updated every step Operator Designates Target Target Tracker Pose Estimator

13. FY04 Mid-Year Review – August 4, 2004 - 13 Feature Tracker 1.Rover acquires stereo images from tracking cameras (e.g. navigation) 2.Rover sends one (left) image to the ground system 3.Operator selects a point in the image and sends point to the rover 4.Rover receives the target and computes the 3D location using stereovision 5.Tracker creates KLT template windows of various sizes around the target 6.For each drive command, the algorithm: 1.Creates normalized cross correlation (NCC) template of the target 2.Moves the rover one step 3.Estimates change in rover pose (using visual odometry from hazard cameras) 4.Points the tracking cameras 5.Acquires an image pair 6.Matches NCC template across a search window in the new image. 7.Verifies 3D location of target is within an error bound based on rover pose 8.If fails, use predicted location from rover pose alone 9.Matches three different size affine (KLT) templates across the search window. 1.Starts with largest size original template for a coarse match 2.Verifies the 2D location 3.[Verifies the 3D location] 4.If fails to match, uses previous results (NCC result if using largest template or previousresult if using smaller templates) 5.Repeats steps with smaller KLT template to refine target position 7.Scientist selects a single point to track in one panoramic camera (mast mounted 16mm camera) 8.Compute 3D point location using stereo 9.Grow point to template window that maximizes flat area 10.Command rover motion (approx. 25 cm, max. 10  heading change) 11.Estimate rover motion using visual odometry 12.If visual odometry fails, apply adaptive-view based matching 13.Point 4 mm mast cameras using rover motion estimate (4) and 3D target location (1 or 10) 14.Track target in 4 mm cameras Use normalized cross-correlation between consecutive frames (allows large motion of target between frames) Update affine parameters between single original template and current image (accurate localization with no drift) 15.Triangulate 3D target location using 4mm camera images 16.Repeat steps 6-8 with 16mm cameras 17.Repeat from step 3

14. FY04 Mid-Year Review – August 4, 2004 - 14 Tracking Results over Rough Terrain Tracking Video View from 4 mm camera View from 16 mm camera

15. FY04 Mid-Year Review – August 4, 2004 - 15 Same Traverse Different Target View from 4 mm camera View from 16 mm camera

16. FY04 Mid-Year Review – August 4, 2004 - 16 Publications White papers: –I. Nesnas, M. Bajracharya, E. Bandari, R. Madison, C. Kunz, M. Deans, M. Bualat, “Visual Target Tracking for Rover-based Planetary Exploration,” submitted to IEEE Aerospace Conference, Big Sky, Montana, March 2004 New Technology Report –Title: 2D/3D Visual Tracker for Rover Platforms - NTR Number: 40696

17. FY04 Mid-Year Review – August 4, 2004 - 17 Tracking Target Designated from 10 m Demonstrates tracking a target designated from 10 m away on Rocky 8 rover in rough terrain Enables single-cycle instrument placement for MSL – reduces 3 sols to 1 sol –Tracked targets in 4 mm camera –Tracked targets in 16 mm camera –While rover driving over rocks up to 1 wheel diameter –Final accuracy to within 2 cm with 4 mm cameras (one pixel error from designated target) –Shorter runs with 16 mm had < 1 cm error in accuracy –Tracks even when no stereo information is available –Integrated in CLARAty