1. Prospect Eleven: Princeton Universitys Autonomous Vehicle Entry 2005 DARPA Grand Challenge Thursday, February 6, 2006 National Physical Laboratory Teddignton, Middlesex, UK Alain L. Kornhauser Team Leader, Prospect Eleven Professor, Operations Research & Financial Engineering Princeton University Co-founder & Board Chair, ALK Technologies, Inc. Applications of Knowing Where am I Two Examples: CoPilot|MinETA Real-Time Dynamic Minimum ETA Sat/Nav

2. Prospect Eleven: The Making, Testing and Running of Princetons Entry in the 2005 DARPA Grand Challenge Alain L. Kornhauser Team Leader, Prospect Eleven Professor, Operations Research & Financial Engineering Andrew Saxe08Gordon Franken08Scott Schiffres06 Brendan Collins08Bryan Cattle07Rachel Blair06 Josh Herbach08Anand Atreya07Kamil Choudhury06

3. Jan 26, 2006 The DARPA Grand Challenge Defense Advanced Research Projects Administration DARPA Grand Challenge Created in response to a US Congressional and DoD mandate, it was a field test intended to accelerate research and development in autonomous ground vehicles that will help save lives on the battlefield. The Grand Challenge brought together individuals and organizations from industry, the R&D community, government, the armed services, academia, students, backyard inventors, and automotive enthusiasts in the pursuit of a technological challenge. The First Grand Challenge: Across the Mojave, March 2004 From Barstow, California to Primm, Nevada offered a $1 million prize. From the qualifying round at the California Speedway, 15 finalists emerged to attempt the Grand Challenge. However, the prize went unclaimed as no vehicles were able to complete the difficult desert route. The 2005 Grand Challenge October 8, 2005 in the desert near Primm. Prize increased to $2 million.

4. Jan 26, 2006 18 -month Life Cycle of Prospect Eleven 20042005 Jan Automate Brakes Feb Install Computers Mar2004 Grand ChallengeMar Mount GPS AprAnnouncement of 2005 GCApr Work on Machine Vision MayDecide to get involvedMay1 st Site Visit JunHire 5 summer internsJunNot top 40, Alternate Status Jul Literature surveyJul Re-Do Stereo Vision Aug Literature surveyAug 152 nd Site Visit SepSubmit ApplicationSep 27National Qualifying Event OctOct 82005 Grand Challenge NovReceive salvaged 05 Canyon Fall Break Take care of unfinished business Dec Automate SteeringDec

5. Jan 26, 2006 Prospect Eleven & Competition

6. Jan 26, 2006 Constraints Very little budget Simplicity Guiding Principles Objective Enrich the academic experience of the students

7. Jan 26, 2006 http://www.pcmag.com/slideshow_viewer/0,1205,l=&s=1489&a=161569&po=2,00.asp Homemade Unlike the fancy drive by wire system employed by Stanford and VW, Princetons students built a homemade set of gears to drive their pickup. I could see from the electronics textbook they were using that they were learning as they went.

8. Jan 26, 2006 Teamwork Real-time Decision System Andrew Saxe08 Object Detection System Brendan Collins08 Mechanical Systems Gordon Franken08 Planning Systems Josh Herbach08 Electronic Systems Bryan Cattle07 Computing Systems Anand Atreya07 Control Systems Scott Schiffres06 Organizational Systems Rachel Blair06 Team Leader Alain Kornhauser*71P03

9. Jan 26, 2006 Mechanics Design and Fabrication of: Vehicle Actuators –Steering –Brakes –Transmission Sensor Housings –Stereo Camera –GPS Antennae System Protection –Shock-absorbing computers –Secure installation of components

10. Jan 26, 2006 Vehicle Actuators Steering –Motor from Bosch cordless drill. –Modified gears mounted directly to steering wheel. –Rotary encoder used for position feedback. Brakes –Bicycle brake cable used to depress pedal. –Motor-driven linear actuator. –Load-cell used for tension feedback. –Also: pneumatic e-brake, for fail-safe operation. Transmission –Ability to shift from forward to reverse. –Not used due to several issues: Slow actuation Difficulty interfacing with vehicle transmission Not open up an Achilles heel. Drill motor Steel Gears Rotary Encoder bike cable brake pedal motor Load cell

11. Jan 26, 2006 Sensor Housings Stereo Camera –Camera rated for indoor office use only !!! –Water & air tight enclosure –Mounts for photographic filters GPS Antennae –4 GPS receivers –Need good location on top of vehicle for best data. –Wooden boom also supported rotating beacon

12. Jan 26, 2006 System Protection Shock Absorption for computers. –Old way: Computers were wrapped in foam. –New way: Computers are rack-mounted on shock- isolating feet. Secure installation of other components. –Batteries, wires, power supplies, air compressor… –The shake test Shock-absorbing feet

13. Jan 26, 2006 Stereo Vision System Principle of Operation: Difference between two cameras gives depth information Steps: –Compute disparity image –Find obstacles in each column –Approximate with rectangles –Filter in time domain

14. Jan 26, 2006 Challenges –Quantization –Noise –Lighting condition –Field of view –Occlusion –Range

15. Jan 26, 2006 The left camera sees: Point Grey Bumblebee Focal Length: 4mm Resolution: 640x480 Black & White

16. Jan 26, 2006 The right camera sees: Point Grey Bumblebee Focal Length: 4mm Resolution: 640x480 Black & White

17. Jan 26, 2006 Yielding a disparity map: Intensity indicates distance: the lighter, the closer White indicates an invalidated location.

18. Jan 26, 2006 Processing in each column: Intensity indicates confidence that an obstacle exists at that location. A darker line indicates a higher confidence.

19. Jan 26, 2006 Bounding with rectangles: Darker rectangles indicate higher confidence.

20. Jan 26, 2006 Data Representation Reference: http://www.darpa.mil/grandchallen ge/TechPapers/Stanford.pdf vs.

21. Jan 26, 2006 Tracking

22. Jan 26, 2006 Navigation Blind-Mans Cane Approach: –Find best instantaneous velocity vector (heading and speed) within visible cone ahead. ? Desired V

23. Jan 26, 2006 Step 1: Tube Projection

24. Jan 26, 2006 Angle Distance Gap Identification Shoot the Gap V

25. Jan 26, 2006 Examples of Shoot the Gap Gotta Reverse Not without Problems May Get Squeezed Yippes!

26. Jan 26, 2006 Interfacing to Existing Vehicle Data Existing electronic information available on 05 GMC Canyon: Engine status –temp, RPM, diagnostic codes,... Transmission – what gear are we in? 4 WD on/off Car traction control –are wheels slipping, are we really moving? Wheel speed/odometry – how far have we moved? how fast? Throttle is electronic All monitored & used by Prospect Eleven

27. Jan 26, 2006

28. Other thoughts

29. Jan 26, 2006 Feedback Control Systems Speed Control Module: –Receives desired speed from Navigation –Decides how to modulate throttle and brake –Requirements Gracefully reach and maintain desired speed Smooth acceleration and braking –To minimize skidding on loose sand wet grass Strict adherence to speed limit –Minimize overshoot Steering Control Module –Receives desired heading angle and speed from Navigation –Converts to desired steering wheel angle –Constrains wheel angle as function of speed to avoid roll-over. –Control Coefficients tuned to trade off response and overshoot

30. Jan 26, 2006 Speed Control -- Implementation Proportional Integral Control –Output = K P e v + K I e v dt + K D (de v dt) –Where e velocity = V desired – V current K P term deals with the bulk of error K I integrates up error to eliminate steady state error K D reduces overshoot and ringing by slowing response Speed Control Desired Speed Current Speed Latest Speeds Percent Throttle Percent Brake Tension

31. Jan 26, 2006 Why P11 stopped after 9.4 miles We store a list all obstacles we are currently concerned about –Each time a new obstacle is received, we add it to this list –Each obstacle needs to have its relative position updated each time we get new vehicle position information It gets these updates by subscribing to the RelativeFrameUpdated event When we have passed an obstacle by more than twenty feet, we remove it from the list –However, we were not unsubscribing the obstacle from the RelativeFrameUpdated event Thus, nine miles down the road, our computer was still processing a bush it saw near the beginning of the course! C# is a powerful language, but it has to be used carefully

32. Timeline of Accomplishments of Princetons Prospect Eleven DARPA Grand Challenge Team May 2004 - Nov 2005 SegmentDescriptionImagesVideos Going Back Oct 30- Nov 2, 05 After completing 10 miles in GCE, Prospect Eleven returns to the desert to complete the 2005 and the 2004 Grand Challenge courses GPS tracks for 3 days 2004 DGC Nov 2, 05 GPS Tracks 0.3mb Run Images 3mb Cruisin04 2.5mb Return to Beer Bottle Pass, Nov 1, 05 GPS Tracks 0.3mb 2005 DGC, Oct 31, 05 GPS Tracks 0.1mb Run Images 3mb Cruisin05 16mb Gate05 3mb, Gate2 05 4mb Changing one line, Oct 30, 05 Images 0.2mbFixing Calibrating 2005 GCE Oct. 8, 05 DARPA Grand Challenge Event, 132 mile course in desert around Primm, NV; 23 entries; Prospect Eleven is #10 seed Run SummaryStart_11mb PassBy_29mb Summary 17mb NQE Sep 27-Oct 5, 05 National Qualifying Event held at California Speedway. 43 bots competing for 23 positions on 2.2 mile course NQE CourseCrashOutside_7mb CrashInside_4mb PrefectRun5_9mb Run-up to NQE Aug 17-Sep 26 Modifications and testing after invitation as one of three Alternates to the NQE 2 nd Site Visit Aug 16, 05 2 nd chance at demonstrating capabilities of P11 to DARPA officials at West Windsor athletic fields after earning Alternate status Pre-2 nd _SiteVisit_6mb 1 st Site Visit May 3, 05 Demonstration of Prospect Elevens capabilities and potential to DARPA officials, mixed performance Automation of P11 Nov 04-May 05 Conversion of a GMC 05 Canyon to Prospect Eleven: Automation of brake, throttle, steering; GPS; vision P11 Tech Paper Application & Prep May 04-Nov 05 Putting the team together, planning, organization, & literature search.

33. DAPRPA Grand Challenge Event, October 07, 2005 The Day Before 2005 Grand Challenge Event

34. L2R: Rachel Blair06, Dan Chiou05, Prof. Alain Kornhauser*71 P03, Ben Essenburg05, Bryan Cattle07, Josh Herbach08. Jeff Jones*05, Kamil Chihoudy06, Scott Schiffres06, Anand Atreya07; Launch Team: Andrew Saxe08, Brendan Collins08, Gordon Franken08 DAPRPA Grand Challenge Event, October 08, 2005 The DARPA Grand Challenge Event Team 2005 Grand Challenge Event

35. DAPRPA Grand Challenge Event, October 08, 2005 The Launch of Prospect Eleven 2005 Grand Challenge Event

36. DAPRPA Grand Challenge Event, October 08, 2005 Return of Prospect Eleven @ 8 Mile Mark 2005 Grand Challenge Event

37. DAPRPA Grand Challenge, Unfinished Business, October 31, 2005 Approaching Beer Bottle Pass 19:02 2005 Grand Challenge Course

38. DAPRPA Grand Challenge, Unfinished Business, October 31, 2005 Beer Bottle Pass 19:10 PST 2005 Grand Challenge Course

39. DAPRPA Grand Challenge, Unfinished Business, November 1, 2005 The Assent: 17:00 Rerun to Beer Bottle Pass

40. DAPRPA Grand Challenge, Unfinished Business, November 1, 2005 The Assent: 17:01 Rerun to Beer Bottle Pass

41. DAPRPA Grand Challenge, Unfinished Business, November 2, 2005 12:25 2004 Grand Challenge Course

42. DAPRPA Grand Challenge, Unfinished Business, November 2, 2005 14:26 2004 Grand Challenge Course

43. DAPRPA Grand Challenge, Unfinished Business, November 2, 2005 The Finish 18:44 2004 Grand Challenge Course L2R: Prof. Alain Kornhauser*71 P03, Andrew Saxe08, Bryan Cattle07, Scott Schiffres06

44. Prospect Elevens Unfinished Business October 31-November 2, 2005 DARPA Grand Challenge 04 & 05 Courses GPS Tracks and Timing Maps DAPRPA Grand Challenge, Unfinished Business, October 31, 2005 Finish, 05 Course

45. Jan 26, 2006 Start 7:53:30 PST Finish 19:35:49 Elapsed Time 11:43:49 Pause Time ~2:55:49 Autonomous Time 8:48:59 8:44:48 1:10 Pause 10:48:04 3:45 Open & pass thru gate 9:52:53 8:14 Open gate & pass through 13:18:53 1:10 Pause 18:20:53 38:50 prepare inside video 19:00:21 – 0:56 Adjust Camera 19:02:53 – 1:52 Wait for chase car 10:37:37 5:54 Open & pass thru gate 18:02:50 2:08 Pause 11:15:15 5:14 Pause at ranch, go thru gate 11:29:52 42:00 Pull chase car out of mud 13:16:05 1:32 Rejoin course 10:53:33 2:15 Pause 14:16:03 2:23 remove rubble blocking I-15 underpass 12:42:35 4:40 Pause 12:33:51 1:19 Pause 15:13:31 1:30 Wait for traffic to clear on NV 161 16:03:07 1:53 Pause 16:07:48 4:32 to take pictures 16:12:50 11:44 negotiate bulldozed area 16:31:17 2:12 Survey another bulldozed area 8:20:55 5:20 Pictures at Primm return 8:31:42 8:53 Take down fence & pass through GPS Tracks & Timing of Prospect Elevens Autonomous run of the 05 GC Course 10/31/05

46. Jan 26, 2006 Red: 8:00 Outbound 10/31 Deviation to avoid lake Yellow: 16:30 Return 10/31 Deviation to avoid lake Blue: 8:00 Outbound 11/2 Lake dried sufficiently, no deviation required. GPS Tracks & Timing of Prospect Eleven Diverting Around Not-so-dry Lake 05 GC Course 10/31/05

47. DAPRPA Grand Challenge, Unfinished Business, November 1, 2005 Return to Beer Bottle Pass GPS Tracks of return to Beer Bottle Pass 11/01/05

48. DAPRPA Grand Challenge, Unfinished Business, November 1, 2005 Return to Beer Bottle Pass 2005 Grand Challenge Course, 1.9 miles Start Down: 17:10:17 End Up: 17:04:48 Duration: 0:10:21 Start Up: 16:54:27 End Down: 17:18:04 Duration: 0:7:47 GPS Tracks & Timing of Prospect Elevens Autonomous run up then back down Beer Bottle Pass 11/02/05

49. Jan 26, 2006 7:28:33 4:35 negotiate bulldozed area 7:55:26 33:08 Divert to get gas for P11 8:39:13 11:50 take pictures 8:34:01 4:30 take pictures 10:13:43 2:10 cross road 9:42:26 1:50 take pictures 10:04:59 2:07 take pictures 10:28:11 3:20 take pictures 7:08:36 Start 04 GC Course GPS Tracks & Timing of Prospect Elevens Autonomous run of the 04 GC Course 11/02/05 DAPRPA Grand Challenge, Unfinished Business, November 2, 2005

50. Jan 26, 2006 14:50:xx 4:00 Avoid Washout in 4 spots 15:10:10 2:04 Inspect burial monument 12:05:25 0:30 Xing CA 127 12:08:00 2:00 take pictures 13:08:54 1:05:56 Fix steering encoder 12:38:03 2:17 take pictures 12:19:37 1:50 take pictures 10:57:13 6:30 take pictures 12:53:36 0:54 Pause 14:33:15 2:45 take pictures 14:44:13 2:47 take pictures GPS Tracks & Timing of Prospect Elevens Autonomous run of the 04 GC Course 11/02/05 DAPRPA Grand Challenge, Unfinished Business, November 2, 2005

51. Jan 26, 2006 17:16:33 15:28 Replace Front Left Blowout 16:12:07 0:50 Xing road 15:19:00 4:32 take pictures 15:46:03 2:07 take pictures 15:38:01 1:37 take pictures 16:33:15 2:45 Letting traffic pass on Randall Drive 16:35:14 1:00 Xing Yermo Road 17:32:27 4:00 Inspecting Silt Filled underpasses < 4 ft. clearance 18:41:26 1:10 Pause let cars pass on Stoddar Valley Rd GPS Tracks & Timing of Prospect Elevens Autonomous run of the 04 GC Course 11/02/05 18:18 Top 18:35 Bottom Daggett Pass Start 7:08:36 Finish 18:44:19 – Arrival at gate of Slash X Cafe Elapsed Time 11:37:43 Pause Time ~03:28:00 Autonomous Time 08:09:00 File photo Finish, 04 Course

52. Jan 26, 2006

53. Accomplishments: Invited to National Qualifying Event Seeded 10 th for Grand Challenge Accomplished 10 miles of Autonomous Driving in GC Completed the 2005 & 2004 Courses during Fall Break Overview: http://orfe.princeton.edu/~alaink/Prospect11/Movies/P11SummaryMovie.wmv Wasnt so easy, Outside: http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE2CrashOutsideView.mov Wasnt so easy, Inside: http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE2CrashInside.mov Essentially Perfect: http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE5PerfectRun.mov Launch at Grand Challenge: http://orfe.princeton.edu/~alaink/Prospect11/Movies/GCE-Start.AVI Doing Better than 2004s Best: http://orfe.princeton.edu/~alaink/Prospect11/Movies/GCE_PassBy8Miles.avi Return to the Desert; 2005 Course: http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC05_10h29mFastDryLake.AVI http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC05_10h34mFastDryLake.AVI Return to the Desert; Return to Beer Bottle Pass: http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGCBB_Small.mov Return to the Desert; 2004 Course: http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC04_11h59mCruisin.AVI Some Videos:

54. Jan 26, 2006 Lessons Learned It is non-trivial to Just Do It –You must respect Uncertainty (and plan for it) –Harmonize Accuracy –Time is your Friend (only know what you need to know when you need to know it) –More is not necessarily Better –Always assume your code has bugs Stereo Vision Does Work! Three (3) Regimes of Autonomous Control –Under 7 mph –Between 7- 25 mph –Above 25 mph

55. Jan 26, 2006 Operational Processes used by Capital District Advanced Traveler Information System (CD-ATIS) CoPilot|MinETA: Real-Time Dynamic Minimum ETA Sat/Nav by Alain L. Kornhauser Co-founder & Board Chair, ALK technologies, Inc.

56. Jan 26, 2006 16 1 1 1 2 2 2 3 3 3 4 3 55 6 6 8 8 9 6 PROBLEM: How to get from A to B Many Paths Each with a Different Value to the Decision Maker Each Segment Changing with Uncertainty over Time Things Change!! 4

57. Jan 26, 2006 Link Travel Times Historic, Actual & Forecast During Day One week-day on one link Things change!

58. Jan 26, 2006 Real-Time Dynamic Minimum ETA Sat/Nav 250 Volunteers using CoPilot|Live commuting to/from RPI CoPilot continuously shares real-time probe-based traffic data CoPilot continuously seeks a minimum ETA route Advance project Illinois Universities Transportation Research Consortium The late 90s Conducted its version of the abandoned ADVANCE ( Advanced Driver and Vehicle Advisory Navigation ConcEpt ) project Link &

59. Jan 26, 2006 Project Objectives Create: real-time data collection from vehicles and dissemination to vehicles of congestion avoidance information which is used to automatically reroute drivers onto the fastest paths to their destinations Target locations: small to medium-sized urban areas Aspects: operations, observability, controllability, users, information transfer to travelers

60. Jan 26, 2006 3-month field test Capital District (Albany), NY, USA Journey-to-work 200 participants 80 Tech Park employees 120 HVCC staff & students Techy travelers Network: Freeways & signalized arterials Congested links Path choices exist Experiment Details

61. Jan 26, 2006 Basic Operational Architecture Two-way cellular data communications between Customized Live|Server at ALK Customized CoPilot|Live In vehicles 6 Destinati on 1 2, 4 3 5 7 8

62. Jan 26, 2006 The In-Vehicle Device Sprint PCS Vision card and battery pack: Communication with the server CoPilot GPS unit: Determines location Pocket PC with 256 MB SD card: Software platform and audio communication with the driver

63. Other Sat/Nav Platforms

64. Jan 26, 2006 Every Second CoPilot|Live Determines Where am I, Then… CoPilot|Live Where Am I, Then… ALK Server Updates: TT(m i, m j ) If Momument, m j, is passed Send m i, m j, tt k (m i, m j )= t(m i ) - t(m i ) (52 bytes) Set i=j

65. Jan 26, 2006 Every n Minutes ALK Server Builds: set U k Sends: TT(m i, m j ) for every (i,j) in U k CoPilot|Live … Send… Current Location & Destination, Last update time (42 bytes) ALK Server … Send… New TT(m i, m j ) for every (i,j) in U k (280 bytes/100arcs) CoPilot|Live … Updates TT(m i, m j ) in U k, ETA on current route, Finds new MinETA route, if MinETA substantially better then… Adopt new route ALK Server … Determines U k : set of TT(m i, m j ) within bounding polygon of (Location;Destination) k that have changed more than y% since last update. CoPilot|Live Sends: Where am I, Dest., Last update Receives/Posts: updates Computes: MinETA Updates route, if better

66. Jan 26, 2006 When Available ALK Server … Receives: Other congestion information from various source, blends them in TT(m i, m j ) ALK Server Updates: TT(m i, m j )

67. Jan 26, 2006 What We Heard I find it interesting how willing I am to listen to a machine tell me which route to take I like using it for when I have no idea on how to get somewhere, and it is good for my normal route because it keeps me out of traffic on route 4. It is great, it took a while to trust it telling me where to go, but i like it because i cant get lost! Thanks. This thing is awesome. I was a little skeptical at first but once i got the hang of it I dont know how I went along without it. I think any student commuting to school will benefit from this. I'm very impressed with the CoPilot program thus far. The directions are accurate and it adapts quickly to route changes.

68. Jan 26, 2006 1 2 3 also Can Watch Vehicles

69. Jan 26, 2006 North American Monument Network ~ 125,000 North American Monuments ~10 6 (m i, m j ) 1 st Commercial application in PC*Miler v19 (contains Median Travel Time by Time-of-Day for all NA) (mi, mj) near Troy (mi, mj) larger area

70. Jan 26, 2006 Thank you Alain L. Kornhauser alaink@alk.com www.alk.eu.com www.princeton.edu/~alaink/