1. Procedures for FOD Detection System Performance Assessments Radar-Based and Dual Sensor Systems Beth Woodworth Edwin Herricks University of Illinois

2. Introduction I will be talking today about two performance assessment efforts. The initial effort began in 2004 and was completed with assessment of an installed system on a runway at PVD in 2008. The second began in 2007 and was completed in 2009.

3. Sensors Evaluated Qinetiq Tarsier TM – Radar based system – Installed at PVD Xsight FODetect TM – Hybrid system – Installed at BOS

4. Testing of QinetiQ Tarsier TM at PVD 94Ghz radar Detects FOD at ranges up to 1 km 2 sensor units placed on towers scanned Runway 5/23 at PVD Primary performance criterion was a standard -20dBm 2 target at a range of 1km

5. Testing of FODetect TM at BOS 74GHz radar and video camera with image processing Sensors replaced edge lights Partial installation at BOS Sensors covered 550 ft on Runway 15R 10 surface detection units (SDUs) – 5 SDUs on each side of the runway Each sensor covered approximately 75ft x 200ft Primary performance criterion detection of 0.8 in (2 cm) object

6. General Assessment Protocol Testing scheduled over approximately 12-month period to capture varied weather conditions – PVD – June 2007- March 2008 – BOS – June 2008 - March 2009 Targets selected to challenge each of the systems and included Different sizes Different shapes Different materials Different hazard potentials

7. Detecting a Target Radar – Reflectivity – energy returned to sensor – “Complex Scatterers” – most FOD has a variety of edges which have different likelihoods of being detected. Electro-Optical – Background – Item surface conditions Color, Flat/Glossy – Illumination – Lighting and shadow

8. Overall Goals of Testing Calibration – Standard Target Testing – Uses items of known detection characteristics – Provides information on system reliability and robustness Performance – Group of FOD items with different hazard potentials – Repeatedly placed at known locations in defined positions Blind – Random placement and actual and simulated FOD items – This test most represents typical detection needs at an airport – Challenges system to detect a diverse and unpredictable set of items.

9. General Methods Safety – – Because assessments were conducted at operational airports, safety considerations were paramount. – Personnel were trained and briefed regularly by airport operations personnel. – Although runways were closed, safety areas were identified and drills held so that all assessment personnel were aware of their environment. – Rigorous accounting was made of all experimental FOD taken on the airport.

10. General Methods FOD log – Over 1000 items placed and retrieved! – all items were written in a log before proceeding to the runway – after return and often during the testing the log was checked again the items in the box FOD box – items put in boxes with dividers in order to be easily and quickly counted Marking – Used for calibration and performance testing – Small, nearly invisible UV marks were placed on the runway – UV lights used to find markings – Locations were surveyed using a differential GPS accurate to millimeters in the X/Y plane – Surveyed locations were compared with locations provided for each detection by each technology

11. General Methods Radar – Does NOT depend on illumination – All assessment campaigns at PVD were performed during normal 12 pm to 5 am runway closures Electro-Optical – Did depend on illumination. – Testing at BOS was on Runway 15R which saw limited use allowing testing at different times. – Tests were performed at BOS during the day, at night, and at dawn and dusk to capture a range of lighting conditions.

12. Calibration – Standard Target Testing GOAL = confirm detection performance using objects with defined detection characteristics Targets selected based on technology – Metal cylinders 3in X 4.5in 1.25in with a nominal reflectivity of 0dBm 2 (Large) – Metal cylinders 2in X 2.5in with a nominal reflectivity of - 10dBm 2 (Medium) – Metal cylinders 1.5in x 1.25in with a nominal reflectivity of -20dBm 2 (Small) – Metal 2in spheres – Colored PVC cylinders 1.5in X 1.25in colored grey, white and black

13. Metal cylinders 3in X 4.5in 1.25in 0dBm 2 (Large) Metal cylinders 2in X 2.5in -10dBm 2 (Medium) Metal cylinders 1.5in x 1.25in -20dBm 2 (Small) Metal spheres 2inColored PVC cylinders 1.5in X 1.25in Calibration Targets

14. Calibration- Standard Target Testing for Tarsier TM 4 types of metal targets 6 transects along full length of the runway Transects 2, 3, and 4 within 1km of both radars Transects 1, 5, and 6 used for system assessments Transec t Distance from (m) Distance from (ft) Distance from S Radar 2 (m) Distance from S Radar 2 (ft) 1167554945551821 212684160220723 386428343561168 450216477082322 520466811403738 6443145415475075

15. Calibration for FODetect TM Metal cylinders 1.5in x 1.25in tested the radar portion of the sensor 3 colors of PVC cylinders (white, grey, black) and small metal cylinder Targets placed at varying distances from sensor in a test rectangle (approx. 150ft x 70ft) Targets placed to test the system and a single sensor Standard target array included 1 metal, 1 white, 1 grey, and 1 black cylinder Groups of targets oriented differently

16. Performance Testing Items not selected based on sensor type as with calibration testing Used examples of common items found at airports Selection of items based on frequency of occurrence and hazard potential Calibration items also used during performance testing Items placed in know positions at specific angles

17. Table of Standard FOD Items Used in Performance Assessments FOD Item a Expected HazardFrequency of Occurrence 1. Small Piece of ConcreteHighCommon 2. Standard Lug Nut From Service Vehicle HighCommon 3. Roller BearingHighCommon 4. Chunk of RubberLowCommon 5. Mechanics WrenchHighCommon 6. Fuel CapHighCommon 7. Cotter KeyModerateCommon 8. Plastic Bottle/Bottle CapLowCommon 9. Strapping MaterialModerateCommon 10. Expansion Joint Material LowCommon 11. Construction Material– Galvanized Nails or Sheetrock Screws Moderate Based on Construction Activity 12. Runway Infrastructure Part–Piece of Runway Light or Signage HighUncommon 13. Small FastenersModerateCommon 14. Metal StripHighUncommon 15. Fiberglass DoorModerateCommon 16. Asphalt ChunkHighCommon This table contains the items selected by CEAT

18. Performance Testing Methods at PVD 6 arrays related to standard target transects In array 5x5 grid established, approximately 25ft between items 23 targets total per array (no items set on runway lights) Position of items varied between campaigns (random selection of location achieved in test planning); items in the same position in all arrays in any single campaign. All items rotated 45 degrees after each scan to cover 8 cardinal points of the compass

19. Line established midway between the centerline and the edge line 5 identical performance items placed at equal distances on this line across the length of test rectangle Items rotated so detections at 0, 45, and 90 degrees relative center line recorded Performance Testing Methods at BOS

20. Blind Testing Used actual FOD items collected from runways Range of sizes, materials, colors Over 100 items in collection 30 items randomly selected for each campaign Placement locations chosen at random from a grid Items dropped or tossed so the orientation was random Up to 10 items used at one time. Blind Testing Grid at PVD

21. Typical Grouping of FOD Blind Items Blind Testing Grid of Locations at BOS

22. Inclement Weather Testing Objective to evaluate sensor performance in variable weather conditions Opportunistic based on long term plan, but scheduled adjusted to coincide with storm events. Access to the runways in snow was impossible so assessments performed just after snow emergency conditions

23. Inclement Weather Assessment - Rain PVD - Tarsier TM radar system October 2007 Calibration targets were deployed as a rain squall moved across the runway BOS – FODetect TM hybrid system March 2008 One light rain event at dusk

24. Inclement Weather Assessment - Snow PVD – Tarsier TM radar system – Assessments after 2 snow events Jan. 24, 2008 – flurries but no accumulation Feb. 12, 2008 – snow changing to sleet – Time for placement and retrieval were limited due to needs to clear the runway

25. Inclement Weather Assessment - Snow BOS – FODetect TM hybrid system – Winter Storm Jan. 7-12 – Weather Conditions: – rain, freezing rain, freezing drizzle, mist, ice pellets, fog, snow – Totaling 2.1in of wet precipitation Testing took place on Jan 8, 9, and 13, 2009 – Testing on Jan. 29 and 30 after snow events as well

26. Summary 2 systems – radar and hybrid Assessments performed at PVD and BOS Assessments took place in variable weather conditions General protocol – calibration, performance, blind – consistent for all FOD detection system types Actual procedures were adapted to each senor and adjusted for runway availability Targets were selected based on – the performance claims of manufacturers – related to sensor characteristics and performance parameters