J. Garbarino, C. Roussi, B. White ALGORITHM/SYSTEM OVERVIEW RITARS-11-H-MTU1 Technical Advisory Committee Project Update – March 24, 2015

Overview Review –Data Collection System Airborne platform(s) Camera, lens, GPS Intervalometer –Data Processing System Algorithm Suite Results Future steps 2

Unmanned Platform Bergen Folding Hexacopter –7kg flight-ready –Gyro-stabilized platform Nikon D800 w/GPS Custom Intervalometer 3

Manned Platform Cesna 152 or equivalent ($170/hr) Nikon D800 w/ mm f/2.8 lens ($5100) Intervalometer ($200) Tyler mini-gyro stabilized mount ($500) 4

System Specifications 5 Hexacopter Batteries26Ah, 14.8V Weight4kg empty, 5.5kg w/ batteries Flight Time20min hover, 10min full power Range5km Ceiling~3000m Misc accessories Charger, flight controller, tools Cost$5200 D800 Resolutio n 36Mp (7360x4912) Weight1kg (1.3kg w/ lens) Frame advance 5 fps max Speed1/8000s – 30s ISO Cost$2800 Nikkor 50mmLens F-stopf/1.4 – f/16 Weight0.3kg Field of view31 deg Cost$480 Intervalometer(custom) Interval Range5s – 1/4s Weight200g Battery9V alkaline Cost$200 CostsUnmannedManned Equipment$8700$5800 Operating1 FTE1 FTE + $170/hr Nikkor 70mm- 200mm Lens F-stopf/2.8 – f/22 Weight1.5kg Field of view12-34 deg Cost$2400 Tyler Mountgyro Size25” x 20” x 13” Weight27kg w/ batteries Cost$500

Unmanned Concept of Operations Typical Collection involves: –Assemble hexacopter and pre- flight check – 7min –Determine camera settings and controller setup – 2min –Flight collection – 2min for 100m –Stow equipment – 5min –Charge batteries – 20min Typical selection and processing – 4 hrs 6 Collect Data Select Site Select Data Process Data Evaluate in Roadsoft

Manned Concept of Operations Typical Collection involves: –Emplacing tyler mount (may involve a modified aircraft w/ port in hull) – 10min –Determine camera settings and controller setup – 2min –Flight collection 4s for 100m 2-passes needed for coverage –Stow equipment – 10min –Charge batteries – 20min Typical selection and processing – 4 hrs 7 Collect Data Select Site Select Data Process Data Evaluate in Roadsoft

Platform Comparison 8 Hexacopter ProsCons Easy to transport and deployCan only collect 1km of road on a set of batteries Data can be collected by a single personMay require closing road for safety Operating costs only involve 1 person’s hourly rate Some ground-truth can be collected manually, if needed No recurring costs Manned Aircraft ProsCons Can collect more road segments during a flight Requires a modified aircraft for gyro-stabilized mount Can operate over a wider range of weather conditions Involves a pilot and an operator Requires multiple passes over the same road to get sufficient coverage for 3D reconstruction Requires more careful selection of images (manually) for processing

Software Architecture Because we are incorporating legacy code, third- party tools, and custom code, we need a flexible architecture –Developed in C, C++, Python, bash –Flexible control, with tools calling each other as needed 9

Algorithm Use Structure from Motion (SIFT+ Bundler + PVMS) to turn 2D images into 3D point-cloud reconstruction –SIFT = scale-invariant feature transform –PVMS = patch-based multi-view stereo Form a surface from the 3D point-cloud –Form grid, Fourier Filter, Marching Cubes to triangulate Find the depth/height map of the surface –Singular Value Decomposition (SVD) –Rotate so z-axis is “up” (depth) 10

Algorithm Find and select the road in the scene –Image entropy measure (road is “smoother”) Rotate extracted road into new coordinate system –Makes it easier to take cuts along and across road Analyze for features of interest –Gabor Filtering, Circular Hough Transform, Cuts for profiles of road and drainage Convert to PASER-like metrics Generate XML output suitable for RoadSoft processing 11

Algorithm 12

Algorithm Details 13

Example Image Taken from 25m, 2m/s 14

Example Reconstruction 15 images used for reconstruction 15 Bundler outputDensified point cloud 3D surface from point cloud Height-field from surface

Road Segmentation from Depth Map 16 Rotated Depth Map Mask of Road Surface from image Entropy Extracted Road

Depth Map Detail 17

Input to Crown Measurement 18 Across Road Along Road Example crossection plot meters

Pothole Detection 19 XML Report

Ruts and Corrugations 20

Performnace Summary Crown estimates vary from manual ratings slightly –We measure the crown everywhere; manual inspections sample the surface poorly Ruts: Pd = 67%, Pfa = 19% Corrugations: Pd = 100%, Pfa = 38% Potholes: Pd = 95%, Pfa = 4% 21

Observations Deep ruts are sometimes labeled as potholes Strings of potholes along the driving direction are sometimes labeled as corrugations Strips of grass on the road surface cause false alarms Manual scoring is a trade-off between accuracy and time –Spot checks –Spend long enough to get a “good enough” estimate Automated scoring finds everything –Can be both good and bad 22

Room for Improvement - General Each step in the process can be addressed –Collection parameters –Image quality –Image pre-processing to enhance –Processor operating points –Algorithm choices for current distresses Old can be refined New can be tried Expanded Applications 23

Room For Improvement - Specific Automatic rating/rejection of unsuitable images –Blurred images limit reconstruction accuracy Tuning of algorithms –Each process has “knobs” to adjust performance –Internal operations can be refined E.g. changing entropy estimation routine –Throughput optimization Script additions to expose more controls –Adding switches makes it more flexible –Set/reset detection points Adding data exploitation routines (not all information that can be gleaned from the data has been) –Intersection geometry –Texture changes 24

Example One way of characterizing an intersection is by abstracting its geometry –Leveraging computer vision morphological tools Sample intersection w/ non-perfect segmentation, followed by medial-axis transformation –Finding the “skeleton” of the intersection –Imaging a brush-fire starting at the boundary; the place where the fire meets is the media axis 25