1. ASPRS Positional Accuracy Standards for Digital Geospatial Data Drafting Committee: Chair: Douglas L. Smith, David C. Smith & Associates, Inc. Dr. Qassim A. Abdullah, Woolpert, Inc. Dr. David Maune, Dewberry Hans Karl Heidemann, USGS REVISION 7, VERSION 1 NOVEMBER 14, 2014 1

2. ASPRS Positional Accuracy Standards for Digital Geospatial Data  Replaces: ASPRS Accuracy Standards for Large-Scale Maps (1990) ASPRS Guidelines, Vertical Accuracy Reporting for Lidar Data (2004)  Developed by: ASPRS Map Accuracy Standards Working Group, PAD, PDAD and LIDAR joint committee for map accuracy standard update  In Final Approved Version REVISION 7, VERSION 1, Nov. 14, 2014 Approved and adopted by ASPRS during the board meeting on Monday Nov. 17, 2014 in Denver during ASPRS 2014 PECORA conference 2

3. New Standard for a New Era Motivation Behind the New Standard: Legacy map accuracy standards with accuracy thresholds, such as the ASPRS 1990 standard and the NMAS of 1947, are outdated (over 30 years since ASPRS 1990 was written). Many of the data acquisition and mapping technologies that these standards were based on are no longer used. More recent advances in mapping technologies can now produce better quality and higher accuracy geospatial products and maps. Legacy map accuracy standards were designed to deal with plotted or drawn maps as the only medium to represent geospatial data. The NSSDA has no accuracy thresholds as it is guidelines and not a standard; primarily provides a common accuracy testing and reporting methodology to facilitate sharing and interoperability of geospatial data. 3

4. New Standard for a New Era Within the past two decades (during the transition period between the hardcopy and softcopy mapping environments), most standard measures for relating GSD and map scale to the final mapping accuracy were inherited from photogrammetric practices using scanned film. New mapping processes and methodologies have become much more sophisticated with advances in technology and advances in our knowledge of mapping processes and mathematical modeling. Mapping accuracy can no longer be associated with the camera geometry and flying altitude alone (focal length, xp, yp, B/H ratio, etc.). 4

5. New Standard for a New Era New map accuracy is influenced by many factors such as: –the quality of camera calibration parameters; –quality and size of a Charged Coupled Device (CCD) used in the digital camera CCD array; –amount of imagery overlap; –quality of parallax determination or photo measurements; –quality of the GPS signal; –quality and density of ground controls; –quality of the aerial triangulation solution; –capability of the processing software to handle GPS drift and shift; –capability of the processing software to handle camera self- calibration, –the digital terrain model used for the production of orthoimagery.. 5

6. New Standard for a New Era These factors can vary widely from project to project, depending on the sensor used and specific methodology. For these reasons, existing accuracy measures based on map scale, film scale, GSD, c-factor and scanning resolution no longer apply to current geospatial mapping practices. Elevation products from the new technologies and active sensors such as lidar and IFSAR are not considered by the legacy mapping standards. New accuracy standards are needed to address elevation products derived from these technologies. 6

7. ASPRS Positional Accuracy Standards for Digital Geospatial Data –Applicability: Defines specific accuracy classes and associated RMSE thresholds for digital orthoimagery, digital planimetric data, and digital elevation data Intended to be technology independent Limited to positional accuracy thresholds and testing methodologies for any mapping applications, and to meet immediate shortcomings in the outdated 1990 and 2004 standards and the NSSDA Is not intended to cover classification accuracy of thematic maps Does not specify the best practices or methodologies needed to meet the accuracy thresholds, though some best practices are included in the Annexes. –Includes: Glossary, Symbols, examples, conversion to legacy standards 7

8. New Standards Highlights –Positional Accuracy Thresholds which are independent of published GSD, map scale or contour interval digital orthoimagery digital elevation data –Additional Accuracy Measures aerial triangulation accuracy, ground control accuracy, orthoimagery seam line accuracy, lidar relative swath-to-swath accuracy, recommended minimum Nominal Pulse Density (NPD) horizontal accuracy of elevation data, delineation of low confidence areas for vertical data required number and spatial distribution of QA/QC check points based on project area 8

9. New Standards Highlights –It is All Metric! –Unlimited Horizontal Accuracy Classes: 9 Horizontal Accuracy Class RMSE x and RMSE y (cm) RMSE r (cm) Horizontal Accuracy at 95% Confidence Level (cm) Orthoimagery Mosaic Seamline Mismatch (cm) X-cm≤X≤1.41*X≤2.45*X≤ 2*X Horizontal Accuracy Standards for Geospatial Data

10. Horizontal Accuracy Class RMSE x and RMSE y (cm) RMSE r (cm) Orthoimage Mosaic Seamline Maximum Mismatch (cm) Horizontal Accuracy at the 95% Confidence Level (cm) 0.630.91.31.5 1.251.82.53.1 2.503.55.06.1 5.007.110.012.2 7.5010.615.018.4 10.0014.120.024.5 12.5017.725.030.6 15.0021.230.036.7 17.5024.735.042.8 20.0028.340.049.0 22.5031.845.055.1 25.0035.450.061.2 27.5038.955.067.3 30.0042.460.073.4 45.0063.690.0110.1 60.0084.9120.0146.9 75.00106.1150.0183.6 100.00141.4200.0244.8 150.00212.1300.0367.2 200.00282.8400.0489.5 250.00353.6500.0611.9 300.00424.3600.0734.3 500.00707.11000.01223.9 1000.001414.22000.02447.7 10 Common Horizontal Accuracy Classes according to the new standard [1] [1]

11. Common Orthoimagery Pixel Sizes Associated Map Scale ASPRS 1990 Accuracy Class Associated Horizontal Accuracy According to Legacy ASPRS 1990 Standard RMSE x and RMSE y (cm) RMSE x and RMSE y in terms of pixels 0.625 cm1:50 11.32-pixels 22.54-pixels 33.86-pixels 1.25 cm1:100 12.52-pixels 25.04-pixels 37.56-pixels 2.5 cm1:200 15.02-pixels 210.04-pixels 315.06-pixels 5 cm1:400 110.02-pixels 220.04-pixels 330.06-pixels 7.5 cm1:600 115.02-pixels 230.04-pixels 345.06-pixels 15 cm1:1,200 130.02-pixels 260.04-pixels 390.06-pixels 11 Examples on Horizontal Accuracy for Digital Orthoimagery interpreted from ASPRS 1990 Legacy Standard.

12. Common Orthoimagery Pixel Sizes Recommended Horizontal Accuracy Class RMSE x and RMSE y (cm) Orthoimage RMSE x and RMSE y in terms of pixels Recommended use 1.25 cm ≤1.3≤1-pixelHighest accuracy work 2.52-pixelsStandard Mapping and GIS work ≥3.8≥3-pixelsVisualization and less accurate work 2.5 cm ≤2.5≤1-pixelHighest accuracy work 5.02-pixelsStandard Mapping and GIS work ≥7.5≥3-pixelsVisualization and less accurate work 5 cm ≤5.0≤1-pixelHighest accuracy work 10.02-pixelsStandard Mapping and GIS work ≥15.0≥3-pixelsVisualization and less accurate work 7.5 cm ≤7.5≤1-pixelHighest accuracy work 15.02-pixelsStandard Mapping and GIS work ≥22.5≥3-pixelsVisualization and less accurate work 15 cm ≤15.0≤1-pixelHighest accuracy work 30.02-pixelsStandard Mapping and GIS work ≥45.0≥3-pixelsVisualization and less accurate work 12 Digital Orthoimagery Accuracy Examples for Current Large and Medium Format Metric Cameras

13. Horizontal Accuracy/Quality Examples for High Accuracy Digital Planimetric Data ASPRS 2014 Equivalent to map scale in Equivalent to map scale in NMAS Horizontal Accuracy Class RMSE x and RMSE y (cm) RMSE r (cm) Horizontal Accuracy at the 95% Confidence Level (cm) Approximate GSD of Source Imagery (cm) ASPRS 1990 Class 1 ASPRS 1990 Class 2 0.630.91.50.31 to 0.631:251:12.51:16 1.251.83.10.63 to 1.251:501:251:32 2.53.56.11.25 to 2.51:1001:501:63 5.07.112.22.5 to 5.01:2001:1001:127 7.510.618.43.8 to 7.51:3001:1501:190 10.014.124.55.0 to 10.01:4001:2001:253 12.517.730.66.3 to12.51:5001:2501:317 15.021.236.77.5 to 15.01:6001:3001:380 17.524.742.88.8 to 17.51:7001:3501:444 20.028.349.010.0 to 20.01:8001:4001:507 22.531.855.111.3 to 22.51:9001:4501:570 25.035.461.212.5 to 25.01:10001:5001:634 27.538.967.313.8 to 27.51:11001:5501:697 30.042.473.415.0 to 30.01:12001:6001:760 13

14. New Standards Highlights –Unlimited Vertical Accuracy Classes: –NVA = Non-vegetated Vertical Accuracy based on RMSE –VVA = Vegetated Vertical Accuracy based on 95 th percentile because errors are not normally distributed as required for RMSE 14 Vertical Accuracy Class Absolute AccuracyRelative Accuracy (where applicable) RMSE z Non- Vegetated (cm) NVA at 95% Confidence Level (cm) VVA at 95 th Percentile (cm) Within- Swath Hard Surface Repeatability (Max Diff) (cm) Swath-to- Swath Non-Vegetated Terrain (RMSD z ) (cm) Swath-to- Swath Non-Vegetated Terrain (Max Diff) (cm) X-cm≤X≤1.96*X≤3.00*X≤0.60*X≤0.80*X≤1.60*X Vertical Accuracy Standards for Digital Elevation Data

15. Vertical Accuracy/Quality Examples for Digital Elevation Data Vertical Accuracy Class Absolute AccuracyRelative Accuracy (where applicable) RMSE z Non- Vegetated (cm) NVA at 95% Confidence Level (cm) VVA at 95 th Percentile (cm) Within-Swath Hard Surface Repeatability (Max Diff) (cm) Swath-to-Swath Non-Veg Terrain (RMSD z ) (cm) Swath-to-Swath Non-Veg Terrain (Max Diff) (cm) 1-cm1.02.030.60.81.6 2.5-cm2.54.97.51.524 5-cm5.09.815348 10-cm10.019.6306816 15-cm15.029.44591224 20-cm20.039.260121632 33.3-cm33.365.31002026.753.3 66.7-cm66.7130.72004053.3106.7 100-cm100.0196.03006080160 333.3-cm333.3653.31000200266.7533.3 15

16. Vertical accuracy of the new ASPRS 2014 standard compared with legacy standards Vertical Accuracy Class RMSE z Non-Vegetated (cm) Equivalent Class 1 contour interval per ASPRS 1990 (cm) Equivalent Class 2 contour interval per ASPRS 1990 (cm) Equivalent contour interval per NMAS (cm) 1-cm1.03.01.53.29 2.5-cm2.57.53.88.22 5-cm5.015.07.516.45 10-cm10.030.015.032.90 15-cm15.045.022.549.35 20-cm20.060.030.065.80 33.3-cm33.399.950.0109.55 66.7-cm66.7200.1100.1219.43 100-cm100.0300.0150.0328.98 333.3-cm333.3999.9500.01096.49 16

17. Examples on Vertical Accuracy and Recommended Lidar Point Density for Digital Elevation Data according to the new ASPRS 2014 standard Vertical Accuracy Class Absolute Accuracy Recommended Minimum NPD (pts/m 2 ) Recommended Maximum NPS (m) RMSE z Non-Vegetated (cm) NVA at 95% Confidence Level (cm) 1-cm1.02.0≥20≤0.22 2.5-cm2.54.9160.25 5-cm5.09.880.35 10-cm10.019.620.71 15-cm15.029.411.0 20-cm20.039.20.51.4 33.3-cm33.365.30.252.0 66.7-cm66.7130.70.13.2 100-cm100.0196.00.054.5 333.3-cm333.3653.30.0110.0 17 3DEP QL2

18. Horizontal accuracy requirements for elevation data 18

19. Expected horizontal errors (RMSE r ) for Lidar data in terms of flying altitude Altitude (m) Positional RMSE r (cm) Altitude (m) Positional RMSE r (cm) 50013.1 3,00041.6 1,00017.5 3,50048.0 1,50023.0 4,00054.5 2,00029.0 4,50061.1 2,50035.2 5,00067.6 19

20. Low Confidence Areas in Lidar Dataset Vertical Accuracy Class Recommended Project Min NPD (pts/m 2 ) (Max NPS (m)) Recommended Low Confidence Min NGPD (pts/m 2 ) (Max NGPS (m)) Search Radius and Cell Size for Computing NGPD (m) Low Confidence Polygons Min Area (acres (m 2 )) 1-cm≥20 (≤0.22)≥5 (≤0.45)0.670.5 (2,000) 2.5-cm16 (0.25)4 (0.50)0.751 (4,000) 5-cm8 (0.35)2 (0.71)1.062 (8,000) 10-cm2 (0.71)0.5 (1.41)2.125 (20,000) 15-cm1 (1.0)0.25 (2.0)3.005 (20,000) 20-cm0.5 (1.4)0.125 (2.8)4.245 (20,000) 33.3-cm0.25 (2.0)0.0625 (4.0)6.010 (40,000) 66.7-cm0.1 (3.2)0.025 (6.3)9.515 (60,000) 100-cm0.05 (4.5)0.0125 (8.9)13.420 (80,000) 333.3-cm0.01 (10.0)0.0025 (20.0)30.025 (100,000) 20

21. Accuracy requirements for aerial triangulation and INS-based sensor orientation of digital imagery Accuracy of aerial triangulation designed for digital planimetric data (orthoimagery and/or digital planimetric map) only: RMSE x(AT) or RMSE y(AT) = ½ * RMSE x(Map) or RMSE y(Map) RMSE z(AT) = RMSE x(Map) or RMSE y(Map) of orthoimagery Accuracy of aerial triangulation designed for elevation data, or planimetric data (orthoimagery and/or digital planimetric map) and elevation data production: RMSE x(AT), RMSE y(AT) or RMSE z(AT) = ½ * RMSE x(Map), RMSE y(Map) or RMSE z(DEM) 21

22. Accuracy requirements for ground control used for aerial triangulation Accuracy of ground controls designed for planimetric data (orthoimagery and/or digital planimetric map)production only: RMSE x or RMSE y = ¼ * RMSE x(Map) or RMSE y(Map), RMSE z = ½ * RMSE x(Map) or RMSE y(Map) Accuracy of ground controls designed for elevation data, or planimetric data and elevation data production: RMSE x, RMSE y or RMSE z = ¼ * RMSE x(Map), RMSE y(Map) or RMSE z(DEM) 22

23. Examples on Aerial Triangulation and Ground Control Accuracy Product Accuracy (RMSE x, RMSE y ) (cm) A/T AccuracyGround Control Accuracy RMSE x and RMSE y (cm) RMSE z (cm) RMSE x and RMSE y (cm) RMSE z (cm) 50255012.525 23 Product Accuracy (RMSE x, RMSE y, or RMSE z ) (cm) A/T AccuracyGround Control Accuracy RMSE x and RMSE y (cm) RMSE z (cm) RMSE x and RMSE y (cm) RMSE z (cm) 5025 12.5 Aerial Triangulation and Ground Control Accuracy Requirements, Orthoimagery and/or Planimetric Data and Elevation Data Aerial Triangulation and Ground Control Accuracy Requirements, Orthoimagery and/or Planimetric Data Only

24. Reporting Horizontal Accuracy “This data set was tested to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data (2014) for a ___ (cm) RMSE x / RMSE y Horizontal Accuracy Class. Actual positional accuracy was found to be RMSE x = ___ (cm) and RMSE y = ___ cm which equates to +/- ___ at 95% confidence level.” “This data set was produced to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data (2014) for a ___ (cm) RMSE x / RMSE y Horizontal Accuracy Class which equates to +/- ___ cm at a 95% confidence level.” 24

25. Reporting Vertical Accuracy “This data set was tested to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data (2014) for a___ (cm) RMSE z Vertical Accuracy Class. Actual NVA accuracy was found to be RMSE z = ___ cm, equating to +/- ___ at 95% confidence level. Actual VVA accuracy was found to be +/- ___ cm at the 95% percentile.” “This data set was produced to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data (2014) for a ___ cm RMSE z Vertical Accuracy Class equating to NVA =+/-___cm at 95% confidence level and VVA =+/-___cm at the 95% percentile 25

26. Recommended Number of Check Points Based on Area Project Area (Square Kilometers) Horizontal Accuracy Testing of Orthoimagery and Planimetrics Vertical and Horizontal Accuracy Testing of Elevation Data sets Total Number of Static 2D/3D Check Points (clearly-defined points) Number of Static 3D Check Points in NVA Number of Static 3D Check Points in VVA Total Number of Static 3D Check Points ≤50020 525 501-75025201030 751-100030251540 1001-125035302050 1251-150040352560 1501-175045403070 1751-200050453580 2001-225055504090 2251-2500605545100 26

27. ASPRS low confidence areas for lidar (and photogrammetry) 27 Where the bare-earth DTM may not meet overall data accuracy requirements (dashed contours in the past) Low confidence areas are required and delivered as 2-D polygons based on four criteria: –Nominal ground point density (NGPD) –Cell size for raster analysis –Search radius to determine average ground point densities –Minimum size area appropriate to aggregate ground point densities and show a generalized Low Confidence Area (minimum mapping unit)

28. New Standards Highlights –Not Yet Addressed: Methodologies for accuracy assessment of linear features (as opposed to well defined points) Rigorous total propagated uncertainty (TPU) modeling (as opposed to -- or in addition to – ground truthing against independent data sources) Robust statistics for data sets that do not meet the criteria for normally distributed data and therefore cannot be rigorously assessed using the statistical methods specified herein Image quality factors, such as edge definition and other characteristics Robust assessment of check point distribution and density Alternate methodologies to TIN interpolation for vertical accuracy assessment 28

29. NSSDA Issue The NSSDA repeatedly references Greenwalt and Schultz, 1968 but then appears to substitute RMSE (measure of accuracy) where Greenwalt and Schultz used sample standard deviation (measure of precision). Results are similar with a large number of checkpoints and as mean errors approach zero. ASPRS also chose RMSE but without reference to Greenwalt and Schultz In 1968, it was nearly impossible to have checkpoints with known absolute accuracy, as we approximate today with GPS surveys relative to CORS. 29

30. Normal Error Distribution The NSSDA assumes that the data set errors are normally distributed and that any significant systematic errors or biases have been removed. Errors for lidar datasets in vegetated terrain typically do not follow a normal error distribution. ASPRS states: “It is the responsibility of the data provider to test and verify that the data meet those requirements including an evaluation of statistical parameters such as the kurtosis, skew, and mean error, as well as removal of systematic errors or biases in order to achieve an acceptable mean error prior to delivery.” 30

31. Mean Errors “The exact specification of an acceptable value for mean error may vary by project and should be negotiated between the data provider and the client. As a general rule, these standards recommend that the mean error be less than 25% of the specified RMSE value for the project. If a larger mean error is negotiated as acceptable, this should be documented in the metadata. In any case, mean errors that are greater than 25% of the target RMSE, whether identified pre-delivery or post-delivery, should be investigated to determine the cause of the error and to determine what actions, if any, should be taken. These findings should be clearly documented in the metadata.” 31

32. Example on Applying the New Vertical Standard for LiDAR 32 This actual LiDAR dataset, with 20 QA/QC checkpoints in each of 5 land cover categories, has a normal error distribution except for two outliers that are typical in forests, weeds and crops.

33. Standard statistics show skewed results in 2 land cover categories with 1 outlier each Land Cover Category Checkpoints Minimum (cm) Maximum c(m) Mean (cm) γ 1 skew γ 2 kurtosis ѕ std dev (cm) RMSE z (cm) Open Terrain 20-108-2-0.19-0.6455 Urban Terrain 20-15111-0.840.2277 Weeds & Crops 20-134922.689.4313 Brush Lands 20-10174-0.18-0.3178 Fully Forested 20-137033.0811.461817 Consolidated 100-157023.1817.1211 33

34. Example on Applying the New Vertical Standard for LiDAR 34 Land Cover Category Prior MethodsOld/New Terms NSSDA’s Accuracy(z) at 95% confidence level based on RMSE z x 1.9600 NDEP’s FVA, plus SVAs and CVA based on 95 th Percentile NDEP Accuracy Term ASPRS Vertical Accuracy ASPRS Accuracy Term Open Terrain10 cm FVA 12 cmNVA Urban Terrain14 cm13 cmSVA Weeds & Crops25 cm15 cmSVA 16.7 cmVVA Brush Lands16 cm14 cmSVA Fully Forested33 cm21 cmSVA Consolidated22 cm13 cmCVAN/A

35. Reconciling ASPRS and NSSDA, and Other Issues 3.1.2, Scope: NSSDA also applies to fully georeferenced maps; ASPRS does not. 3.2.1, Spatial Accuracy: “Accuracy” vs. “RMSE”. Both documents define each, making a clear distinction, but then use RMSE as Accuracy. 3.2.2, Accuracy Test Guidelines, Para.4: "When 20 points are tested, the 95% confidence level allows one point to fail the threshold given in the product specification." This gives the impression that one point can be thrown away. Reword or delete. 35

36. Reconciling ASPRS and NSSDA, and Other Issues 3.2.2, Accuracy Test Guidelines, Para.5: Add Percentile (ASPRS) to these three methods (Deductive Estimate, Internal Evidence, Comparison to Source). 3.2.3, Accuracy Reporting, Para 1: DEMs/Contours, other data without well-defined points should not report H-Acc. This is not clear in the ASPRS standard 3.2.3, Accuracy Reporting, Para 2: Accuracy reporting for single "datasets" that have multiple elements. ASPRS does not address this. 36

37. Reconciling ASPRS and NSSDA, and Other Issues 3.2.3, Accuracy Reporting, Para 3&4: Native units vs. all- metric. All-metric (more specifically, all Meters) makes comparison across datasets easier. USGS requires reporting in meters for all lidar tags in XML metadata. Appendix 3-B, Horizontal Accuracy Computations: ASPRS does not reference the NMP Technical Instructions, Procedure for Map Accuracy Testing (1987). Is it still necessary? Appendix 3-C, Testing Guidelines: Some good verbage here (not in ASPRS) that should be retained. 37

38. Reconciling ASPRS and NSSDA, and Other Issues The NSSDA gives specific reference to the CSDGM (Content Standard for Digital Geospatial Metadata) entries where accuracies and other relevant information is to be reported; ASPRS does not. This needs to be reviewed and retained; perhaps include ISO entries as well? Direct adoption of the ASPRS Standard, as-is, would leave some gaps and confusion but it can be fixed in future editions of the standard. Reconciling both documents into a single standard, preferable the latest ASPRS 2014, will take a bit of editing and review, but they are very close and the result would be well worth the effort. 38

39. The Standard Web Site The final standard document is posted on the web page: http://www.asprs.org/PAD-Division/ASPRS-POSITIONAL- ACCURACY-STANDARDS-FOR-DIGITAL-GEOSPATIAL- DATA.html 39

40. Thank You! 40