1. 1 Geomatics and Water Resources Research Group Seminars Autumn Term 2007 Dr. Fernando J. Aguilar Torres Department of Agricultural Engineering, University of Almeria, Spain A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DIGITAL ELEVATION MODELS Newcastle, September 2007

2. 2 1.Introduction. 2.Accuracy assessment of DEMs. 3.Reference Standards (official guidelines). Are they enough? 4.Do we really know the reliability of our DEM accuracy measures? 5.Our methodological proposal in the case of LiDAR derived DEMs. 6.Modelling LiDAR error. Preliminary results. 7.Conclusions. Schedule A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007

3. 3 1. Introduction A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 DEM? What is a DEM? Z = f(x,y) A DEM is a digital and mathematical representation of an existing or virtual terrain by means of storing the land elevations (void of vegetation and manmade features) usually at regularly spaced intervals in x and y directions.

4. 4 1. Introduction A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 DEM?

5. 5 1. Introduction A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Applications of DEMs  Hydrological and erosion models  Viewshed analysis and visual impact  Flood risk analysis  Planning of land development. Suitability models (GIS)  Civil Engineering (cut and fills calculation)  Relief description and geomorphology (slopes, aspects and so on)  Topographic correction of remote sensing imagery, insolation and shadowing models  3D visualisation and virtual environments  Orthoimages generation

6. 6 1. Introduction A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Orthoimages generation 2D environment 3D environment

7. 7 1. Introduction A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Error propagation from DEM to the final product Aguilar et al. Annual International Conference ADM and INGEGRAF. Perugia, Italy, June 2007.

8. 8 2. Accuracy assessment of DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Why? Why? A responsible DEM user must be able to answer the following questions (planning):  What precisely is the application for the DEM?  What type of DEM will best meet these needs?  How do I know that I am getting what I ordered? USER PRODUCER

9. 9 2. Accuracy assessment of DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 How to compute it? Statistical inference (Sampling theory) How to compute it? Statistical inference (Sampling theory) Check points selection (finite sample N) Differences between z DEM and z from an independent source of higher accuracy RMSE and ME calculation DEM quality evaluation Check Points Error (Z DEMi -Z CPi = e i )

10. 10 2. Accuracy assessment of DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Types of errors Types of errors 1.Blunders or Outliers 2.Systematic (bias) errors (constant offset) 3.Random errors (random fluctuations in the measurements)

11. 11 2. Accuracy assessment of DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Types of errors Types of errors Systematic errors (A and B) Spatially autocorrelated errors (C) Random errors with no spatial autocorrelation (D) P. Fisher and N. Tate, 2006. Causes and consequences of error in DEMs. Progress in Physical Geography 30(4): 467-489 Ground truth DEM

12. 12 3. Reference Standards A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 National Standard for Spatial Data Accuracy (NSSDA, US) National Standard for Spatial Data Accuracy (NSSDA, US) > 20% Federal Geographic Data Commitee U.S., 1998 Minimum distance between check points >0,10 diagonal > 20 check points Assumption of a normal distribution of residuals and the absence of systematic errors

13. 13 3. Reference Standards A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 National Standard for Spatial Data Accuracy (NSSDA, US) National Standard for Spatial Data Accuracy (NSSDA, US) 95% confidence level Compiled to meet...... meters vertical accuracy at 95% confidence level Check points selection (finite sample N>20) Differences between z DEM and z from an independent source of higher accuracy RMSE calculation Vertical accuracy = 1,96.RMSE

14. 14 3. Reference Standards A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 But is it enough? Some questions arise... But is it enough? Some questions arise...  It is assumed that residuals at check points follow a normal distribution and systematic errors have been “reasonably” removed (no bias), which is known as the “strong assumption”.  We need at least 20 check points. But it is supposing error normal distribution. If not, how many check points do we need? 30, 50, maybe 100? Who controls the reliability of the accuracy assessment process?

15. 15 3. Reference Standards A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 ASPRS Guidelines. Vertical accuracy reporting for LiDAR ASPRS Guidelines. Vertical accuracy reporting for LiDAR Non-open terrain Open terrain Flood, M., 2004. http://www.asprs.org/society/divisions/ppd/standards/Lidar%20 guidelines.pdf Fundamental accuracy (NSSDA protocol) Supplemental accuracy

16. 16 3. Reference Standards A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 ASPRS Guidelines. Supplemental accuracy ASPRS Guidelines. Supplemental accuracy Flood, M., 2004. http://www.asprs.org/society/divisions/ppd/standards/Lidar%20 guidelines.pdf Residuals + - 95th percentile = vertical accuracy at 95% confidence level Maybe used regardless of whether or not the errors follow a normal distribution and whether or not errors qualify as outliers. 5% of the errors will be of larger value.

17. 17 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Bringing Reliability Bringing Reliability We need to quantify which is the error we are committing when we say “the RMSE of this DEM resulted to be..... meters” That error should depend on the number of check points used and somehow the “quality” of the sample from which we have computed the total error (RMSE).

18. 18 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 The Li’s model The Li’s model Li, Z., 1991. Effect of check points on the reliability of DTM accuracy estimates obtained from experimental tests. PE&RS 57(10): 1333-1340. How many check points do we need to evaluate the error at a confidence level of 90% (R=10%): Hypothesis: normal distribution of errors and no bias

19. 19 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 The Aguilar’s model The Aguilar’s model Aguilar F.J. et al., 2007. A theoretical approach to modelling the accuracy assessment of DEMs. PE&RS 73(12): to be published in December. Error population  2 = standardised kurtosis = (  4 /  4 )-3  1 = skewness =  3 /  3 Any assumption, any restriction to use it

20. 20 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Morphology  (cm)  (cm) Skewness (  1 ) Kurtosi s (  2 ) Mountainous0.0211.160.3912.18 Rolling 10.012.720.8413.20 Flat0.012.080.6423.99 Steep rugged hillside 0.4841.010.6021.55 Highly rugged-0.87135.020.1231.95 Slightly mountainous 0.126.361.1221.12 Rolling 2-0.081.84-0.3729.66 Newcastle, September 2007 Residuals datasets (raw data) Residuals datasets (raw data) Leptokurtosis  2 >0 Platykurtosis  2 <0

21. 21 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Residuals datasets (corrected data using 3-sigma rule) Residuals datasets (corrected data using 3-sigma rule) Morphology  (cm)  (cm) SkewnessKurtosisResiduals removed (%) Mountainous-0.128.20-0.043.072.32 Rolling 1-0.131.880.413.792.73 Flat-0.011.35-0.044.152.16 Steep rugged hillside0.2230.340.043.16 1.90 Highly rugged-1.5187.570.026.052.25 Slightly mountainous0.034.420.104.39 2.49 Rolling 2-0.031.11-0.255.112.15

22. 22 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Model validation using Monte Carlo simulation method Model validation using Monte Carlo simulation method Aguilar et al., 2007 (raw data) Aguilar et al., 2007 (filtered data) Li, 1991 (raw data) R 2 =97.32% R 2 =99.28% R 2 =58.07% R 2 =82.61% Li, 1991 (filtered data)

23. 23 4. Reliability of DEM accuracy measures? A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Visualisation of theoretical model Visualisation of theoretical model

24. 24 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Estimating LiDAR vertical accuracies Estimating LiDAR vertical accuracies Non open terrain

25. 25 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Model overview Model overview Error population Non-parametric approach using Estimating Functions Theory for computing mean error confidence intervals Statistical inference from N check points (sample size) Godambe, V.P., 1991. Estimating functions. Oxford University Press, Oxford, 356 pages.

26. 26 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Estimating LiDAR vertical accuracies Estimating LiDAR vertical accuracies Aguilar, F.J. and Mills, J.P. Accuracy assessment of LiDAR derived digital elevation models. The Photogrammetric Record, under review.

27. 27 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Datasets from EuroSDR project on laser scanner Datasets from EuroSDR project on laser scanner 7 datasets with 15 reference data

28. 28 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Datasets from EuroSDR project on laser scanner Datasets from EuroSDR project on laser scanner Terrascan TM last pulse data filtering Comparison with reference data Error datasets for non open terrain

29. 29 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Error datasets from EuroSDR Error datasets from EuroSDR SamplesPointsMean (m)Sd (m)γ1γ1 γ2γ2 % data outliers 11169950·160·443·3011·823·5 12252030·040·166·2654·410·5 2197420·020·051·834·773·1 22211930·040·135·1232·391·7 23108710·050·205·0631·261·4 2436950·040·175·3145·811·6 31153150·010·041·294·161·4 4116260·251·115·0524·411·6 42117430·020·072·9813·222.0 51137010.000·060·265·321·1 52173680·080·302·528·411·7 53247020·160·716·6457·081·6 5438630.000·082·7919·272·1 61310570·010·103·8424·071·5 71125170·020·112·3310·802·1

30. 30 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Results for EuroSDR error datasets Results for EuroSDR error datasets Results corresponding to dataset 1, sample 1

31. 31 5. Accuracy assessment of LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Results for EuroSDR error datasets Results for EuroSDR error datasets

32. 32 6. Modelling error for LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Outlining the approach Outlining the approach Non-open terrain Open terrain

33. 33 6. Modelling error for LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Outlining the approach Outlining the approach Computation at N check points on open terrain IDW method power to 2

34. 34 6. Modelling error for LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Calibrating the empirical component (Information Loss) Calibrating the empirical component (Information Loss) 29 morphologies of 4 has with average slopes ranging from 3% to up to 82% R 2 = 0.9856

35. 35 6. Modelling error for LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Validating the model Validating the model 33 GPS-obtained check points Dataset of LiDAR data captured by Riegl Q560 sensor in August 2006 over Bristol area (Ordnance Survey project). Average density > 0.5 points/m 2 With the permission of the Ordnance Survey Sd = 0.124 m max error = 0.37 m min error = -0.17 m mean error = 0.04 m

36. 36 6. Modelling error for LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Validating the model Validating the model spacing 4.4 m spacing 23.5 m

37. 37 6. Modelling error for LiDAR derived DEMs A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Behaviour of the model Behaviour of the model spacing 3.1 m spacing 4.1 m SDE = 0.15 m spacing 7.1 m

38. 38 7. Conclusions A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007  Very little work has been done to determine the minimum data requirements for specific applications of DEMs, although there is a increasing tendency to collect larger volumes of elevation data. In the majority of the cases it is preferable to have an optimised DEM adapted to our needs rather than to have a vast amount of data, which will be more difficult to handle.  The reference standards methods for accuracy assessment of DEMs are based on hypothesis very restrictive and sometimes not according to reality, above all in the case of LiDAR data un non open terrain.  The tools expound in this talk are seeking to establish more general protocols for testing the quality of the product delivered from the part of the producer or even checking the quality of the own control quality, if there was.

39. 39 8. That’s all A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DEMs Newcastle, September 2007 Thank you very much for your kind attention