LiDAR Data Products in Indiana ISPLS Workshop January 18 th, 2013 Christopher Morse NRCS Indiana GIS Coordinator Special thanks to: Jim Sparks Phil Worrall R. Wilkinson Helping People Help the Land USDA is an equal opportunity provider and employer. Data Overview and Access

Digital Elevation Data A Short Review  Raster data (evenly spaced, gridded data)  Cells hold values for the height of a feature or site referenced to a common vertical datum  Resolution refers to the size of the pixels in the data  A DEM with 30 meter resolution is composed of all cells being 30 meters x 30 meters in the x and y directions and each cell holds a single elevation value (z)  Elevation value (z) could be stored in a variety of units Helping People Help the Land USDA is an equal opportunity provider and employer.

Digital Elevation Data Definitions paraphrased from Maune et al, 2 nd edition DEM Users Manual (Terms often used interchangeably)  DEM = Digital Elevation Model; Typically Bare Earth or terrain  DTM = Digital Terrain Model; Similar to DEM with the addition of some elevations for significant topographic features on the land defined by mass points or break lines  DSM = Digital Surface Model; Similar to a DEM or DTM, but shows the tops of all surfaces including buildings, trees, and other features above the bare earth Helping People Help the Land USDA is an equal opportunity provider and employer. Image from State of Indiana Orthophotography and LiDAR Program, presentation by R.N. Wilkinson

Digital Elevation Data Definitions paraphrased from Maune et al, 2 nd edition DEM Users Manual:  Mass Points are irregularly spaced vector points with an x,y, and z value.  Breaklines are linear features that describe changes in the terrain surface (roads, streams, building footprints, etc…) Helping People Help the Land USDA is an equal opportunity provider and employer.

Digital Elevation Data Common Resolutions, locally (not all resolutions are available in all areas)  USGS primarily 30 meter, 10 meter; some 3 meter Helping People Help the Land USDA is an equal opportunity provider and employer.

Digital Elevation Data Common Resolutions, locally (not all resolutions are available in all areas)  Indiana has 2005 elevation data at 5 foot resolution for all parts of the state  Indiana is generating new DTM data for at 5 foot resolution  Local governments may hold a variety of high resolution products Helping People Help the Land USDA is an equal opportunity provider and employer.

Digital Elevation Data Why generate new DEMs at the same resolution?  Funds  Dramatically Improved Vertical Accuracy  USGS National Elevation Data (2003) = 2.44 meters RMSE  Indiana 2005 Data = 6 feet RMSE  Indiana Data = 18.5 cm RMSE Vertical Accuracy is a critical factor when considering best supportable contour interval Helping People Help the Land USDA is an equal opportunity provider and employer.

Digital Elevation Data What does that mean?  Best supportable auto-generated contours:  USGS National Elevation Data = over 20 foot interval  Indiana 2005 Data = 20 foot interval  Indiana Data = 2 foot interval  Auto generated results limited without use of breaklines Helping People Help the Land USDA is an equal opportunity provider and employer. Image source: wikipedia.org/wiki/Contour_line

Digital Elevation Data Indiana’s new DEMs  Derived from LiDAR  Actually a DTM due to the inclusion of breaklines for some hydro features  Rivers greater than 100 feet wide and water bodies of 2 acres or greater digitized from accompanying imagery  Rivers digitized in direction of flow and water bodies with a set elevation (water bodies will be “flat” and rivers will flow “downhill”)  No LiDAR points used within 1.5 meters of a digitized breakline  State Plane Coordinates (NAD 83 feet, NAVD 88)

Digital Elevation Data Indiana’s new DEMs  5 foot horizontal pixel resolution  Supports 2 foot contours (thus at least 18.5 cm vertical RMSE accuracy met)  Part of the IndianaMap   DEMs Available for download from:  Indiana Spatial Data Portal:

Digital Elevation Data Resources on Digital Elevation Data USGS National Elevation Dataset (NED) Info: Text: Digital Elevation Model Technologies and Applications: The DEM Users Manual, 2 nd Edition Edited by David F. Maune, PhD, CP Published by ASPRS, 2007 Helping People Help the Land USDA is an equal opportunity provider and employer.

LiDAR Data  LiDAR = Light Detection And Ranging  Uses an active sensor to emit energy (light) and detect returned energy  Can be collected day or night) Image from State of Indiana Orthophotography and LiDAR Program, presentation by R.N. Wilkinson

LiDAR Data  Airborne and Terrestrial capabilities  Combines GPS and an Inertial Measurement device to compute x,y,z positions  Every point recorded has an x,y,z, and intensity value

LiDAR Data  All reflections of emitted energy are returned, generating a point cloud of the data  The point cloud contains data points for scan hits at multiple heights on objects, as well as some noise due to atmospheric conditions.  These hits are referred to as returns and are referenced in ascending order from highest elevation to lowest elevation for a set of returns  Top of a building or tree is the 1 st return  Canopy of a tree or side of a building is 2 nd or 3 rd return, and so on as the returned hits descend in elevation

LiDAR Data All returns 1 st return 2 nd return 3 rd return 4 th return Image from Lidar Technology Overview, presentation by USGS, June 2007

LiDAR Data  The vendor uses classification algorithms on the data  Vendor delivers a data product depending on the customer’s specifications

LiDAR Data Indiana’s LiDAR Data ( )  Classified Point Data  1.5 meter Nominal Pulse Spacing (the estimated average spacing of irregularly-spaced points in both the along-track and cross-track directions – FEMA)  LAS files in 5000’ x 5000’ tiles  Data Delivered in appropriate State Plane Coordinate System (NAD 83 Feet, NAVD88)

LiDAR Data Indiana’s LiDAR Data ( ) Classification Scheme (This is not the same as 1 st return, 2 nd return)  1 = Processed but Unclassified  2 = Bare Earth/Ground  7 = Noise  9 = Water  10 = Ignored Ground (breakline proximity)  13 = Bridges (over 100 feet in length; foot bridges not included)

LiDAR Data Some LiDAR Resources USGS LiDAR Guidelines (replaces draft versions 13 and 14) NDEP Guidelines for Digital Elevation Data, Version 1.0 (2004) (Currently in work for an updated release) Education (most class materials available freely)

LiDAR Data LiDAR’s Limitations (in designs)  Site changes – snapshot on day of flight  Grade breaks – collection pattern is “random” and not based on changes in grade as a field survey  Critical elevations – may not detect control elevations such as building floor elevations, edges of concrete, property boundaries or culvert inlet/outlet elevations (requires local benchmarking at site and adjustment of data to benchmark)  Vegetation – May affect readings, dependent on quality of the data, density of vegetation. Tillage may affect surface smoothness (can affect slope calculations)  Water – LiDAR can penetrate water, but type of laser and water turbidity can affect this. Standing water can invalidate a local elevation estimate from LiDAR. If you believe a data result is due to influence of water, don’t use it for an elevation

LiDAR Data LiDAR’s Uses (in designs)  Planning – Visualization of data and its derivaties (hillshades and contours) can be very useful in planning  Preliminary Design – LiDAR relative accuracy is typically very high for a site, so preliminary design for a number of uses can be done with CAD generated surfaces and later tied to a site’s elevations through adjustment to benchmarks obtained in a field survey (if the site has not undergone major change since the collect)

Indiana Data ( )

Acquiring Indiana LiDAR Data IndianaMap & Indiana Spatial Data Portal ( View and File-based access to point cloud and hydro- flattened DEM datawww.IndianaMap.org Open Topography Server (UCSD) Key advantages  User Defined Area of Interest  Mitigates need for local storage of unneeded data  Opens the door to Indiana data for all users  Leverage server side processing for extraction and derivatives Helping People Help the Land USDA is an equal opportunity provider and employer.

OT Links Open Topography Home Page:   Open Topography Data Page:  cid=datasets cid=datasets Indiana’s LiDAR Data Home Page:*  * Recommend you use this home page – Indiana news, tips, tricks, documentation, and instructions will be poster here. Helping People Help the Land USDA is an equal opportunity provider and employer.

Select a Region Helping People Help the Land USDA is an equal opportunity provider and employer.

Results Helping People Help the Land USDA is an equal opportunity provider and employer.

More Helping People Help the Land USDA is an equal opportunity provider and employer.

Get Data (Top) Helping People Help the Land USDA is an equal opportunity provider and employer.

Get Data (Mid) Helping People Help the Land USDA is an equal opportunity provider and employer.

Get Data (Bottom) Helping People Help the Land USDA is an equal opportunity provider and employer.

Choices That’s a lot of boxes! How do I get what I need? Helping People Help the Land USDA is an equal opportunity provider and employer.

Choices Understand what data you really need  What task are you trying to accomplish?  What data do you really need for that task?  Decisions will depend on uses for LiDAR in which you are engaged Helping People Help the Land USDA is an equal opportunity provider and employer.

Choices Understand what data you really need  Do you need LiDAR points (LAS files) for particular areas?  Get it here  Do you need the bare earth points or the entire point cloud?  Decision driven by your intended use  Do you need a DEM, TIN or derivatives?  You may not want to get it here (yet) unless you need a custom DEM or cannot process TINs or generate derivatives in your own applications Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 1 - Basics  Select Area Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 1 - Basics  Review number of points and modify extent if necessary  Set Ground Classification  This choice drives the nature of the derived DEM/TIN later  Select Coordinate System  Note that this choice will affect your units in later steps Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 2 – Point Data Format  Preference and capabilities  LAS = Larger d/l, no decompression needed  LAZ = smaller d/l, must decompress  ASCII = largest d/l, no decompression needed Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation  Gridding Parameters  Remember: Units match projection choice from step #1c.  Resolution: At least the point spacing of the dataset (1.5 m or about 5 feet)  Radius: At least the resolution of the dataset. The larger the radius, the more “smoothed” the DEM. Using 2x the resolution guarantees a “3-cell filter” Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Methods Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Zmean Grid – A basic averaging method  Can be used when making any surface to average out data irregularities that would be emphasized by Zmin or Zmax  If you chose all points in step #1b, but you still want a DEM of the “bare earth”, you can select Zmin instead, but it will have different results than Zmean. The differences may or may not be significant to your work, but only you can determine that.  Not clear if this implements any nearest neighbor weightings, but from the description it seems not. Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Zidw Grid  IDW = Inverse Distance Weighting – explicitly implements an assumption that things that are close to one another are more alike than those that are farther apart. Is also an averaging method.  Resulting surface will not pass through the sample points.  No option to control the power factor (possibly default of 2)  IDW in general is not recommended for gridding Terrains per Maune et al, Digital Elevation Models Technologies and Applications: The DEM Users Manual, 2 nd Edition; page 10. It is not clear if the particular IDW algorithm and settings at Open Topography would contradict that. Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  The Zmean to Zidw difference (2ft contours from 5 ft derived DEM, smoothed with PAEK, 50 ft) Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Which method should I use?  It depends on your needs  Zmin can give bare earth values when using ground returns or entire point cloud, but will always assign the lowest value in the search radius  Zmax can give a first return surface when using all returns  Zmean is a basic average of points to simulate bare earth when using ground returns (can be a very reasonable DEM) or of entire surface when using all returns  Zidw is like Zmean but more specialized to weight points that are nearer – may result in some sharp exaggerations or newly introduced inaccuracy. Typically not the best for gridding terrain data. Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Which method should I use?  The good news  It’s multiple choice! Try them all at once.  Alternately, you can generate your own DEM from the LiDAR points you download using a variety of tools of your own and you do not need to generate a DEM here at all (however, this example is for a DEM, so this is just a reminder that this step is optional and specialized).  You could also download the final project DEMs from ISDP, and in the future, OT, without processing here at all and you can resample from those. Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Formats – ArcASCII Grid, GeoTIFF, IMG, or All Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 3 – DEM Generation (cont’d)  Null Filling  Will fill in small blank areas in the DEM being generated at 3, 5, or 7 pixel filter sizes  Your choice depends on your project needs, but setting a value here will minimize tiny holes of no data in the resulting surface  Will not fix large holes from water bodies or buildings Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 4 – Derivative Products  Hillshade will use altitude of 45 degrees and azimuth of 315 degrees  Slope will be degrees, not percent Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Section 5 – Visualization Products  Optional  Can generate files for use in Google Earth  Not used in this example (will uncheck) Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Runtime Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Results Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Results  Data can be added straight to ArcMap, however statistics will not be calculated by default  You can use tools in ArcGIS to calculate statistics  Alternately, you can force the statistics to calculate under the Symbology tab by switching back and forth from None to Standard Deviations under the Stretch method drop down box Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Results (Zmean, elevation, hillshade) Helping People Help the Land USDA is an equal opportunity provider and employer.

Ground Return (LAS, DEM, No TIN) Example Results (Zidw, elevation, hillshade) Helping People Help the Land USDA is an equal opportunity provider and employer.

Limitations Area of interest scope is limited to 50 million points if you are not logged into OT You can increase this to 150 million points when logged into OT Point density and natural geography will cause the approximate area corresponding to that number of points to fluctuate Ground vs. All will return different point count Helping People Help the Land USDA is an equal opportunity provider and employer.

Limitations Working with LAS Data  Typically requires a tool that supports LAS format  Alternately can convert to a 3d x,y,z format Raw LiDAR data can have unclassified points and gaps in coverage depending on the return or classification selected Helping People Help the Land USDA is an equal opportunity provider and employer.

Limitations Derivative products can have similar gaps, such as the DEM or hillshade generated at OT Break lines are not used to generate data from OT This is due to the file being generated from ONLY the LiDAR points by the OT site DEMs Delivered from the vendor have been post processed to fill in such areas and to use some breaklines (will be added to OT, currently hosted at ISDP) Helping People Help the Land USDA is an equal opportunity provider and employer.

Some Uses for OT Generated DEM Analysis in open terrain with few to no structures or water bodies Can compensate for gaps from buildings and water bodies in the OT generated DEM using some techniques in ArcGIS Spatial Analyst to further fill holes in the surface Helping People Help the Land USDA is an equal opportunity provider and employer.

Fill the blanks in the DEM  Raster calculator (re-run until no NoData returns)  Arc10: Con(IsNull(Raster), FocalStatistics(Raster, NbrCircle(10, "CELL"), "MEAN"), Raster) Where Raster is the name of the DEM layer, Circle is the search type, 10 is the radius in cells. These criteria can be modified.  Arc 9: Con(IsNull(Raster), FocalMean(Raster, rectangle,10,10), Raster) Where Raster is the name of the DEM layer, rectangle is the search type, 10x10 is the rectangle size (in cells). These criteria can be modified. Helping People Help the Land USDA is an equal opportunity provider and employer.

Fill the blanks in the DEM  Model Builder (Arc 10)  Same expression, constructed in model builder Helping People Help the Land USDA is an equal opportunity provider and employer.

Fill the blanks in the DEM  Python (with Arc 10)  Add a while loop to continue processing as long as IsNull generates any True (1) results Helping People Help the Land USDA is an equal opportunity provider and employer.

Recommended Settings To get a “bare earth” dataset at Open Topography  Return Classification = Ground  Coordinate System = User preference  Point Format = User Preference  DEM Generation (optional)  Method = Zmean grid  Resolution = at least 1.5 m (UTM) or 5 ft (St. Plane)  Radius = at least 3x the resolution  Null Filling = 7 (smooth out small gaps)  Should produce essentially a bare earth DEM, gaps for water and buildings will be present Helping People Help the Land USDA is an equal opportunity provider and employer.

Recommended Settings Example of a Zmean surface (no exaggeration) Helping People Help the Land USDA is an equal opportunity provider and employer.

Recommended Settings To get a “1 st return” dataset at Open Topography  Return Classification = All  Coordinate System = User preference  Point Format = User Preference  “DEM” Generation (optional)  Method = Zmax grid  Resolution = at least 1.5 m (UTM) or 5 ft (St. Plane)  Radius = same as resolution (minimize radius to decrease blending of vertical features)  Null Filling = 7 (smooth out small gaps)  Should produce a surface approximating a DSM, but will lack true vertical definition of features, gaps for water may be present, may be affected by noise in the data Helping People Help the Land USDA is an equal opportunity provider and employer.

Recommended Settings Example of a Zmax surface Helping People Help the Land USDA is an equal opportunity provider and employer.

Questions? Chris Morse USDA-NRCS NRCS Indiana GIS Coordinator Helping People Help the Land USDA is an equal opportunity provider and employer.