1. Generating Raster DEM from Mass Points via TIN Streaming Jack Snoeyink UNC Chapel Hill Jonathan Shewchuk UC Berkeley Tim Thirion UNC Chapel Hill Martin Isenburg UC Berkeley Yuanxin Liu UNC Chapel Hill raster DEM LIDAR points

2. Generating DEMs from LIDAR LIDAR points raster DEM resampling interpolatingtriangulating temporary TIN i j

3. Billions of Points courtesy of www.ncfloodmaps.com 0.5 billion points 11 GB Neuse-River Basin Elevation map Contour map

4. Streaming TIN output output final DEM to disk DEM Generation via TIN Streaming small buffer in main memory Streaming Computation of Delaunay Triangulation read points from disk Immediate Rasterization of Streaming TIN store elevation rasters to temporary files (grouped by rows)

5. Example Result 500,141,313 Points 11 GB ( binary, xyz, doubles )  on a household laptop with two harddisks  50,394 × 30,500 DEM 3 GB ( binary, BIL, 16 bit, 20 ft ) raster DEM LIDAR points  in 67 minutes   64 MB of main memory   270 MB temporary disk space 

6. Related Work

7. break data into (overlapping) tiles popular GIS packages – ArcGIS 9.1 ( limit: ~ 25 million points, 1 GB ) – QTModeller 4 ( limit: ~ 50 million points, 1 GB ) – GRASS ( limit: ~ 25 million points, 1 GB ) Rasterizing Billions of Points [ Agarwal et al. ‘06 ]

8. Inverse Distance Weighting ( IDW ) Natural Neighbors Kriging Splines TIN – linear – quintic … Interpolation Methods

9. External Memory Quadtree rearrange points on disk into quadtree From Point Cloud to Grid DEM: A Scalable Approach [ Agarwal et al. ‘06 ] process cell – find its neighbors – interpolate all ( i,j ) – write rasters to temporary file sort temporary file on ( i,j ) & write DEM

10. reorganize computations … not points Our Approach: TIN Streaming – compute TIN in places where all points have already arrived – raster, output & deallocate – keep parts that miss neighbors enhance points with spatial finalization

11. reorganize computations … not points Our Approach: TIN Streaming enhance points with spatial finalization – compute TIN in places where all points have already arrived – raster, output & deallocate – keep parts that miss neighbors

12. reorganize computations … not points Our Approach: TIN Streaming – compute TIN in places where all points have already arrived – raster, output & deallocate – keep parts that miss neighbors enhance points with spatial finalization

13. Delaunay Triangulation a triangulation in which every triangle has an empty circumscribing circle standard algorithm for TIN construction good properties for interpolating points

14. locate triangle enclosing the point Incremental Point Insertion find and remove all triangles with non- empty circumcircles triangulate by connecting new point [ Lawson ’77 ] [ Bowyer ’81 ] [ Watson ’81 ]

15. Spatial Finalization Tags inject tags into the point stream that inform when grid cells are free of future points finalized region future points only arrive in these cells use tags to certify triangles as being part of the final triangulation and output them

16. Streaming TIN Generation output to diskinput from disk small buffer in main memory triangulate point stream seamlessly output completed parts & reuse memory finalization tags small buffer in main memory another small buffer finalization tags Streaming Computation of Delaunay Triangulations [ Isenburg et al. ‘06 ] output (pipe) streaming TIN to rasterizer

17. input (pipe) streaming TIN from triangulator Rasterization of Streaming TIN input from disk yet another small buffer output final DEM to disk points from single UTM time zone triangles with long edges are killed output to a single row-ordered DEM

18. Streaming Rasterization Pipeline spfinalize -i points.raw -ospb | spdelaunay –ispb –osmb -ospb | tin2iso –ismb –o dem.bil

19. The Finalizer (spfinalize)

20. Spatial Finalization of Points  compute bounding box

21. 2 11 3 4 5 5 1 2 6 1 8 4 7 5 5 7 1 7 4 1 9 7 6 4 Spatial Finalization of Points 8 9 7 9 8 5 4 8 2 7 7 8 8 6 77 73 9 7 88 9 6 6 2 7 2 7 3 7 5 8 79 8 7 6  compute bounding box  create finalization grid – count number of points per cell

22. 5 8 4 7 9 7 6 4 8 9 7 9 8 5 4 8 2 7 7 8 8 6 77 73 9 7 88 9 6 6 2 7 2 7 3 7 5 8 79 8 7 6 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

23. 1 7 4 7 9 7 3 4 8 9 7 9 8 5 4 8 2 7 7 8 8 6 77 73 9 7 88 9 6 6 2 7 2 7 3 7 5 8 79 8 5 6 0 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

24. 0 2 4 5 9 6 0 4 8 9 7 9 8 5 4 8 2 7 7 8 8 6 77 73 9 7 88 9 6 6 2 7 2 7 3 7 5 8 79 8 5 6 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

25. 0 0 3 3 9 3 0 4 8 9 7 9 8 5 4 8 2 7 7 8 8 6 77 73 9 7 88 9 6 6 2 7 2 7 3 7 5 8 79 8 0 4 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

26. 07 0 4 8 9 7 9 8 5 4 8 2 7 7 8 8 6 77 73 9 7 88 9 6 6 2 7 2 7 3 7 1 7 21 1 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

27. 02 0 4 8 9 7 9 8 5 4 8 2 4 5 1 6 5 76 88 8 6 6 2 7 2 3 1 6 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

28. 02 0 4 3 2 9 7 4 1 7 2 1 23 3 5 5 2 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

29. 02 0 4 1 Spatial Finalization of Points  compute bounding box  create finalization grid – count number of points per cell  output finalized points – buffer per grid cell – if full, output points in one randomized chunk followed by finalization tag

30. Exploit the Existing Coherence raw point stream finalized point stream document with finalization tags enhance by releasing chunks of points  benefits of global sort, but cheaper

31. Exploit the Existing Coherence document with finalization tags enhance by releasing chunks of points  benefits of global sort, but cheaper raw point stream finalized point stream

32. Unfinalized Region as Quadtree as tags finalize cell after cell, maintain remaining cells in adaptive quadtree  used for efficient overlap tests by the triangulator finalization tags

33. The Triangulator (spdelaunay)

34. Modified Incremental Delaunay insert points when reading a finalization tag – find certified triangles – output them to disk or pipe – deallocate data structure

35. Modified Incremental Delaunay insert points when reading a finalization tag – find certified triangles – output them to disk or pipe – deallocate data structure

36. Modified Incremental Delaunay insert points when reading a finalization tag – find certified triangles – output them to disk or pipe – deallocate data structure

37. The Rasterizer (tin2dem)

38. write raster directly into final DEM on disk write raster to temporary files on disk ( use small buffer, compress when writing buffer ) Rasterizing the Streaming TIN

39. Results

40. Neuse- River Basin

41. Pipeline Details for the 20 ft DEM 6 min 5 MB 512 x 512 grid 10 MB 60,388 triangles 19 MB 444 million rasters 43 min 35 MB 2,249,268 points 270 MB 239 files 8 min 46 MB 128 rows per file

42. 53 hours – 3.4 GHz desktop – 10,000 RPM disks – scratch space: quadtree: ~ 8 GB rasters: ~ 2.5 GB Comparison to ( 20 ft ) [ Agarwal et al. ‘06 ] 67 minutes – 2.1 GHz laptop – 5,400 RPM disks – scratch space: compressed rasters: 270 MB 86 % of CPU time for expensive RST – remaining 7.5 hours constructing quadtree: 1.1 hours finding cell neighbors: 5.6 hours

43. Demos & Discussion

44. Demos streaming generation of DEM extraction of elevation contours

45. final DEM Streaming DEM Rasterization raw points

46. avoid temporary query structure on disk – data-driven, no gathering of neighbors Main Features order of magnitude faster than method that uses external memory no global sort – don’t fight the data, use existing coherence interleave input & computation & output – 100% CPU usage while interpolating

47. other interpolation methods – natural neighbors – inverse distance weighting Future Work breaklines / barriers ( e.g. rivers, roads ) – use Constraint Delaunay Triangulation stream other tasks – thinning of points – compression of raw LIDAR data ( useful? ) – hydrological enforcement ( streamable? )

48. Acknowledgements data – Kevin Yi, Duke support – NSF grants 0429901 + 0430065 "Collaborative Research: Fundamentals and Algorithms for Streaming Meshes." – NGA award HM1582-05-2-0003 – Alfred P. Sloan Research Fellowship

49. Thank You executables, video, slides, paper, and soon also source code: http://www.cs.unc.edu/~isenburg/