1. National Geospatial Intelligence Agency (NGA) Sensor Independent Point Cloud (SIPC)
01 Jun 2015
GPOC: Dr. Narinder Chauhan, NGA/T
Technical Lead: Michael Ingram, Exelis, GS
PM: Yesenia Nunez-Santos, Exelis, GS
Presenter: Jason Smith, PhD, Exelis, GS

2. Topics OGC Open Letter and LAS Why SIPC SIPC Definition Summary
SIPC Format
SIPC Status
SIPC Transition Efforts
SIPC Benefits
SIPC Examples
Sample Code
November 21, 2018

3. UNCLASSIFIED
OGC Open Letter and LAS
There is a concern of new “proprietary” / closed format for compressed LAS proposed by ESRI
LASZip (*.laz) within LASTools / LibaLAS is already an NGA-approved point cloud compression format
However, this should not be confused as a technical recommendation to persist LAS
LAS 1.2 is the current point cloud format standard for NGA
However, LAS has known limitations for the DoD
Inflexible point record formats
Excessive point record metadata not applicable to Geiger mode systems
Version 1.2 at the time did not support additional signal data per point
Needed for more advanced sensors
Binary Point Format (BPF) developed by MIT/LL, JHU APL, NGA
Solved these 2 issues
However, inflexible in bit depth and does not support sparse data or metadata 3

4. Why SIPC Needed Sensor Independent Point Cloud format
UNCLASSIFIED
Why SIPC
Needed Sensor Independent Point Cloud format
Due to LAS and BPF shortcomings, decided to develop ideal format for current and future systems
What is best for the data, let the tools follow
Leverage more advanced file format technology
Both LAS and BPF are non-standard, binary blobs, requiring documentation and continuous and repeated development of byte counting code for different applications
Computer Science community has developed standard file formats for any data:
XML, JSON, CDF, netCDF, HDF, SQLite3
Include their own Application Programing Interface (API)
University of New Hampshire, Data Visualization Lab
“If you find yourself creating a new binary format for your work, please stop. There are too many formats in the world and formats like HDF5 and SQLite3 provide very powerful containers that preexisting libraries understand and these formats are self describing in that you can ask them what they store. It is difficult to create a good binary file format and you will likely make many mistakes that would be avoided. Providing clear documentation of binary file formats is extremely easy to get wrong.”
HDF5: 4

5. SIPC Definition Summary
UNCLASSIFIED
SIPC Definition Summary
Sensor Independent Point Cloud (SIPC) is a new standard development for the transmission of LIDAR point cloud data ready for analyst exploitation
File format = Hierarchical Data Format, Version 5 (HDF5)
Metadata = Conceptual Model and Metadata Dictionary (CMMD)
Data Structure = 8 dimensional sparse point cloud based on physics and redundancy of information
P = {x, y, z, t, i, l, u, c}
Slower changing variables higher in the hierarchy
All information in ground space

6. SIPC Format Goal Determine optimal file format for point cloud data
UNCLASSIFIED
SIPC Format
Goal
Determine optimal file format for point cloud data
Long term solution, what is best for data, not what is compatible now
Method
Extensive 4 phase AoA over 20 file formats
Detailed report available
Conclusion
HDF5 optimal format for ANY point cloud data
Self-Discoverable – no ICD needed
Hierarchical – inherent reduction in redundant data
Object Oriented – maps directly to CMMD objects
Existing API – writing code for readers and writers trivial:
C++, Java, Python, MATLAB, IDL
Used by NASA for all remote sensing data

7. UNCLASSIFIED
SIPC Status
Reviewed by LIDAR Interoperability Working Group (LIWG) in 2014
Led by LTC Crider, NGA
Resulting version called 1.00, dated
Volume 1, Design and Implementation Description Document
SIPC_Vol-2_Ver-1-00_ pdf
Volume 2, HDF Implementation Profile
SIPC_Vol-2_Ver-1-00_ pdf
Currently in review by NITF Technical Board (NTB) and DISA
Based on ULEM, GPM draft came out around same time
Have memo for potential CMMD changes

8. SIPC Transition Efforts
UNCLASSIFIED
SIPC Transition Efforts
File Converters
MATLAB executables (*.exe) for Win64, LAS 1.2 and BPF 3.x
C++ 11 executables (~ 8 x faster), dependency on PDAL, Win64, LAS 1.2
Supports UTM and L3DC coordinate systems
Supports ULEM direct and indirect (UTM only)
SIPC Viewers
IDL SIPC Viewer, PointVis, integrating into Viper
TDA Adoption
Exploitation algorithms being modified – first 1 complete
Cloud Exploitation of SIPC data
Enterprise Solution installed at NGA

9. SIPC Benefits In addition to HDF5 benefits
UNCLASSIFIED
SIPC Benefits
In addition to HDF5 benefits
Data is in ECEF 3D coordinate system
Data is tiled for rapid searching and partitioning
Can be tiled in some spatial acceleration structure, such as octree or quadtree
Hierarchical structure of signal data
Additional data per point can be sparse and vary in bit depth

10. SIPC Example Example SIPC file in HDFView Metadata and data fabricated
UNCLASSIFIED
SIPC Example
Example SIPC file in HDFView
Metadata and data fabricated

11. UNCLASSIFIED
Sample Code, Write

12. UNCLASSIFIED
Sample Code, Read

13. UNCLASSIFIED
Open Discussion