1. Profile of NPOESS HDF5 Files
Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews
NPOESS Program
Raytheon Company
Aurora, Colorado

2. NPOESS HDF5 General Overview
Ron Andrews

3. HDF5 Conceptual Diagram

4. HDF5 XML User’s Block
The XML User’s Block for NPOESS Data Products provides a ‘quick-look’ into the metadata of the associated HDF5 file.
The XML User’s Blocks are defined in the following volumes:
CDFCB-X Volume V – contains the XML User’s Block formats for the RDRs, SDRs, TDRs, EDRs, ARPs, and IPs
CDFCB-X Volume VI – Ancillary, Auxiliary, Reports, and Messages – contains the XML User’s Block formats for the ancillary, auxiliary, reports, and messages that are wrapped in HDF5
The size of the HDF5 XML User’s Block will be a factor of 512 bytes, with a maximum size which is still to be determined
Provides elements:
N_Processing_Domain
Mission, Platform, and Instrument Names
N_Dataset_Type_Tag
Number_of_Data_Products
CollectionShortName(s)
Aggregation Information
Timestamps (Creation Timestamp, Observation Timestamps)
Percent Missing Data

5. General HDF5 File Structure

6. HDF5 File Root Group Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
Calendar_Date
H_STR X
Identification_Information.Dataset_Identifier
Instrument_Short_Name
Mission_Name
N_Creation_Date
N_Creation_Time
N_Dataset_Source
N_Geo_Ref
N_NPOESS_Document_Ref
N_Orbit_Position_Predicted (NPOESS only)
N_Subsystem_SW_Version
N_Update_Date
N_Update_Time
Platform_Short_Name
Time_Of_Day

7. HDF5 File Product Group Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
Band_ID
H_STR X
Band_Measurement_Mode_ID
N_Anc_Type_Tasked
N_Collection_Short_Name
N_Dataset_Type_Tag
N_Instrument_Flight_SW_Version
H_I
N_Packet_Type
N_Packet_Type_Count
N_Processing_Domain
N_Spacecraft_SW_Version
Operational_Mode
Processing_Level_Identifier

8. HDF5 File Product Aggregation Dataset Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
AggregateBeginningDate
H_STR X
AggregateBeginningGranuleID
AggregateBeginningOrbitNumber
H_ILLU
AggregateBeginningTime
AggregateEndingDate
AggregateEndingGranuleID
AggregateEndingOrbitNumber
AggregateEndingTime
AggregateG-RingLatitude
H_F Array
AggregateG-RingLongitude
AggregateNumberGranules
H_F
AggregatePercentMissingDataAve

9. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
Ascending/Descending_Indicator
H_C X
Beginning_Date
H_STR
Beginning_Time
Cloud_Cover
H_F
East_Bounding_Coordinate
Ending_Date
Ending_Time
G-Ring_Latitude
H_F Array
G-Ring_Longitude
Input_Dataset_Identifier.Dataset_Identifier
N_Algorithm_Version
N_Anc_Filename
N_Anc_Type
N_Aux_Filename
N_Beginning_Orbit_Number
H_ILLU
N_Beginning_Time_IET
H_ILL

10. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
N_Columns
H_IL X
N_Day_Night_Flag
H_STR
N_Deliverable_IP_Filename
N_Ending_Time_IET
H_ILL
N_Graceful_Degradation
N_Granule_ID
N_Granule_Quality_Value
H_F
N_Granule_Version
N_LEOA_Flag
N_Levels
N_LUT_Version
N_Nadir_Latitude_Max
N_Nadir_Latitude_Min
N_Nadir_Longitude_Max
N_Nadir_Longitude_Min
N_Percent_Erroneous_Data

11. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
N_Percent_Missing_Data
H_F X
N_Percent_Not-Applicable_Data
N_Recent_Maneuver
N_Reference_ID
H_STR
N_Rows
N_Satellite/Local_Azimuth_Angle_Max
H_IL
N_Satellite/Local_Azimuth_Angle_Min
N_Satellite/Local_Zenith_Angle_Max
N_Satellite/Local_Zenith_Angle_Min
N_Solar_Azimuth_Angle_Max
N_Solar_Azimuth_Angle_Min
N_Solar_Zenith_Angle_Max
N_Solar_Zenith_Angle_Min
N_Spacecraft_Manuever
N_Spacecraft_SW_Version
N_Tile_ID

12. HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
North_Bounding_Coordinate
H_I X
Processing_Level_Identifier
H_F
South_Bounding_Coordinate
H_STR
West_Bounding_Coordinate

13. HDF5 File Spacecraft Diary Group Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
N_Collection_Short_Name
H_STR X
N_Dataset_Type_Tag
N_Packet_Type
N_Packet_Type_Count
Processing_Level_Identifier

14. HDF5 File Spacecraft Diary Aggregation Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
AggregateBeginningDate
H_STR X
AggregateBeginningTime
AggregateEndingDate
AggregateEndingTime

15. HDF5 File Spacecraft Diary Dataset Metadata Elements – Example CDFCB-X Table
Field Name
Data Type
RDR
SDR/TDR
IP/ARP/EDR
GEO
Gridded IP
ANC
AUX
Input_Dataset_Identifier.Dataset_Identifier
H_StR X
N_Beginning_Time_IET
H_ILL
N_Ending_Time_IET
N_Support_Data_Version
H_STR

16. HDF5 Conceptual Diagram - Data

17. NPOESS Quality Flags Overview
The concept is to provide for consistently stored, high density, quality information about the delivered data – simplifying usability while maintaining storage efficiency
Quality flags are qualifications of one or more consecutive bits in each byte.
Quality flag arrays follow the structure of the data product
The size of the arrays are equal to or less than the size of the data to which the quality information applies (dimensions retain correspondence to data product arrays)
Quality flags will appear in the HDF5 files as n number(s) of two or three dimensional, 1-byte arrays.
The number of arrays is dependant on the number and size of each data product's quality flag definitions
Quality flags will be packed such that each flag is entirely contained within a single byte, occasionally resulting in a byte with reserved or meaningless bits
Byte alignment is the same for every quality flag array
First bit is the LSB, bytes are Big-Endian
Each byte may contain multiple bit-level flags

18. 2-Dimensional Array Example

19. 3-Dimensional Array Example

20. Detailed NPOESS UML Models
Ken Stone

21. RDR UML Model

22. Common RDR Layout

23. SDR/TDR UML Model

24. EDR UML Model

25. Ancillary/Auxiliary UML Models

26. NPOESS Sample Data
Patrick Purcell

27. Product Example UML Model

28. HDF5 View - ImageView

29. Product Profile Example

30. HDF5 API – C++ Reader Sample Code
Note: A granule holds a reference to a dataset region ...
... while an aggregation contains a reference to an object.

31. NPOESS HDF5 Summary
The NPOESS Program delivers the official deliverable data products (RDR, SDR/TDR, EDR/ARP/IP) and dynamic ancillary data and auxiliary data in HDF5 Files
The HDF5 Files have an XML User Block that can be accessed without HDF5 tools - provides a “quick-look” into the metadata before opening the HDF5 file
Metadata within the HDF5 files are stored as attributes
There are general UML Models for the NPOESS official delivered data that provide a common framework
Official deliverable data products are organized by reference objects (aggregations) which contain one or more reference regions (granules)
Although data may be accessed directly through the All_Data group, the Data_Products group provides integrated access:
Allows the user to access both metadata and data through a common HDF5 group
Metadata is accessed directly by reading the Attribute values
Datasets may be accessed by dereferencing the object ID stored in the Data_Products Group for the aggregation or granule