Profile of NPOESS HDF5 Files Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews NPOESS Program Raytheon Company Aurora, Colorado

NPOESS HDF5 General Overview Ron Andrews

HDF5 Conceptual Diagram

HDF5 XML User’s Block The XML User’s Block for NPOESS Data Products provides a ‘quick-look’ into the metadata of the associated HDF5 file. The XML User’s Blocks are defined in the following volumes: CDFCB-X Volume V – contains the XML User’s Block formats for the RDRs, SDRs, TDRs, EDRs, ARPs, and IPs CDFCB-X Volume VI – Ancillary, Auxiliary, Reports, and Messages – contains the XML User’s Block formats for the ancillary, auxiliary, reports, and messages that are wrapped in HDF5 The size of the HDF5 XML User’s Block will be a factor of 512 bytes, with a maximum size which is still to be determined Provides elements: N_Processing_Domain Mission, Platform, and Instrument Names N_Dataset_Type_Tag Number_of_Data_Products CollectionShortName(s) Aggregation Information Timestamps (Creation Timestamp, Observation Timestamps) Percent Missing Data

General HDF5 File Structure

HDF5 File Root Group Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX Calendar_Date H_STR X Identification_Information.Dataset_Identifier Instrument_Short_Name Mission_Name N_Creation_Date N_Creation_Time N_Dataset_Source N_Geo_Ref N_NPOESS_Document_Ref N_Orbit_Position_Predicted (NPOESS only) N_Subsystem_SW_Version N_Update_Date N_Update_Time Platform_Short_Name Time_Of_Day

HDF5 File Product Group Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX Band_ID H_STR X Band_Measurement_Mode_ID N_Anc_Type_Tasked N_Collection_Short_Name N_Dataset_Type_Tag N_Instrument_Flight_SW_Version H_I N_Packet_Type N_Packet_Type_Count N_Processing_Domain N_Spacecraft_SW_Version Operational_Mode Processing_Level_Identifier

HDF5 File Product Aggregation Dataset Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX AggregateBeginningDate H_STR X AggregateBeginningGranuleID AggregateBeginningOrbitNumber H_ILLU AggregateBeginningTime AggregateEndingDate AggregateEndingGranuleID AggregateEndingOrbitNumber AggregateEndingTime AggregateG-RingLatitude H_F Array AggregateG-RingLongitude AggregateNumberGranules H_F AggregatePercentMissingDataAve

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX Ascending/Descending_Indicator H_C X Beginning_Date H_STR Beginning_Time Cloud_Cover H_F East_Bounding_Coordinate Ending_Date Ending_Time G-Ring_Latitude H_F Array G-Ring_Longitude Input_Dataset_Identifier.Dataset_Identifier N_Algorithm_Version N_Anc_Filename N_Anc_Type N_Aux_Filename N_Beginning_Orbit_Number H_ILLU N_Beginning_Time_IET H_ILL

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX N_Columns H_IL X N_Day_Night_Flag H_STR N_Deliverable_IP_Filename N_Ending_Time_IET H_ILL N_Graceful_Degradation N_Granule_ID N_Granule_Quality_Value H_F N_Granule_Version N_LEOA_Flag N_Levels N_LUT_Version N_Nadir_Latitude_Max N_Nadir_Latitude_Min N_Nadir_Longitude_Max N_Nadir_Longitude_Min N_Percent_Erroneous_Data

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX N_Percent_Missing_Data H_F X N_Percent_Not-Applicable_Data N_Recent_Maneuver N_Reference_ID H_STR N_Rows N_Satellite/Local_Azimuth_Angle_Max H_IL N_Satellite/Local_Azimuth_Angle_Min N_Satellite/Local_Zenith_Angle_Max N_Satellite/Local_Zenith_Angle_Min N_Solar_Azimuth_Angle_Max N_Solar_Azimuth_Angle_Min N_Solar_Zenith_Angle_Max N_Solar_Zenith_Angle_Min N_Spacecraft_Manuever N_Spacecraft_SW_Version N_Tile_ID

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX North_Bounding_Coordinate H_I X Processing_Level_Identifier H_F South_Bounding_Coordinate H_STR West_Bounding_Coordinate

HDF5 File Spacecraft Diary Group Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX N_Collection_Short_Name H_STR X N_Dataset_Type_Tag N_Packet_Type N_Packet_Type_Count Processing_Level_Identifier

HDF5 File Spacecraft Diary Aggregation Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX AggregateBeginningDate H_STR X AggregateBeginningTime AggregateEndingDate AggregateEndingTime

HDF5 File Spacecraft Diary Dataset Metadata Elements – Example CDFCB-X Table Field Name Data Type RDR SDR/TDR IP/ARP/EDR GEO Gridded IP ANC AUX Input_Dataset_Identifier.Dataset_Identifier H_StR X N_Beginning_Time_IET H_ILL N_Ending_Time_IET N_Support_Data_Version H_STR

HDF5 Conceptual Diagram - Data

NPOESS Quality Flags Overview The concept is to provide for consistently stored, high density, quality information about the delivered data – simplifying usability while maintaining storage efficiency Quality flags are qualifications of one or more consecutive bits in each byte. Quality flag arrays follow the structure of the data product The size of the arrays are equal to or less than the size of the data to which the quality information applies (dimensions retain correspondence to data product arrays) Quality flags will appear in the HDF5 files as n number(s) of two or three dimensional, 1-byte arrays. The number of arrays is dependant on the number and size of each data product's quality flag definitions Quality flags will be packed such that each flag is entirely contained within a single byte, occasionally resulting in a byte with reserved or meaningless bits Byte alignment is the same for every quality flag array First bit is the LSB, bytes are Big-Endian Each byte may contain multiple bit-level flags

2-Dimensional Array Example

3-Dimensional Array Example

Detailed NPOESS UML Models Ken Stone

RDR UML Model

Common RDR Layout

SDR/TDR UML Model

EDR UML Model

Ancillary/Auxiliary UML Models

NPOESS Sample Data Patrick Purcell

Product Example UML Model

HDF5 View - ImageView

Product Profile Example

HDF5 API – C++ Reader Sample Code Note: A granule holds a reference to a dataset region ... ... while an aggregation contains a reference to an object.

NPOESS HDF5 Summary The NPOESS Program delivers the official deliverable data products (RDR, SDR/TDR, EDR/ARP/IP) and dynamic ancillary data and auxiliary data in HDF5 Files The HDF5 Files have an XML User Block that can be accessed without HDF5 tools - provides a “quick-look” into the metadata before opening the HDF5 file Metadata within the HDF5 files are stored as attributes There are general UML Models for the NPOESS official delivered data that provide a common framework Official deliverable data products are organized by reference objects (aggregations) which contain one or more reference regions (granules) Although data may be accessed directly through the All_Data group, the Data_Products group provides integrated access: Allows the user to access both metadata and data through a common HDF5 group Metadata is accessed directly by reading the Attribute values Datasets may be accessed by dereferencing the object ID stored in the Data_Products Group for the aggregation or granule