IPDA 15-16 July 2013 CDF and PDS Todd King, Joseph Mafi, Steven Joy
PDS (Planetary Data System) NASA's project charged with archiving planetary data. An active archive that makes available well documented, peer reviewed data to the research community. Establishes the requirements, specifications and guidelines for proper archiving. 9 April 2019 Information and Overview
PDS Archive Requirements PDS is required to preserve archived data for 200 years. Adopted standards are defined by recognized organizations. Data storage formats must be durable. Useful as-is. Data values are stored, no software based calculations Data structures must be described in a way that allows users to access the data in the absence of any format-specific readers. 9 April 2019 Information and Overview
CDF (Common Data Format) Self-describing format for the storage of scalar and multidimensional data in a platform and discipline independent way. Libraries and toolkits are available for the most commonly used platforms and programming languages. CDF specification is self-published and publicly available (currently version 3.4). Blends metadata (attributes) and data (variables) into one format. Additional specifications for metadata (CDF Tools, ISTP/IACG) 9 April 2019 Information and Overview
Information and Overview CDF Internals Magic Numbers CDF Descriptor Record (CDR) Global Descriptor Record (GDR) Attribute Descriptor Record (ADR) * Attribute zEntry Descriptor Record (AzVDR) zVariable Descriptor Record (zVDR) * Variable indeX Record (VXR) Binary Data Variable Values Record (VVR) 9 April 2019 Information and Overview
Information and Overview The Need to Support CDF Various NASA (Heliophysics Division) and international projects have chosen CDF as their official science data format. MAVEN BepiColombo and so have space agencies JAXA PDS prefers to archive projects’ official science products. Instrument/Science team use of archive data products best method of insuring data quality. 9 April 2019 Information and Overview
Archiving CDF in PDS4 … is possible Restrictions for CDF files: Single file CDF MSB byte order (Network Encoding) Contiguous variable storage (no frgamentation) All records and values are written (physical) Adhere to CDF Tools and ISTP/IACG metadata guidelines. No sparse records. No compression. CDF data will be described using PDS4 data structures. CDF attributes (metadata): Header objects. CDF variables (data): Array objects. 9 April 2019 Information and Overview
Information and Overview Archiving Support Copies of the CDF Specification will be included in the archive. Tools: CDF tools (Goddard) PDS4 metadata generation tool igpp.docgen with pds.cdf parser Format conversion tool CDF to ASCII table (in development) 9 April 2019 Information and Overview
Additional Information White Paper - Supporting CDF in PDS4, Todd King Java tools: igpp.docgen – Process Apache Velocity Templates pds.cdf – Pure Java CDF parser. Library, command-line tool, and embedded in igpp.docgen. 9 April 2019 Information and Overview
Information and Overview Recommendation Allow CDF which meet archive standards. Call it “CDF-A”? 9 April 2019 Information and Overview