Development of a CF Conventions API Russ Rew GO-ESSP Workshop, LLNL

2 CFlib Goals l Goals of a C API for CF conventions: u Support the creation and access of datasets conforming to CF Conventions u Provide a higher-level interface than netCDF for coordinate systems to locate data in space and time and to identify and compare physical quantities u Reduce programming effort needed to create datasets that conform to CF conventions or to correctly interpret CF-conforming data u Preserve existing intellectual rigor of CF u Anticipate likely changes and additions to CF

3 Current CF API Efforts l What exists now? u Some netCDF Java capabilities for reading CF conventions to create coordinate-system objects u Climate Model Output Rewriter (CMOR) in Fortran 90 to convert netCDF data to CF conventions for PCMDI archives u Other frameworks with IO library interfaces for CF? u Jonathan Gregory’s draft specification for a CF processing library u Ed Hartnett’s early draft of some CF API requirements and early prototype of some CF APIs for ESMF

4 Some CFlib Issues l Is it the right time to develop a CF API? u Is CF 1.0 mature and stable enough for API standardization? u Is the current conventions document an adequate specification? u Would it be better to delay until agreement is reached on the next CF version? l Have sufficient resources been identified to develop and maintain CFlib? u Some support available from ESMF (0.25 FTE for near future) u Should Unidata seek resources for CFlib? u Would an Open Source community development effort be more desirable or practical?

More CFlib Issues l Sociological issues: u What is relationship of CF governance to CFlib? u How would decisions be made for an Open Source effort to develop CFlib? l Technical issues: u Should CFlib API be designed to be independent of netCDF and udunits? u What should be the relationship of CFlib to Unidata’s Common Data Model or to netCDF-4? u Is a Fortran-77 interface needed for CFlib? u How important is preserving complete COARDS convention compatibility?

Commitment to Backward Compatibility l CF conventions must preserve access to earlier forms of CF-compliant data. A CF conventions API must respect the version specified by the Conventions attribute. l Future changes to CF should be based on experience with working reference implementations. Because preserving access to archived data for future generations is sacrosanct

NetCDF-4 Features Affecting Conventions l Multiple unlimited dimensions l Compound types (structs) l Variable-length types for ragged arrays l String type l Unsigned numeric types l Unicode names l Uses for Groups

8 Guidelines for Quality Standards l Some lessons from “The Rise and Fall of CORBA” (Henning, 2006) u Standards consortia need iron-clad rules to ensure that they standardize existing best practice (don't innovate in standards!) u No standard should be approved without a reference implementation. u No standard should be approved without having been used to implement a few projects of realistic complexity. u To create quality software [or standards], the ability to say "no" is usually far more important than the ability to say "yes."

9 … the ephemeral nature of both data formats and storage media threatens our very ability to maintain scientific, legal, and cultural continuity, not on the scale of centuries, but considering the unrelenting pace of technological change, from one decade to the next. … And that's true not just for the obvious items like images, documents, and audio files, but also for scientific images, … and simulations. In the scientific research community, standards are emerging here and there—HDF (Hierarchical Data Format), NetCDF (network Common Data Form), FITS (Flexible Image Transport System)—but much work remains to be done to define a common cyberinfrastructure. “Eternal Bits: How can we preserve digital files and save our collective memory?,” MacKenzie Smith, IEEE Spectrum, July 2005 MacKenzie Smith, Associate Director for Technology at the MIT Libraries, Project director at MIT for DSpace, a groundbreaking digital repository system Data is Part of Our Legacy

10 Questions?