Status for Endeavor 6: Improved Scientific Data Access Infrastructure Russ Rew August 25, 2004

Overview Endeavor 6 summary Importance Status Resources Prospects for success Conclusions

More than just netCDF-4 project … Endeavor 6 Summary More than just netCDF-4 project … Continued netCDF-3 evolution and support: large file support, documentation, Windows support, HTTP access, NcML support, complete examples Development of netCDF-4 over HDF5: multiple unlimited dimensions, parallel I/O on multiprocessors, packing Development of advanced infrastructure: Java interface to netCDF, common data model, metadata representations, data server technologies, udunits, standardization efforts

Importance of Endeavor 6 Infrastructure emphasized by CYRDAS report, Atkins report, frameworks, community projects, … NetCDF-3 is most downloaded, most used Unidata software by an order of magnitude: average 420 downloads per week for last year (of complete netCDF-3 source by distinct hosts) Latest Java netCDF source release downloaded by 846 distinct hosts NetCDF generates significant support load: 18 responses per week NetCDF-3 will see continued use after netCDF-4 release: archives, tools, projects (e.g. pnetcdf, SDO), and conventions … but endeavor 6 overlooked in Unidata Strategic Plan

Status of Endeavor 6 NetCDF 3.6.0 first major netCDF-3 release since 3.5 in March 2001, provides large file support NetCDF-4 currently a month behind schedule, aiming for August 2005 release NCSA started late on HDF5 enhancements Java netCDF 2.2 will be a major upgrade with support for common data model (netCDF-3, netCDF-4, HDF5,OPeNDAP, GRIB, ADDE) Pressure for parallel I/O eased by pnetcdf release High interest in netCDF-4, udunits improvements, Java netCDF, common data model, server technologies, standardization

NetCDF-3 Enhancements for NetCDF-4 To provide stable foundation for incorporating netCDF-4 smooth transition for current users Automated multi-platform testing Documentation converted to maintainable form, new language-independent Users Guide Added large file support with backward compatibility Added default format interfaces Better Windows and .Net support

Current NetCDF-4 Technical Status Implement netCDF-3 over HDF5, to demonstrate backward compatibility with API and format done Determine needed HDF5 enhancements Prepare netCDF-3 for incorporation with netCDF-4 nearly done Design netCDF-4 interface to add enhancements made possible with HDF5 in progress Implement needed HDF5 enhancements Implement netCDF-4 over enhanced HDF5 not started yet

Resources for Endeavor 6 NetCDF-3 development neglected from 2000 to 2003, still catching up Current netCDF effort about 2.0 FTEs NASA funded development, not deployment or support Supporting both netCDF-3 and netCDF-4 will require additional resources

NetCDF-4 Project Schedule Currently behind schedule for a July 2005 release July 2004: version 3.6.0 - revised documentation, 64-bit file offsets, default format functions October 2004: version 3.7.0 - use of autotools January 2005: version 3.7.1: netCDF-4 prototype included, support for multiple unlimited dimensions March 2005: version 4.0.0_beta - test relelase July 2005: version 4.0.0 - first netCDF-4 production release

NetCDF-4: Remaining Tasks Next for netCDF-4: interface additions for multiple unlimited dimensions, group interfaces, dynamic schema modification, new data types, packed data, parallel I/O, compression HDF5 enhancements zero-length attributes shared dimensions creation order access for objects Testing in models (CCSM, WRF, ESMF, ...)

Are Resources Adequate for Success? Large File Support in 3.6.0 is necessarily complex, will generate more user support NetCDF-4.0.0 will really by a 1.0 release, can anticipate problems and initial high support requirements We lack expertise in parallel I/O, multiprocessors, I/O tuning for large models, HPC requirements Server technologies like HTTP access, NcML needed for C-based interfaces Current 2.0 FTEs feels marginal, inadequate for needs

Conclusions Resources currently allocated to endeavor 6 do not reflect its importance Many beneficiaries of endeavor 6 outside Unidata community: NOAA, NASA, DOE, DOD, ARM, USGS, ESRI, … Many outside developers have contributed: 200 users and developers credited in netCDF credits Common data model a good infrastructure investment We need to find ways to leverage open source communities and non-Unidata beneficiaries to provide more resources for development and support