1. Status for Endeavor 6: Improved Scientific Data Access Infrastructure
Russ Rew
August 25, 2004

2. Overview Endeavor 6 summary Importance Status Resources
Prospects for success
Conclusions

3. More than just netCDF-4 project …
Endeavor 6 Summary
More than just netCDF-4 project …
Continued netCDF-3 evolution and support: large file support, documentation, Windows support, HTTP access, NcML support, complete examples
Development of netCDF-4 over HDF5: multiple unlimited dimensions, parallel I/O on multiprocessors, packing
Development of advanced infrastructure: Java interface to netCDF, common data model, metadata representations, data server technologies, udunits, standardization efforts

4. Importance of Endeavor 6
Infrastructure emphasized by CYRDAS report, Atkins report, frameworks, community projects, …
NetCDF-3 is most downloaded, most used Unidata software by an order of magnitude: average 420 downloads per week for last year (of complete netCDF-3 source by distinct hosts)
Latest Java netCDF source release downloaded by 846 distinct hosts
NetCDF generates significant support load: 18 responses per week
NetCDF-3 will see continued use after netCDF-4 release: archives, tools, projects (e.g. pnetcdf, SDO), and conventions
… but endeavor 6 overlooked in Unidata Strategic Plan

5. Status of Endeavor 6
NetCDF first major netCDF-3 release since 3.5 in March 2001, provides large file support
NetCDF-4 currently a month behind schedule, aiming for August 2005 release
NCSA started late on HDF5 enhancements
Java netCDF 2.2 will be a major upgrade with support for common data model (netCDF-3, netCDF-4, HDF5,OPeNDAP, GRIB, ADDE)
Pressure for parallel I/O eased by pnetcdf release
High interest in netCDF-4, udunits improvements, Java netCDF, common data model, server technologies, standardization

6. NetCDF-3 Enhancements for NetCDF-4
To provide
stable foundation for incorporating netCDF-4
smooth transition for current users
Automated multi-platform testing
Documentation converted to maintainable form, new language-independent Users Guide
Added large file support with backward compatibility
Added default format interfaces
Better Windows and .Net support

7. Current NetCDF-4 Technical Status
Implement netCDF-3 over HDF5, to demonstrate backward compatibility with API and format
done
Determine needed HDF5 enhancements
Prepare netCDF-3 for incorporation with netCDF-4
nearly done
Design netCDF-4 interface to add enhancements made possible with HDF5
in progress
Implement needed HDF5 enhancements
Implement netCDF-4 over enhanced HDF5
not started yet

8. Resources for Endeavor 6
NetCDF-3 development neglected from 2000 to 2003, still catching up
Current netCDF effort about 2.0 FTEs
NASA funded development, not deployment or support
Supporting both netCDF-3 and netCDF-4 will require additional resources

9. NetCDF-4 Project Schedule
Currently behind schedule for a July 2005 release
July 2004: version revised documentation, 64-bit file offsets, default format functions
October 2004: version use of autotools
January 2005: version 3.7.1: netCDF-4 prototype included, support for multiple unlimited dimensions
March 2005: version 4.0.0_beta - test relelase
July 2005: version first netCDF-4 production release

10. NetCDF-4: Remaining Tasks
Next for netCDF-4: interface additions for multiple unlimited dimensions, group interfaces, dynamic schema modification, new data types, packed data, parallel I/O, compression
HDF5 enhancements
zero-length attributes
shared dimensions
creation order access for objects
Testing in models (CCSM, WRF, ESMF, ...)

11. Are Resources Adequate for Success?
Large File Support in is necessarily complex, will generate more user support
NetCDF will really by a 1.0 release, can anticipate problems and initial high support requirements
We lack expertise in parallel I/O, multiprocessors, I/O tuning for large models, HPC requirements
Server technologies like HTTP access, NcML needed for C-based interfaces
Current 2.0 FTEs feels marginal, inadequate for needs

12. Conclusions
Resources currently allocated to endeavor 6 do not reflect its importance
Many beneficiaries of endeavor 6 outside Unidata community: NOAA, NASA, DOE, DOD, ARM, USGS, ESRI, …
Many outside developers have contributed: 200 users and developers credited in netCDF credits
Common data model a good infrastructure investment
We need to find ways to leverage open source communities and non-Unidata beneficiaries to provide more resources for development and support