1. Sun Earth Connection Distributed Data Services Presented at the Principle Investigator's Meeting NASA's Applied Information Systems Research Program 5 April 2005 Peter Cornillon OPeNDAP and Peter Fox HAO/UCAR

2. Project Objective  To integrate a number of current IT efforts into a suite of data services designed to facilitate the access to and use of data for Sun Earth Connection studies. The collection of services to be developed is based on the OPeNDAP Data Access Protocol (DAP), a protocol in heavy use in the Earth sciences.

3. Presentation Objective  To introduce OPeNDAP to the AISRP community.  To summarize project plans

4. Outline  Evolution of data systems  OPeNDAP: History  OPeNDAP: Current status and AISRP plans

5. Data systems generally involve some combination of the following elements Access/Delivery Archive Discovery Analysis/Visualization

6. These elements have been developed and managed as monolithic systems by the groups assembling the elements. In the Past

7. toward system elements  The integration of independently designed, implemented and maintained system elements. Trend in Data System Development systems  Is away from centrally designed, implemented and maintained systems

8. With a Plethora of System Elements Archive Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Archive Archive Discovery Discovery Archive Access/Delivery Access/Delivery

9. Archive Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Archive Archive Discovery Discovery Archive Access/Delivery Access/Delivery Access/Delivery The OPeNDAP Data Access Protocol Is an access/delivery element.

10.  The OPeNDAP philosophy focuses on the development of a highly modular system based on the OPeNDAP data access protocol (DAP). OPeNDAP Philosophy  The DAP allows system elements to talk to one another; it’s the glue between system elements. – The DAP

11. Some History

12. Distributed Oceanographic Data System (DODS)  Conceived in 1992 at a workshop held at URI.  Objectives were: –to facilitate access to PI held data as well as data held in national archives and –to allow the data user to analyze data using the application package with which he or she is the most familiar.  Basic system designed and implemented in 1993-1994 by Gallagher and Flierl with NASA and NOAA funding.  From 1994 to present it has been extended with AISRP, NSF, NOPP and other NASA funding.

13. Distributed Oceanographic Data System DODS consisted of two fundamental parts:  a discipline independent core infrastructure for moving data on the net,  a discipline specific portion related to data – population, location, specialized clients, etc.

14. DODS  OPeNDAP & NVODS To isolate the discipline independent part of the system from the discipline specific part, two entities have been formed:  The Open Source Project for a Network Data Access Protocol (OPeNDAP)  The National Virtual Ocean Data System (NVODS)

15. OPeNDAP  Formed to maintain, evolve and promote the discipline neutral Data Access Protocol that was the DODS core infrastructure  OPeNDAP is a 501 c(3) non-profit corporation

16. Considerations with regard to the development OPeNDAP  Many data providers  Many data formats  Many different semantic representations of the data  Many different client types

17.  The DAP has been designed to be as general as possible without being constrained to a particular discipline or world view. The Data Access Protocol (DAP)  The DAP is a discipline neutral data access protocol; it can be used in astronomy, medicine, earth science,…

18. Current Status and AISRP Plans

19. The OPeNDAP data access protocol is used by a variety of system elements.  Servers  Clients  Aggregation Servers  Processing Servers  Ancillary Information Services  Data System Integrators (ODC)  Servers  Clients  Aggregation Servers  Processing Servers  Ancillary Information Services  Data System Integrators (ODC)  Servers  Clients  Aggregation Servers  Processing Servers  Ancillary Information Services  Data System Integrators (ODC)

20. Servers  Servers receive requests and provide responses via the DAP.  Servers convert the data from the form in which they are stored to the OPeNDAP data model.  Servers provide for subsetting of the data.

21. Data SPDML Data netCDF Data HDF4 Data Matlab Data DSP Data JGOFS Tables Data JDBC SQL Data FITS Data CDF Data FreeFrom Flat Binary Data CEDAR Data ESML General Data SPDML OPeNDAP Servers AISRP

22. OPeNDAP/NVODS Server Sites OPeNDAP Server Sites

23. Servers  Servers may also provide other services  Directory traversal.  Browser-based form to build URL.  ASCII version of data.  Metadata associated with the data.

24. OPeNDAP System elements  Servers  Clients  Aggregation Servers  Processing Servers  Ancillary Information Services  Data System Integrators (ODC)

25. Clients  Clients make requests and receive responses via the DAP.  Clients convert data from the OPeNDAP data model to the form required in the client application.

26. IDL Client netCDF C FerretGrADS netCDF Java IDVVisADncBrowse Matlab Client AccessExcel IDL Client OPeNDAP Clients AISRP

27. OPeNDAP System elements  Servers  Clients  Aggregation Servers  Processing Servers  Ancillary Information Services  Data System Integrators (ODC)

28. Archive Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Archive Archive Discovery Discovery Archive Access/Delivery Access/Delivery Data System Integrators  In an environment consisting of highly distributed system elements, the notion of a data system is amorphous at best.  The data system integrator is an element which assembles a suite of data system elements that together provide seamless access to the data from discovery to use.

29. Data System Integrators  The data system integrator defines the data system.  The data system integrator generally ‘speaks’ several different protocols.

30. ArchiveArchiveArchiveArchive Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Data System Integrators The data system integrator brings order to a disordered array of system elements. A rc hi v e Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Application/Visualization Data GCM D NVOD S ODC

31. IDL GrADS Ferret IDV VisAD ncBrowse Matlab Excel Access OPeNDAP GFDL netCDF GSFC Binary URI HDF GCMD NVOD S ODC IDL GrADS Ferret IDV VisAD ncBrowse Matlab Excel Access OPeNDAP GFDL netCDF GSFC Binary URI HDF GCMD NVOD S The ODC - a Data System Integrator AISRP GrADS Ferret IDV VisAD ncBrowse Matlab Excel Access OPeNDAP JHU CDF GSFC SPDML HAO CEDAR GSFC HAO IDL

32. http://opendap.org http://opendap.org Is anyone going to SF airport tonight? http://opendap.org