1 Earth System Grid Center for Enabling Technologies OPeNDAP Services for ESG March 9, 2016 Peter Fox, Patrick West, Stephan Zednik RPI Performance Measures x.x, x.x, and x.x

2 Earth System Grid Center for Enabling Technologies: (ESG- CET) In ESG II (in regard to data)  Server side aggregation  DAP object transfer via HTTP and GridFTP  GSIFTP integration, myProxy support  OPeNDAP-g -> led to architecture of OPeNDAP/ Hyrax (primarily BES is from OPeNDAP-g)

3 Earth System Grid Center for Enabling Technologies: (ESG- CET) Requirements leading to OPeNDAP-g  Separation of the core Data Access Protocol (DAP) from the transport protocol (HTTP).  High Performance Computing. The previous CGI based servers did not have the capacity required by ESG. Error and memory handling added.  Security. Once the OPeNDAP was independent of the transport protocol, adding security was possible by relying on the Globus gsiFTP system.  Aggregation. OPeNDAP 3.0 did not operate on aggregated datasets. OPeNDAP-g does.  Transport protocol independence and HPC were incorporated back into OPeNDAP leading to the current version. Security and aggregation remain an ESG only feature.

4 Earth System Grid Center for Enabling Technologies: (ESG- CET) OPeNDAP-g Architecture DispatcherBES  OPeNDAP-g Dispatcher (e.g. ESG Front-end Server)  Receives requests and asks the BES to fill them  Uses Apache Modules  Does not directly ‘touch’ data, handles URLs Data  Back End Server (BES)  Reads data files, Databases, et c., returns info  May return DAP objects or other data (netCDF)

5 5 OPeNDAP Hyrax Architecture OLFSBES  OPeNDAP Lightweight Front end Server (OLFS)  Receives requests and asks the BES to fill them  Uses Java Servlets  Does not directly ‘touch’ data  Multi-protocol Data  Back End Server (BES)  Reads data files, Databases, et c., returns info  May return DAP2 objects or other data  Does not require web server Client

6 6 GridFTP DAP2 GridFTP DAP2 HTTP DAP2 HTTP DAP2 ASCII output HTML form Info output OPeNDAP Lightweight Front end Server THREDDS** Request Formulation** Request from client Response to client BES SOAP-DAP (HTTP) DAP2 (GridFTP, HTTP) RDF, OWL, JSON (HTTP) PML output

7 7 Hyrax/ Back-end Server Network Protocol and Process start/stop activities Data Store Interfaces e.g. IOSP BES Framework PPT/PPTS* Initialization/ Termination DAP2 Access NetCDF3HDF4RDF/ SPARQL … Provenance Commands** e.g. server-side, Ferrett BES Commands/ XML Documents *PPT is built in (other protocols) **Some commands are built in Data Catalogs

8 Earth System Grid Center for Enabling Technologies: (ESG- CET) OPeNDAP-g services for ESG  Data access via Dataportals. In this Use Case users interact with the portal, browse the catalogs and decide what data to download. The portal passes the request to OPeNDAP-g which executes it and returns the data to the portal. The portal returns a URL to the user to download the data.  Data access via netCDF library. In this Use Case users link their applications with OPeNDAP-g client library for netCDF. The user can “open” a URL that refers to ESG data. The library fetches the data for the user and makes it a local netCDF file  The performance requirements for ESG II were met. However, ESG-CET scales up these requirements.

9 Earth System Grid Center for Enabling Technologies: (ESG- CET) Status of the Community OPeNDAP Server Software  Together the OLFS and the BES are known as Hyrax  Hyrax 1.6 provides support for NcML-based aggregation  Data response streamed back as netCDF file  RDF response type  Updated DDX response type (Data Definition XML)  Beginning development of DataDDX – multi-mime response with data and DDX  Full security audit and static code analysis certification to comply with NOAA and NASA requirements

10 Earth System Grid Center for Enabling Technologies: (ESG- CET) ESG-CET and data  Large data sets, numbers and sizes High performance Flexible architecture, both client and several types and numbers of servers Aggregation Server side operations Multiple transport protocol options  Full ESG security support as well as loose federation  Read-only client access via API (netCDF/CDM) To satisfy the new goals, the OPeNDAP services for ESG have been re-architected. We now use parts of the standard OPeNDAP framework Hyrax, focusing on high performance for the client side and extended flexibility.

11 ESG-CET and Products (server side functions)  Goal: drop in replacement for the TDS part of FTDS in LAS  Requires netCDF-Java Input-Output Service Provider (IOSP) adapter for Hyrax/BES  Use case examination will be required Earth System Grid Center for Enabling Technologies: (ESG- CET)

12 Security Infrastructure status  OPeNDAP BES Security SSL authentication between gridFTP middle tier and the BES. No persistent SSL connections are maintained  RNI Integration with ESG Security Infrastructure RNI client supports gsiFTP connections to ESG GridFTP servers ESG GridFTP server handles authentication of user Neill's ESG GridFTp authz callout plugin handles authorization of data request ESG/RNI GridFTP DSI module handles data request, forwards request to ESG BES server running RNI module Earth System Grid Center for Enabling Technologies: (ESG- CET)

13 Client status  RNI Version 0.1 implemented using netCDF version 3 and OPeNDAP’s libnc-dap  In communication with Unidata regarding integration of RNI client with new NetCDF version 4  Developing ncml aggregation in both client and server RNI  Full ESG security support Earth System Grid Center for Enabling Technologies: (ESG- CET)

14 Earth System Grid Center for Enabling Technologies: (ESG- CET) The Remote NetCDF Invocation (RNI)  The client is the netCDF library. It has exactly the same API as the standard C library netCDF, but it can deal with local files or files reachable via HTTP, PPT or gridFTP.  The third tier, the BES server can be reached only via PPT. NetCDF services for all NetCDF calls are implemented a a BES module.  The middle tier, acts like a proxy between the RNI client and server and deals with security.

15 Earth System Grid Center for Enabling Technologies: (ESG- CET) RNI Architecture CLIENT DATA GridFTP OPeNDAP BES NetCDF Library RNI Module connection acts like RNI Library

16 Earth System Grid Center for Enabling Technologies: (ESG- CET) Characteristics of the RNI as part of a data access system  Full Support of standard OPeNDAP URLs. RNI is being developed with the integrated Unidata/OPeNDAP netCDF library (and CDM)  Transparent access to either standard netCDF files and aggregated datasets via the NetCDF Markup Language (NCML).  For remote containers, all write operations are disable for security. That is, for HTTP/HTTPS, PPT and gridFTP/gsiFTP the RNI system is a read only API.  RNI utilizes Just in Time access. Caching is only for metadata. No pre-fetching of data.  RNI transparently accesses secure (gsiFTP, HTTPS) or insecure (gridFTP, HTTP) remote data.

17 Earth System Grid Center for Enabling Technologies: (ESG- CET) Status of the RPI work (OPeNDAP)  The primary accomplishments for this subproject in the past year has been The complete request-response for all netCDF API calls has been completed, that is, the two ends have been developed. We have highly optimized core components of the standard OPeNDAP framework to support the performance goals. We have established the knowledge of how the middle tier (the proxy) will be incorporated into the complete system. Our work was presented in AGU fall 2007 and EGU  So far all of the goals established for the first stage of the project has been completed.  Next stage integration with product server for gateway and data node data access Functional (with enhancements) replacement for TDS

18 Future, will not elaborate  Storage Resource Manager / DMLite as a client  Return as RDF  Return as PML (Provenance – Proof Markup Language) Earth System Grid Center for Enabling Technologies: (ESG- CET)