OPeNDAP in the Cloud OPeNDAP James Gallagher, Nathan Potter and NOAA/NODC Deirdre Byrne, Jefferson Ogata, John Relph 26 June 2013

Not only is the OPeNDAP request smaller, it's also just the data the person wants Full dataset OPeNDAP request 4% Download100% Download

Not only is the OPeNDAP request smaller, it's also just the data the person wants Full dataset OPeNDAP request 4% Download100% Download

NOAA Environmental Data Management Conceptual Cloud Architecture* Potential locations of cloud-enabled OPeNDAP instances *Aadapted from NOAA Environmental Data Management Framework Draft v0.3 Appendix C - Dr. Jeff de La Beaujardière, NOAA Data Management Architect

No vendor lock-in! No Stovepipes! - flexible storage method (what is the client of 2020? we don't know!) Hierarchical/human browsable Constraints file dataset file

No vendor lock-in! No Stovepipes! - flexible storage method (what is the client of 2020? we don't know!) Hierarchical/human browsable Constraints file dataset file

Solution: Data Store Catalog Data S3 HTTP GET & HEAD requests

Solution: Data Store Catalog Data S3 HTTP GET & HEAD requests

Solution: Data Store Catalog Data S3 HTTP GET & HEAD requests

Solution: Data Store Catalog Data S3 HTTP GET & HEAD requests

Web requests S3 Catalog, or data request XML or data file

To enhance performance, data were accessed from S3 only when not already cached. OPeNDAP Catalog requests S3 OPeNDAP Server catalog cache XML File User catalog Request Catalog Access THREDDS catalog or HTML EC2 data cache

To enhance performance, data were accessed from S3 only when not already cached. OPeNDAP Data requests S3 OPeNDAP Server catalog cache Data File User data Request Data Access Data Slice EC2 data cache

To enhance performance, data were accessed from S3 only when not already cached. OPeNDAP Data requests S3 OPeNDAP Server catalog cache Data File User data Request Data Access Data Slice EC2 data cache

Observations S3FS & Amazon's APIs: vendor lock-in

Observations S3FS & Amazon's APIs: vendor lock-in XML catalogs were flexible

Observations S3FS & Amazon's APIs: vendor lock-in XML catalogs were flexible Catalogs didn't need to be stored in S3

Observations S3FS & Amazon's APIs: vendor lock-in XML catalogs were flexible Catalogs didn't need to be stored in S3 Glacier: Places additional demands on clients.

Summary OPeNDAP server with minimal changes Served data stored in S3

Summary OPeNDAP server with minimal changes Served data stored in S3 Data accessible to other systems Solution widely applicable