LP DAAC OPeNDAP Services Jason Werpy LP DAAC Enterprise Architect SGT, Inc., Contractor to the USGS EROS Center *Work performed under USGS contract G15PD00766

OPeNDAP – The base of LP DAAC Data access Tiered Concept LP DAAC created a tiered approach to data access

A Quick Aside – HTTP access to everything OPeNDAP utiilzes HTTP to access data, so LP DAAC decided to deploy all the components for data access as HTTP services as well. Why HTTP services? Simple Standard based protocol Securable Dynamic interaction methods (get/post, etc.) Stream based data delivery

OPeNDAP – The base of LP DAAC Data access Data Access methods OPeNDAP - Open-source Project for a Network Data Access Protocol. Provides a direct way of subsetting LP DAAC data from our Data Pool.

LP DAAC OPeNDAP Deployment Hyrax was chosen for the LP DAAC deployment because of its capabilities for handling of MODIS tile data. Due to the uniform tiled nature of the LP DAAC’s MODIS data, we chose to aggregate data based on date. Due to the configuration of the ECS data pool we chose to programmatically create and update our aggregation files instead of using the scanning based aggregation capabilities. Aggregation scripts are available at ECC, contact me for further details. LP DAAC OPeNDAP Services http://opendap.cr.usgs.gov/opendap/hyrax

LP DAAC OPeNDAP Deployment To handle large volumes of data access LP DAAC deployed a cluster of servers to provide OPeNDAP services.

OPeNDAP Data Access Data Services Middleware API UI

OPeNDAP Data Access Data Services Middleware API UI

LP DAAC OPeNDAP Data Access link Data Services Middleware API UI

OPeNDAP Data Access Access and extract discrete subsets Spatial Band/Layers Temporal (from aggregation) Request data via URL http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] Returns data that the user intends to use! Reduces time spent downloading and processing Minimizes the amount of data downloaded Converts the data format if desired Data Services Middleware API UI

OPeNDAP Data Access OPeNDAP request: Return: http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.json?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] Return: { "name": "h11v04.ncml", "attributes": [ {"name": "HDF_GLOBAL","attributes": [ {"name": "HDFEOSVersion", \ "value": ["HDFEOS_V2.9"]} ] } ], "leaves": [ { "name": "sur_refl_b01", "type": "Float32", "attributes": [ {"name": "coordinates", "value": ["Latitude Longitude"]}, {"name": "long_name", "value": ["Surface_reflectance_for_band_1"]}, {"name": "units", "value": ["reflectance"]}, {"name": "scale_factor_err", "value": [0.0000000000000000]}, {"name": "add_offset_err", "value": [0.0000000000000000]}, {"name": "calibrated_nt", "value": [5]}, {"name": "orig_scale_factor", "value": [0.00010000000000000000]}, {"name": "orig_add_offset", "value": [0.0000000000000000]}, {"name": "_FillValue", "value": [-28672]}, {"name": "valid_min", "value": [-0.009999999776]}, {"name": "valid_max", "value": [1.599999905]}, {"name": "grid_mapping", "value": ["MOD_Grid_250m_Surface_Reflectance_eos_cf_projection"]} ], "shape": [1,11,11], "data": [[[0.5359, 0.5657, 0.5258, 0.4739, 0.4648, 0.5087, 0.5234, 0.5234, 0.6214, 0.5793, 0.5374], [0.5759, 0.5414, 0.5338, 0.4489, 0.432, 0.5036, 0.6051, 0.6214, 0.5793, 0.455, 0.446], [0.4594, 0.5409, 0.6224, 0.6289, 0.6289, 0.6051, 0.5005, 0.5005, 0.3325, 0.3092, 0.3826], [0.5831, 0.5831, 0.6126, 0.6307, 0.6307, 0.5277, 0.612, 0.3325, 0.3092, 0.332, 0.4036], [0.6329, 0.6126, 0.6061, 0.5691, 0.5691, 0.612, 0.5841, 0.5841, 0.4301, 0.4782, 0.6271], [0.6092, 0.6092, 0.6152, 0.5691, 0.6297, 0.6035, 0.5052, 0.5052, 0.4782, 0.6518, 0.6913], [0.5712, 0.6152, 0.6248, 0.5686, 0.5686, 0.3522, 0.5088, 0.5912, 0.6316, 0.4729, 0.5756], [0.5737, 0.5737, 0.524, 0.5368, 0.3522, 0.5691, 0.5944, 0.4085, 0.4729, 0.5375, 0.5375], [0.5606, 0.5211, 0.5368, 0.5874, 0.5874, 0.343, 0.26, 0.26, 0.3825, 0.5794, 0.5863], [0.5883, 0.5591, 0.5263, 0.4325, 0.343, 0.4485, 0.626, 0.626, 0.5462, 0.3682, 0.3463], [0.5435, 0.5263, 0.4325, 0.4887, 0.4887, 0.5175, 0.626, 0.5475, 0.4101, 0.4592, 0.4203]]] } ], "nodes": [] } Data Services Middleware API UI

OPeNDAP Data Access OPeNDAP URLs are complex Need to know: Syntax Product name Layer name Time Geolocation Tile locations (H & V) – Data is aggregated over time by tile Pixel location (Ydim & Xdim) http://opendap.cr.usgs.gov/opendap/hyrax/MOD09Q1.005/h11v04.ncml.json?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] Data Services Middleware API UI

With the OPeNDAP in Place what can be done? Using these building blocks the LP DAAC (or anyone who wants to really) can build new interfaces and systems like….

AppEEARS Application for Extracting and Exploring Analysis Ready Samples Web application interface for accessing and extracting LP DAAC’s tiled MODIS and WELD data Data Services Middleware API UI

https://lpdaacsvc.cr.usgs.gov/appeears/ AppEEARS location https://lpdaacsvc.cr.usgs.gov/appeears/ Data Services Middleware API UI

Questions? Questions? Data Services Middleware API UI

AppEEARS Built on top of services and tools Capabilities Easy access and selection of MODIS and WELD land data Extracts pixel values for sample point locations Data exploration and interaction Dynamic Graphs Tables Download Bundle Analysis-ready data Data Services Middleware API UI

For More Information about our services Subscribe to our listserv: https://lists.nasa.gov/mailman/listinfo/lpdaac

BEDI – LP DAAC Response Middleware Web Services Tilemap – Coverts Latitude and Longitude values into tile and pixel values for datasets located on a common grid structure. Fromjulian – Converts dates from a Julian date format to a mm-dd-yyyy format. Tojulian – Converts dates from a mm-dd-yyyy format to a Julian date format of the number of days since 01-01-2000. Product (coming soon!) – Service that provides information about the different products available from the LP DAAC. Inventory – Service that returns information about the granules in the LP DAAC inventory. Powered by NASA Common Metadata Repository (CMR). Data Services Middleware API UI