U.S. Department of the Interior U.S. Geological Survey LP DAAC Big Earth Data Initiative (BEDI) Developed Web Services 1 Jason Werpy LP DAACEnterprise Architect SGT, Inc., Contractor to the USGS EROS Center *Work performed under USGS contract G15PD00766

NASA’s LP DAAC  Land Processes (LP) Distributed Active Archive Center (DAAC)   Located in Sioux Falls, SD at the USGS Earth Resources Observation Science (EROS) Center 2

LP DAAC Surface Reflectance, Land Cover, Vegetation Indices LP DAAC Surface Reflectance, Land Cover, Vegetation Indices

Big Earth Data Initiative  NASA’s BEDI efforts  Increase accessibility of Earth Science Data  Increase availability of Earth Science Data  Increase usability of Earth Science Data 4

BEDI – LP DAAC Response  The LP DAAC decided that to address the BEDI goals, we would work on services designed and architected to give users improved data visibility, access, and tools. 5

BEDI – LP DAAC Response  Tiered Concept  LP DAAC created a tiered approach to data access 6 Data Services Middleware API UI

A Quick Aside – Getting data  Why HTTP services?  Simple  Standard bases protocol  Securable  Dynamic interaction methods (get/post, etc.)  Stream based data delivery  It’s even… 7

A Quick Aside – Getting data  Shown as a good way to get data in the ‘future’… 8

BEDI – LP DAAC Response  Data Access methods  OPeNDAP - Open-source Project for a Network Data Access Protocol. Provides a direct way of subsetting LP DAAC data from our Data Pool. 9 Data Services Middleware API UI

BEDI – LP DAAC Response  Middleware Web Services  Tilemap – Coverts Latitude and Longitude values into tile and pixel values for datasets located on a common grid structure.  Fromjulian – Converts dates from a Julian date format to a mm- dd-yyyy format.  Tojulian – Converts dates from a mm-dd-yyyy format to a Julian date format of the number of days since  Product (coming soon!) – Service that provides information about the different products available from the LP DAAC.  Inventory – Service that returns information about the granules in the LP DAAC inventory. Powered by NASA Common Metadata Repository (CMR). 10 Data Services Middleware API UI

BEDI – LP DAAC Response  Interface  AppEEARS – Interface for selecting point based data from the LP DAAC archive, analyzing that data, and downloading the specific values desired. 11 Data Services Middleware API UI

OPeNDAP Data Access  OPeNDAP was a key component of NASA’s BEDI efforts  OPeNDAP access to the LP DAAC’s Data Pool provides information about singular data granules.  Because MODIS tiles (and WELD tiles) are stored in a uniform grid of tiled data, the LP DAAC was able to create Aggregations by date. This allows an extra parameter that selects data from the date desired.  LP DAAC OPeNDAP Services  12 Data Services Middleware API UI

OPeNDAP Data Access 13 Data Services Middleware API UI

OPeNDAP Data Access 14 Data Services Middleware API UI

OPeNDAP Data Access 15 Data Services Middleware API UI

OPeNDAP Data Access  Access and extract discrete subsets Spatial Band/Layers Temporal (from aggregation)  Request data via URL l.ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010]  Returns data that the user intends to use!  Reduces time spent downloading and processing  Minimizes the amount of data downloaded 16 Data Services Middleware API UI

OPeNDAP Data Access OPeNDAP request: 1[683:1:683][2000:1:2010][2000:1:2010] Return: Dataset: h11v04.ncml sur_refl_b01[0][0], , , , , , , , , , , sur_refl_b01[0][1], , , , , 0.432, , , , , 0.455, sur_refl_b01[0][2], , , , , , , , , , , sur_refl_b01[0][3], , , , , , , 0.612, , , 0.332, sur_refl_b01[0][4], , , , , , 0.612, , , , , sur_refl_b01[0][5], , , , , , , , , , , sur_refl_b01[0][6], , , , , , , , , , , sur_refl_b01[0][7], , , 0.524, , , , , , , , sur_refl_b01[0][8], , , , , , 0.343, 0.26, 0.26, , , sur_refl_b01[0][9], , , , , 0.343, , 0.626, 0.626, , , sur_refl_b01[0][10], , , , , , , 0.626, , , , Data Services Middleware API UI

OPeNDAP Data Access  OPeNDAP URLs are complex  Need to know: Syntax Product name Layer name Time Geolocation  Tile locations (H & V) – Data is aggregated over time by tile  Pixel location (Ydim & Xdim) ascii?sur_refl_b01[683:1:683][2000:1:2010][2000:1:2010] 18 Data Services Middleware API UI

Utility / Metadata Services  These Services can help fill in the “complicated” parts of the OPeNDAP URLs  Geolocation  The Tilemap service was created to convert Latitude and Longitude information into tile H and V coordinates and x and y pixel locations within the tile.  Time Information  Time data in OPeNDAP is returned as “number of days since which is hard to convert “in your head” 19 Data Services Middleware API UI

Utility / Metadata Services  Product Service  The LP DAAC’s (coming soon!) Product Service can be used to get information about LP DAAC products and the bands contained within those products.  Inventory Service  The inventory service provides a view of data granules in the LP DAAC archive, it presents an alternate way to OPeNDAP for data access if a user wants to find entire granules of data. 20 Data Services Middleware API UI

Utility / Metadata Services and Data Services  Demo  Tilemap (also - how do I use this; or – Help!)  An OPeNDAP request (through the LP DAAC aggregated view)  Fromjulian  Tojulian  Inventory Service 21 Data Services Middleware API UI

Demo 22 Data Services Middleware API UI Wait…. we don’t know how this service works! But if you don’t know how to call it the service will tell you…. Service Call

Demo 23 Data Services Middleware API UI Service Help Response

Demo 24 Data Services Middleware API UI 1&latitude=44.306&longitude= Service Call Service Response

Demo 25 Data Services Middleware API UI Located!

Demo  Location is now known  Create OPeNDAP call for that location e[1366:1:1366][176:1:176],Longitude[1366:1:1366][176:1:176],sur_refl_b01[730:1:7 33][1366:1:1366][176:1:176],time[730:1:733]  Output Dataset: h11v04.ncml Latitude[0], Longitude[0], sur_refl_b01[0][0], sur_refl_b01[1][0], sur_refl_b01[2][0], sur_refl_b01[3][0], time, 5860, 5868, 5876, Data Services Middleware API UI

Demo  Output Dataset: h11v04.ncml Latitude[0], Longitude[0], sur_refl_b01[0][0], sur_refl_b01[1][0], sur_refl_b01[2][0], sur_refl_b01[3][0], time, 5860, 5868, 5876, 5884  The [730:1:733] is getting the last four acquisition dates, but what are they? Fromjulian can convert the values to dates , 5868, 5876, 5884 Jan 17, 2016;Jan 25, 2016;Feb 2, 2016;Feb 10, 2016  Tojulian can convert calendar dates to the from dates to calculate these values in reverse. 27 Data Services Middleware API UI

Demo  Other LP DAAC Middleware services provide higher level metadata interactions  Inventory Service – Lists LP DAAC granules and select granule information (powered by NASA CMR)  Lets take a look at this service…. 28 Data Services Middleware API UI

Demo  Inventory Service  The LP DAAC Inventory service returns select information about granules in the LP DAAC inventory. It utilizes the information in the NASA Common Metadata Repository (CMR)  URL for the Service  29 Data Services Middleware API UI

Demo 30 Data Services Middleware API UI Service Help Response

Demo 31 Data Services Middleware API UI Service Form Response

Demo 32 Data Services Middleware API UI Service Form Response

Demo 33 Data Services Middleware API UI Service REST GET Call product=MOD11A1.006& latitude=43.6& longitude=-96.7& date= ,

Demo 34 Data Services Middleware API UI Service Data Response

Demo 35 Data Services Middleware API UI Service REST GET Call product=MOD11A1.006& bbox=-98.7,42.8,-96.7,43.6& julianrange=90-132& years= & response=metadataurl&output=html

Demo 36 Data Services Middleware API UI Service Data Response

Demo 37 Data Services Middleware API UI Click on the URL, get the metadata file 1.0 EDC SC:MOD11A1.006: :03: MOD11A1 6 MOD11A1.A h10v hdf CKSUM DPLIngst. Hint: This would work the same for the browse URL or the data granule URL!

With the Building Blocks in Place What Next?  Using these building blocks the LP DAAC (or anyone who wants to really) can build new interfaces and systems like…. 38

AppEEARS  Application for Extracting and Exploring Analysis Ready Samples  Web application interface for accessing and extracting LP DAAC’s tiled MODIS and WELD data 39 Data Services Middleware API UI

AppEEARS  Built on top of services and tools  Capabilities  Easy access and selection of MODIS and WELD land data  Extracts pixel values for sample point locations  Data exploration and interaction  Dynamic Graphs  Tables  Download Bundle  Analysis-ready data 40 Data Services Middleware API UI

DEMO 41 Data Services Middleware API UI

Future Activities  The LP DAAC is planning on expanding its service offerings in the future  AppEEARS Area Services API and UI  Quality Values Services  Data Analysis Services  Addition of more tiled datasets into the API and UI 42 Data Services Middleware API UI

Future Activities  Area Services will allow for the selection of data over more than just the pixel space.  Data harmonization  Align different datasets  Sample datasets into comparable samples. 43 Data Services Middleware API UI

For More Information about our services 44 Subscribe to our listserv:

AppEEARS has an API too!  Even AppEEARS is built on top of public services, the AppEEARS API is directly usable, too (its not just an interface!)  AppEEARS API  Product Service – list of products available through AppEEARS (note the same help context)  Quality Service – list the quality layers for products  Sample Extraction Service – extracts a sample of data from OPeNDAP  DEMO 45 Data Services Middleware API UI

Future Activities 46 Data Services Middleware API UI

Demo 47 Data Services Middleware API UI Service Help Response - json

Demo 48 Data Services Middleware API UI Service Help Response - xml

Demo  If we know the dates we want are 10/08/2015, 10/16/2015, and 10/24/2015, we can find the number of days since for those dates &output=f2k Output: 5759,5767,5775 Match those dates with the output form the time parameter to get that data ] Dataset: h11v04.ncml time, 5759, 5767, Data Services Middleware API UI

Demo  Now we can get that data ml.ascii?Latitude[1365:1:1365][176:1:176],Longitude[1365:1:1365][176:1:176],sur_refl_b01[717:1: 719][1365:1:1365][176:1:176],time[717:1:719]  Output Dataset: h11v04.ncml Latitude[0], Longitude[0], sur_refl_b01[0][0], sur_refl_b01[1][0], sur_refl_b01[2][0], time, 5759, 5767, Data Services Middleware API UI

Demo  Product Service  The LP DAAC Product service returns select information about products in the LP DAAC archive.  Coming soon to URL  51 Data Services Middleware API UI

Demo 52 Data Services Middleware API UI Service Help Response

Demo 53 Data Services Middleware API UI Service Help - Form Response

Demo 54 Data Services Middleware API UI Service Form Entry

Demo 55 Data Services Middleware API UI Service Data Response [ { "collection_data": { "title": "Surface Reflectance 8-Day L3 Global 500m", "version_id": "006", "collection_name": "MOD09A1.006", "short_name": "MOD09A1", "data_center": "LP DAAC" } } ]

Demo 56 Data Services Middleware API UI Service REST GET Call

Demo 57 Data Services Middleware API UI Service Data Response [ { "collection_data": { "layers": [ { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 1", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b01", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 2", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b02", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 3", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b03", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 4", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b04", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 5", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b05", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 6", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b06", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "16-bit unsigned integer", "scale_factor": "0.0001", "description": "Surface Reflectance Band 7", "units": "Reflectance", "offset": "\n", "valid_range": "-100 to 16000", "pixel_size": "\n", "sds_name": "Sur_refl_b07", "rows_cols": "\n", "fill_value": "-28672" }, { "data_type": "32-bit unsigned integer", "scale_factor": "NA", "description": "Surface reflectance 500m band quality control flags", "units": "Bit Field", "offset": "\n", "valid_range": "0 to ", "pixel_size": "\n", "sds_name": "Sur_refl_qc_500m", "rows_cols": "\n", "fill_value": " " }, { "data_type": "16-bit signed integer", "scale_factor": "NA", "description": "Surface eflectance 500m state flags", "units": "Bit Field", "offset": "\n", "valid_range": "0 to 57343", "pixel_size": "\n", "sds_name": "Sur_refl_state_500m", "rows_cols": "\n", "fill_value": "65535" }, { "data_type": "16-bit signed integer", "scale_factor": "0.01", "description": "MODIS solar zenith angle", "units": "Degrees", "offset": "\n", "valid_range": "0 to 180", "pixel_size": "\n", "sds_name": "Sur_refl_szen", "rows_cols": "\n", "fill_value": "0" }, { "data_type": "16-bit signed integer", "scale_factor": "0.01", "description": "MODIS view zenith angle", "units": "Degrees", "offset": "\n", "valid_range": "0 to 180", "pixel_size": "\n", "sds_name": "Sur_refl_vzen", "rows_cols": "\n", "fill_value": "0" }, { "data_type": "16-bit signed integer", "scale_factor": "0.01", "description": "MODIS relative azimuth angle", "units": "Degrees", "offset": "\n", "valid_range": "-8000 to 18000", "pixel_size": "\n", "sds_name": "Sur_refl_raz", "rows_cols": "\n", "fill_value": "0" }, { "data_type": "16-bit signed integer", "scale_factor": "NA", "description": "Day of the year for the pixel", "units": "Julian day", "offset": "\n", "valid_range": "0 to 366", "pixel_size": "\n", "sds_name": "Sur_refl_day_of_year", "rows_cols": "\n", "fill_value": "-1" } ] } } ]