1. BULK DATA RETRIEVAL ECHO Technical Interchange Meeting April 30 & May 1, 2013 Raytheon EED Program | ECHO Technical Interchange 2013

2. What is it For? Quick access to Publicly Available Data via URLs No processing options User Driven Pull Near-instant Raytheon EED Program | ECHO Technical Interchange 2013

3. State of the Union (Reverb) Put items in your cart, click “Download” URL Options: Data Metadata Browse Download Options Text File FTP Batch Script Raytheon EED Program | ECHO Technical Interchange 2013

4. How Does Reverb Do It Catalog-REST! Granule Searches “atom” format results Scan for “links” to URLs Download a file containing those links. Raytheon EED Program | ECHO Technical Interchange 2013

5. How do we do it? Catalog-REST! Granule Searches “atom” format results Scan for “links” to URLs Create a file containing those links. Get them. Raytheon EED Program | ECHO Technical Interchange 2013

6. Example (cURL)* curl -gG “https://testbed.echo.nasa.gov/catalog- rest/echo_catalog/granules.atom?echo_collection_id=C 3878-LPDAAC_ECS&bounding_box=10.488%2C- 0.703%2C53.331%2C68.906&temporal[]=2009-01- 01T10%3A00%3A00Z%2C2010-03- 10T12%3A00%3A00Z” This gets all granules with: echo_collection_id of: C3878-LPDAAC_ECS Spatial bounding box: 10.488, -0.703, 53.331, 68.906 (W, S, E, N) Time constraint: 2009-01-01T10:00:00Z - 2010-03-10T12:00:00Z ~80 hits! Use -I as options to curl, and look for: “Echo-Hits” * also from perl/bulk/get_bulk.pl Raytheon EED Program | ECHO Technical Interchange 2013

7. What do the Results Look Like? G10607-LPDAAC_ECS SC:MCD43A4.005:2075808749 2009-10-15T14:01:49.076Z MODIS/Terra+Aqua Nadir BRDF-Adjusted Reflectance 16-Day L3 Global 500m SIN Grid V005 MCD43A4.A2009257.h21v08.005.2009276131145.hdf 57.7068 LPDAAC_ECS 2009-09-14T00:00:00.000Z 2009-09-29T23:59:59.999Z 3.85518158489962e-05 29.8878504914521 -0.00342414555683897 40.0119084380163 9.99985925957934 40.6260396971992 10.0030323070386 30.3449665340407 3.85518158489962e-05 29.8878504914521 true DAY YIKES! Raytheon EED Program | ECHO Technical Interchange 2013

8. “link” is your friend! Raytheon EED Program | ECHO Technical Interchange 2013

9. Only link…/data# Please! curl -gG “https://testbed.echo.nasa.gov/catalog- rest/echo_catalog/granules.atom?echo_collection_id=C 3878-LPDAAC_ECS&bounding_box=10.488%2C- 0.703%2C53.331%2C68.906&temporal[]=2009-01- 01T10%3A00%3A00Z%2C2010-03- 10T12%3A00%3A00Z” | perl –ne “printf if m/link.*\/data#/;” “Just show me the results that have ‘data’ type in the links” Slightly more clever perl: … | perl -anF/\”/ -e “printf qq(\$F[1]\n) if m/link.*\/data#/;” But watch out for Windows vs. Mac/Linux quoting! Raytheon EED Program | ECHO Technical Interchange 2013

10. Loop it Over your Echo-Hits append &page_size=500 to URL_String end_pages = (Echo-Hits DIV page_size) + 1 for page 1.. end_pages curl (string+&page_num=$page) | clever.perl >> output.URLs Now you have an output.URLs with lots of URLs in them… Raytheon EED Program | ECHO Technical Interchange 2013

11. So What if I Have Some URLs? Scripting curl to the rescue! Linux/Mac/Unix: for url in $(<output.URLs); do curl $url -OL -s; done Windows: for /f %f in (output.URLs) do curl %f -OL Raytheon EED Program | ECHO Technical Interchange 2013

12. Questions? Raytheon EED Program | ECHO Technical Interchange 2013