1. OpenDAP Server-side Functions for Multi-Instrument Aggregation ESIP Session: Advancing the Power and Utility of Server-side Aggregation Jon C. Currey (NASA), Aron Bartle (Mechdyne), James Gallagher (OpenDAP)

2. Challenge How to match and merge high resolution observation data (L1 & L2) from instruments on separate spacecraft? How do I get just the data I need? I don’t want it all! How do I handle different footprint sizes? What about spectral differences? Maybe I should just stick with gridded data

3. Science Use Cases There are many studies that require access to full resolution observation data. Today we will address only a few: – Intercalibration (GSICS community) – Validation of L2 Retrievals – Surface site data mining

4. Multi-Instrument Intercalibration (MIIC) Funded by ROSES ACCESS Spinoff of the CLARREO mission Addresses how to efficiently match and access instrument data from multiple data centers

5. MIIC Tiered Architecture

6. How does MIIC Aggregate Data? Two critical pieces: – Event prediction – OPeNDAP server-side functions – Together they perform efficient aggregation!

7. Part 1 – MIIC Event Prediction Process LEO-LEO intercalibration event opportunity based on the time when the primary instrument “P” FOVs are within the tent structure built using the secondary instrument location at A and fictitious locations at A+ and A- based on allowable observation time differences (eg., 2.5 minutes). Table shows the effect of orbit altitude differences on frequency of events (crossings). Event boundary trimmed to where instrument viewing condition differences within specified ranges (use viewing zenith, relative azimuth, and solar zenith angles)

8. Part 1 - MIIC Event Prediction Sample Results Can find event opportunities for many spacecraft combinations! NPP VIIRS vs. Aqua CERES Event Prediction, Oct. 2013, daytime, SZA < 75°, |Δ VZA| < 10°, |Δ RAZ| < 30°, |Δ time| < 2.5 min.

9. Part 2 – OPeNDAP Server-side Functions Two main server-side functions: Tuple and Histogram Use OPeNDAP nested functions Data variable functions identify parameters to return Filter functions define filter variables and ranges Filter variable functions are AND’ed Tuple function returns full resolution filtered Array data Histogram function returns filtered data binned in one or two dimensions; returns average + statistics for each bin Option to decimate data prior to tuple and histogram Server-side functions are generic – should work on all netCDF and HDF L1 and L2 files

10. OPeNDAP Tuple Server-side Function Returns Full Resolution Data for Single LEO-GEO Predicted Event Full resolution LEO-GEO predicted matched event, NPP VIIRS vs. GOES13 for Jan. 1, 2013, 51.2S-46.4N, 114.7W-53.2W, T18:49:40Z-T19:17:00Z; 1 km GOES13 Band 1 visible data (left); matched 2km VIIRS Radiance_I3_Avg data from multiple aggregated NPP_VIMD_SS files (right). “Bow tie” fill values at scan angles greater than 31.7° and 56.3° need to be filtered. Accessing GOES and VIIRS data from OPeNDAP servers using tuple server-side function. Full resolution LEO-GEO tuple data is still a significant amount of data to download. Use 2Dhistogram function to grid and reduce data by several orders of magnitude!

11. OPeNDAP 2DHistogram Server-side Function Returns 0.5° Gridded Data for Single LEO-GEO Predicted Event One gridded LEO-GEO event, 0.5 degree, NPP VIIRS (right) vs. GOES13 (left) for Jan. 1, 2013. MIIC 2DHistogram server-side function reduces data by a factor of 3500.

12. Surface Site Data Mining Using MIIC Server-side Filtering and Statistics Deep Convective Cloud (DCC) Filter implemented within OPeNDAP tuple function; cloud top height >10km, cloud optical depth >10, cloud fraction > 99%, latitude band 25°S – 25°N. MIIC event prediction determines file coverage for region, then retrieves only filtered data from OPeNDAP servers. DCC are large cold bright clouds in the tropics (see LW and SW flux).

13. Challenging Validation Example Image on left shows CALIPSO groundtrack (blue) and aggregated VIIRS radiance data Using MIIC to support CrIMSS Validation Study (X. Liu); need to merge CALIPSO, CRIS, ATMS, VIIRS, and CrIMSS data Current system returns full resolution data from each instrument within event boundary (“coarse” match) using tuple server-side function Dealing with multiple instruments types (imager, profiler, spectrometer, sounder) all with different footprint sizes! Need to develop Spatial Registration Server-side function to match data at the footprint scale!

14. New Generalized Spectral Convolution Server-side Function Needed for intercalibration – convolve imager RSRs with reference spectra Demonstrated using hyperspectral SCIAMACHY and L1 MODIS data Allows users to compare (aggregate) imager bands to a hyperspectral reference Massive reduction in data transfer Spectral nadir reflectance over deep convective cloud scene from SCIAMACHY data with MODIS VIS and near-IR bands

15. More traditional types of aggregation (NCML-like) to be handled in OPeNDAP Need to aggregate CALISPO Aerosol and Cloud properties stored in separate files; have same time and spatial scales NPP VIIRS SDR channels stored in separate files; need to access geolocation and muliple bands within same read to perform server-side filtering

16. Summary Within the Earth Sciences community there is still a need to provide efficient access to data located across multi-agency data centers The MIIC tiered system using OPeNDAP server-side functions is one mechanism to match (aggregate) data from multiple instruments Thank you! For more information on MIIC contact jon.c.currey@nasa.gov