1. April 2008ESG All-Hands meeting ESG Product Services Overview of components Issues in need of discussion Steve Hankin, NOAA/PMEL Roland Schweitzer, Weathertop Consulting

2. April 2008ESG All-Hands meeting Questions for consideration 1.Gateway Node versus Data Node components trade-offs: performance, complexity & system loading 2.AuthN & AuthZ implications of legacy back ends 3.How do we do "batch" products in ESG? 4.How to handle curvilinear data? 5.Ajax connections to the UI -- AuthN & AuthZ issues 6.What ESG component should generate CF file subsets? 7.Public access to OPeNDAP. To server-side calculations? 8.NCL back end and updates to CDAT

3. April 2008ESG All-Hands meeting F-TDS URL: “ SST from COADS minus TEMP from Levitus (regridded)” http://host.noaa.gov/thredds/data/coads _expr_{levitus.nc}{DIF=SST-TEMP[g=SST]} TDS (or BES?) Data- Source IOSP command file Ferret XML metadata binary data.dds?DIF URL.dods?DIF Let DIF = SST-TEMP[g=SST] Server-side transformations 2. run 1. write read write F-TDS

4. a “workflow engine”: reads data/makes products Product Server (Java) Remote Data Metadata (XML) Local Data LAS user interface Backend Service ‘Legacy’ application (e.g. graphics … NCL, CDAT, Ferret, NCO, …) REST API AJAX API Backend Service 2 RDBMS F-TDS+ Server-side data reduc’n, regrid, diff’ing UI Metadata (AJAX) product request XML (REST) OPeNDAP DAP (REST) back end request (SOAP) Backend Service 3 e.g. WMS Backend … …

5. April 2008ESG All-Hands meeting Product Server (Java) Remote Data Metadata (XML) Local Data LAS user interface Backend Service ‘Legacy’ application (e.g. graphics … NCL, CDAT, Ferret, NCO, …) REST API AJAX API Backend Service 2 RDBMS F-TDS+ Server-side data reduc’n, regrid, diff’ing UI Metadata (AJAX-REST) product request XML (REST) OPeNDAP DAP (REST) back end request (SOAP) Backend Service 3 e.g. WMS Backend … … If no xforms, regridding or differencing arte needed

6. April 2008ESG All-Hands meeting publication Differencing occurs at the Data node Distinct graphics by institution Server-side data reduction Data Node 2 Backend legacy app. UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) Data Node F-TDS+ Product Server Portal UI Data Nodes talk to other Data Nodes when differencing Level 5 Gateway Node

7. April 2008ESG All-Hands meeting Data Node 2 UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Backend legacy app. Data Node F-TDS+ Product Server Differencing occurs at the Gateway node Distinct graphics by institution Server-side data reduction Backend legacy app. F-TDS Product Server Portal UI Gateway Node Data Nodes don’t talk to other Data Nodes Level 4

8. April 2008ESG All-Hands meeting Data Node 2 UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Backend legacy app. Data Node F-TDS+ Backend legacy app. F-TDS Product Server Portal UI Gateway Node Differencing occurs at the Gateway node Distinct graphics by institution Server-side data reduction Level 3 Data Nodes doesn’t have full LAS

9. April 2008ESG All-Hands meeting Data Node 2 UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node F-TDS+ Backend legacy app. F-TDS Product Server Portal UI Gateway Node Differencing occurs at the Gateway node Uniform graphics for all inst’ns (or complx Gateway) Server-side data reduction Level 2

10. April 2008ESG All-Hands meeting Data Node 2 UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node OPeN- DAP Absolute minimum data node Backend legacy app. F-TDS Product Server Portal UI Gateway Node Differencing occurs at the Gateway node Uniform graphics for all inst’ns (or complx Gateway) NO Server-side data reduction Level 1 aggregation, only xform, regrid, diff.

11. April 2008ESG All-Hands meeting ESG Product Services Topics for discussion 1.Gateway Node versus Data Node components trade-offs: performance, complexity & system loading 2.AuthN & AuthZ implications of legacy back ends 3.How do we do "batch" products in ESG? 4.How to handle curvilinear data? 5.Ajax connections to the UI -- AuthN & AuthZ issues 6.What ESG component should generate CF file subsets? 7.Public access to OPeNDAP. To server-side calculations? 8.NCL back end and updates to CDAT

12. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS+ Backend legacy app. F-TDS+ Product Server Portal UI Gateway Node Recommendation: Level 2 – simple Level 5 – high power publication Backend legacy app. Data Node Level 5 F-TDS+ Product Server differencing 1.Gateway Node versus Data Node components

13. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS Backend legacy app. F-TDS Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS Product Server differencing 2.AuthN & AuthZ implications of legacy back ends Relink with ESG-aware ‘c’ client libraries?

14. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS Backend legacy app. F-TDS Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS Product Server differencing 3.How do we do "batch" products in ESG? What if F-TDS response will be delayed? How to retain Auth info when picking up delayed response?

15. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS Backend legacy app. F-TDS Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS Product Server differencing 4.How to handle curvilinear data? Diagnostic vars often must be computed on the native grids? Need to get a list of all likely diagnostic vars for AR5 in advance

16. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS Backend legacy app. F-TDS Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS Product Server differencing 5.Ajax connections to the UI -- AuthN & AuthZ issues? Anything to be worried about?

17. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS Backend legacy app. F-TDS Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS Product Server differencing 6.What ESG component should generate CF file subsets? (so-called ‘fileout’) Another REST protocol on the OPeNDAP server? What about ‘batch’? As another product’? A Gateway Node capability ?

18. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS Backend legacy app. F-TDS Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS Product Server differencing 7.Public access to OPeNDAP. Server-side calculations, too? (a policy question) Denial of service risk?

19. April 2008ESG All-Hands meeting UI Metadata (AJAX) product request (REST) OPeNDAP DAP (REST) back end request (SOAP) publication Data Node Level 2 F-TDS+ Backend legacy app. F-TDS+ Product Server Portal UI Gateway Node publication Backend legacy app. Data Node Level 5 F-TDS+ Product Server differencing 7.NCL back end and updates to CDAT (just a reminder) NCL and CDAT with SOAP wrappers?

20. April 2008ESG All-Hands meeting Whew! (the end)