Decadal Survey NSIDC Thoughts on DS Data Systems Workshop Ron Weaver and Marilyn Kaminski

Purpose of this UWG Discussion Help us strategize data management for the DS missions and what role you (PoDAG) believe NSIDC should play How do we make the DS data streams work best for the cryosphere research community? What are our opportunities? Who should be our partners?

DS Workshop In late June NASA held a workshop on Decadal Survey data management. It was attended by reps from the DS missions (e.g. IceSAT2, SMAP, DESDynI etc.), the Earth Science Data Centers, including reps from the UWGs From NSIDC, Kaminski and Weaver (1/2 day) and Anderson attended For background on Decadal Survey Missions and data management please consult:

Voltz (What I heard) Missions must integrate well Handle Data to ECS standards Promote Synergy Bring in Foreign Data to DS Mission Support Measurements are Primary Bring Applications and Operations Communities into fold (with research community)

Decadal Survey Missions Soil Moisture Active/Passive (SMAP) ICESat-II Deformation, Ecosystem Structure and Dynamics of Ice (DESDynl) Climate Absolute Radiance and Refractivity Observatory (CLARREO) Hyperspectral InfraRed Imager (HyspIRI)

Where do these elements intersect with NSIDC DAAC systems? NSIDC has scientific links to SMAP, DESDynI, ICESat II and possibly CLARREO If we include ASF we could solidly manage SMAP, DESDynI and ICESat II But we could not manage all of them! So, where do we focus?

Possible Synergies Missions must integrate well – IceSat I, IceSat II, DESDynI (Lidar)IceBridge We are involved to some extent in these Synergy between NSIDC (Lidar) and ASF INSAR for DESDynI – AMSR-E, SSMIS, SMAP We can provide supporting data to SMAP MeaSURE program data from Frozen Ground ESDR (Kimbell), Enhanced PM data streams (Long) Foreign Data – Cryosat, GCOM-W Measurements are primary – Science disciplinary support for sea ice, ice sheets, snow hydrology

Synergies – Data systems Infrastructure for users – Searchlight and CCAWS are in part being built to handle new data streams Remove culture divide – Key element of IceBridge DM task is wider data dissemination and use Tie to SMAP and DESDynI data systems

How to move forward Integrative data strategies – Hold a passive microwave data provider workshop to make sure we are providing supportive data to SMAP, other missions. (should this include scatterometry?) – Expand the IceBridge instrument team integration work to include a broader set of Lidar data – Discuss DESDynI with ASF – Meet with the DS science/instrument teams about data issues How do we/should we lobby HQ

Backup Slides

Notes from the Meeting (Bolded items are of particular interest)

Voltz (What I heard) Missions must integrate well Handle Data to ECS standards Promote Synergy Bring in Foreign Data to DS Mission Support Measurements are Primary Bring Applications and Operations Communities into fold (with research community)

Maiden Present to Near Future System Features Coexistence of heterogeneous, distributed data providers/information Partners Minimal set of core standards Support for community specific standards Reusable software Service Oriented Architecture On-line archives and cross-system service invocation Ease of innovation and technology infusion

Breakouts – Standards Recommended Core Standards be developed and mandated by HQ as level 1 requirements In the next 6 months recommend a metadata standard and a file format standard for new missions/systems

Breakouts – Data Systems Infrastructure and Interfaces Need to provide infrastructure for ‘users’, i.e. that meets user needs as opposed to mission needs. Need tools that remove culture divide between experts and novices (Currently data sets are developed to facilitate mission scientists and processing, not users) Expand definition of ‘decadal missions’ to include other missions and measurements from other sources besides NASA including international missions, other US agencies

Breakouts – Data Systems … page 2 On-the-fly processing – cloud computing – parameter searching (not just granules) – virtualization of products Need to be careful of the concept of ownership of the data and the systems Consider developing a system mode that contains a center of excellence for a particular data product or product suite Creation of interoperable datasets Searching by parameter will require a very robust metadata

Searchlight NSIDC Searchlight Project Objectives (July 2008) Create a simple to use on-line interface(s) for cryospheric data Get NSIDC data (V0, ECS, other) distributed to more users Get the user just the (subsetted/reprojected) data they need, in the format they want, as quickly as possible Allow users to display and interact with the data Create a standard and flexible NSIDC framework for new applications Meet the base objectives of CCAWS, DADDI, CADIS, ACAP, GCAP & The DAAC

Searchlight Performance and Design Objectives Contributing grants and contracts include ACAP(A), CADIS(C), CCAWS(W), DADDI(D), GCAP (G) and the DAAC(T) In the list, letters in parentheses refer to the set of grants and contracts with specific mention of the requirement in the grant or contract proposal. For example, a `Searchlight requirement indicated with (ACW) refers to something specifically mentioned or promised in the proposal materials for ACAP, CADIS and CCAWS. system response times will depend on action performed: – 5-second response time for display results, or in-progress indicator – ??-second response time for data delivery (this should probably be a rate, as in seconds per file ordered, or seconds per GB of data ordered) the user will be able to access the system with a common web browser, with no plug-ins or special software for the user to download; Ajax functionality is specifically in-scope; the current list of supported browsers is – Windows: IE 7.x; Firefox 3.x – Macintosh/Linux: Firefox 3.x system will be built with open source (C) and/or Government-funded technologies (C) (W says "preferably open source") the user experience will be consistent; users will not be required to to know where the data are coming from, and will not be required to learn this in order to use the system. to maximize reuse and minimize maintenance costs, there will be a standard user interface built for each kind of information (spatial, temporal, etc). Each of these interfaces may have different ways for the user to specify input, (e.g. specifying a date using a calendar widget vs. a text box, but there will not be 2 implementations of a calendar widget) the system will incorporate reasonable security measures, for example, using ssl for at least user authentication when users navigate through the system, previously entered values will be remembered by the system

Challenges (from IceBridge proposal work) Timeliness Of Data Delivery Understanding And Responding To User Community Needs Complexity Of Data Streams, Especially Experimental Instrumentation – Changing Formats Of Experimental Data – Dataset Co-registration Appropriate Data Set And System Integration – Develop Tools And Services To Fit Complex Data – Or – Adapt Complex Data To Known Tools And Services Distributed (Virtual?) Data Management Systems – Who Holds What Data? Who Decides? How Does Interoperability Get Accomplished? Note That Archive Volume Is Not On This List IceBridge Data Management. September