Ice Bridge Data Management Scope, Challenges, Solutions Ron Weaver, NSIDC DAAC Manager National Snow and Ice Data Center Supporting Cryospheric Research Since 1976

Topics Vision, Guiding Principles Science Goals Data Management Goals  In Response to Science Goals Proposed Activities Schedule Challenges Questions Next Steps IceBridge Data Management. September

The Global Vision Provide timely access to data and products Provide products useful and understood by the broader data community Campaign data available before next campaign

Science Goals - IceBridge Ice Sheet Surface elevation data after ICESat-1 mission ends  Areas of rapid ice sheet change, Sea ice mass estimates Computed measurements critical to ice models  Bed topography, grounding lines, ice and snow thickness Link ICESat-1 to ICESat-2 through baseline inter- calibration measurements Campaigns to support sea ice (thickness, type, extent) Technology incubator and test-bed  Airborne and satellite sensors, UAV, photon counting, cross- track measurements IceBridge Data Management. September

Data System (at NSIDC) Goals 1.Data system supports science goals and user needs 2.Provide appropriate and adaptable data services and tools for these data 3.Provide (and/or promote) understandable, accessible, and relevant data products to broader science community 4.Manage data sets from IceBridge, ICESat-1 and WFF ATM (via funded ACCESS project) in a coherent and interoperable manner 5.Prototype the ICESat-2 data system 6.Data Stewardship IceBridge Data Management. September

CyberInfrastructure (and related) Challenges Understanding the scientific, especially interdisciplinary, needs Data acquisition—timeliness, consistent formats, easy metadata creation, research vs. operational collection Data system automation and optimization Data for models and synthesis—matching data to model needs Stewardship in a distributed environment (e.g. tracking provenance) Sustained preservation IceBridge Data Management. September

Conceptual Plan Work (lots of it) on harmonizing data for science user communities  “getting things to play well together” Building products (precursors to products) Integrated Tools (and Services) Use existing NSIDC systems, for data management Could be distributed architecture that takes advantage of existing instrument data archives IceBridge Data Management. September

Schematic OIB Data System Architecture Instrument, Flight Dynamics, Ancillary Data Sources Instrument Facility DM System Instrument Facility DM System Aggregator Services DM System Aggregator Services DM System NSIDC DAAC DM System (ECS, non-ECS) NSIDC DAAC DM System (ECS, non-ECS) User Interfaces, Tools and Services Users ECHO/GC MD Metadata Database Data flows Metadata flows

Data Types L0 L1 L2 L3 Lidars ATMGLASLVIS UAF LiDAR Ice Radars PARISKU-IceKu-Snow…… GravityMagnetics Profiles/Swath Points - Waveforms Profiles/Swath Points - Waveforms Point cloud Wavefor m DEM surface DEM surface Profiles/Swath Return Energy - Waveforms Profiles/Swath Return Energy - Waveforms Ice in-situ layers Ice in-situ layers Snow Accum. Snow Accum. DEM bed DEM bed profile s Gravity Potential surface Gravity Potential surface Aeroma g Potential surface Aeroma g Potential surface Surface maps Sea ice – ice sheets Ice sheet roughness Surface maps Sea ice – ice sheets Ice sheet roughness External Products and Services Aggregation Services Models Data Processing Levels ServicesData Castings Tools USERS ICE CAP ICE CAP LDEO

Proposed Task Sequencing ESTABLISH OIB PROJECT TEAM Coordinate DAAC OIB data mgmt. activities at NSIDC Coordinate NSIDC activities with: Science team Instrument teams ESDIS and HQ ESTABLISH OIB PROJECT TEAM Coordinate DAAC OIB data mgmt. activities at NSIDC Coordinate NSIDC activities with: Science team Instrument teams ESDIS and HQ CONDUCT DATA ASSESSMENT Establish OIB initial baseline data list Characterize existing data formats, metadata processes, services, etc. Determine initial LOS for all OIB data sets Assess tools and services needed by OIB Science and Instrument Teams and User Community Provide NASA HQ with work plan with cost estimates for remaining work CONDUCT DATA ASSESSMENT Establish OIB initial baseline data list Characterize existing data formats, metadata processes, services, etc. Determine initial LOS for all OIB data sets Assess tools and services needed by OIB Science and Instrument Teams and User Community Provide NASA HQ with work plan with cost estimates for remaining work BROKER OIB DATA SETS Metadata entries in NSIDC data catalog with LOS designated Collect/extract/reformat documentation and metadata Metadata entries transferred to ECHO and GCMD BROKER OIB DATA SETS Metadata entries in NSIDC data catalog with LOS designated Collect/extract/reformat documentation and metadata Metadata entries transferred to ECHO and GCMD ESTABLISH LIST OF DATA FOR INGEST AT NSIDC Determine LOS and resources needed for transfer Negotiate agreements with Instrument Team(s) Determine transfer phasing ESTABLISH LIST OF DATA FOR INGEST AT NSIDC Determine LOS and resources needed for transfer Negotiate agreements with Instrument Team(s) Determine transfer phasing EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS Determine phasing of development efforts EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS Determine phasing of development efforts IMPLEMENT OIB DATA PORTAL Rapidly implement focused entry point for OIB data and services From metadata entries in NSIDC and ECHO data catalogs with LOS designated Reuse/Augment SearchLight, CCAWS/SAGE technologies IMPLEMENT OIB DATA PORTAL Rapidly implement focused entry point for OIB data and services From metadata entries in NSIDC and ECHO data catalogs with LOS designated Reuse/Augment SearchLight, CCAWS/SAGE technologies IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES Consult with Instrument Teams IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES Consult with Instrument Teams IMPLEMENT DATA AGGREGATION SERVICES INGEST DATA INTO NSIDC VO OR ECS SYSTEMS Phased ingest based on established priority INGEST DATA INTO NSIDC VO OR ECS SYSTEMS Phased ingest based on established priority A A C C D D E E F F H H I I G G B B

Task Sequencing – After Preliminary Discussions With NASA HQ ESTABLISH OIB PROJECT TEAM Coordinate DAAC OIB data mgmt. activities at NSIDC Coordinate NSIDC activities with: Science team Instrument teams ESDIS and HQ ESTABLISH OIB PROJECT TEAM Coordinate DAAC OIB data mgmt. activities at NSIDC Coordinate NSIDC activities with: Science team Instrument teams ESDIS and HQ CONDUCT DATA ASSESSMENT Establish OIB initial baseline data list Characterize existing data formats, metadata processes, services, etc. Determine initial LOS for all OIB data sets Assess tools and services needed by OIB Science and Instrument Teams and User Community Provide NASA HQ with work plan with cost estimates for remaining work CONDUCT DATA ASSESSMENT Establish OIB initial baseline data list Characterize existing data formats, metadata processes, services, etc. Determine initial LOS for all OIB data sets Assess tools and services needed by OIB Science and Instrument Teams and User Community Provide NASA HQ with work plan with cost estimates for remaining work BROKER OIB DATA SETS Metadata entries in NSIDC data catalog with LOS designated Collect/extract/reformat documentation and metadata Metadata entries transferred to ECHO and GCMD BROKER OIB DATA SETS Metadata entries in NSIDC data catalog with LOS designated Collect/extract/reformat documentation and metadata Metadata entries transferred to ECHO and GCMD ESTABLISH LIST OF DATA FOR INGEST AT NSIDC Determine LOS and resources needed for transfer Negotiate agreements with Instrument Team(s) Determine transfer phasing ESTABLISH LIST OF DATA FOR INGEST AT NSIDC Determine LOS and resources needed for transfer Negotiate agreements with Instrument Team(s) Determine transfer phasing EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS Determine phasing of development efforts EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS Determine phasing of development efforts IMPLEMENT OIB DATA PORTAL Rapidly implement focused entry point for OIB data and services From metadata entries in NSIDC and ECHO data catalogs with LOS designated Reuse/Augment SearchLight, CCAWS/SAGE technologies IMPLEMENT OIB DATA PORTAL Rapidly implement focused entry point for OIB data and services From metadata entries in NSIDC and ECHO data catalogs with LOS designated Reuse/Augment SearchLight, CCAWS/SAGE technologies IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES Consult with Instrument Teams IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES Consult with Instrument Teams IMPLEMENT DATA AGGREGATION SERVICES INGEST DATA INTO NSIDC VO OR ECS SYSTEMS Phased ingest based on established priority INGEST DATA INTO NSIDC VO OR ECS SYSTEMS Phased ingest based on established priority A A C C D D E E F F H H I I G G B B PROVIDE APPROPRIATE DATA PRODUCTS DERIVED FROM INGEST DATA STREAMS

Current NSIDC Schematic Architecture Ingest Paths Archival Systems Data & Metadata Production Interfaces & GUIs Services EOS Satellite Data V0 Ingest Paths PDR Server PM Ingest PM Ingest Searchligh t Ingest (TBD) Searchligh t Ingest (TBD) PM Production PM Production Manual Ingest WIST DP Interface DP Interface Machine to machine Sub setters DataPool SAN ECS ECHO V0 SAN V0 MDDB (to be replaced) SearchLight OIB Portal Legacy GUIs SAGE CCAW S GUI Service Casting SLight GUI Lines indicate data and metadata flows Dotted outlines are functions under development Grayed out items are to be replaced Rose colored items are proposed

Data Sets - a very preliminary list InstrumentData System?NSIDC Ingest?Data Levels LVISYes, formats defined, ftp delivery Yes if needed ( use distributed DM?) L0 - L3 ATMYes, formats defined, ftp delivery Yes, plannedL0 - L3 PARIS??L0 Gravity??? KU ice penetrating radar ftp delivery of higher lvl prods. Yes, KU has suggested L0 in NSIDC syst L0-L3 KU snow radarKU ftp siteMaybeL0?, L2-3 Flight dynamics ??L0? Our first guesses, based on limited information. 13

Schedule HQ decision on NSIDC Proposal (November?)  scope, level of effort, schedule NSIDC appoint a project leader and identify team members (January?)  (Proj Mgr, scientist, USO, OPS, Engr)  probably aggregate fte level of effort Implement approved tasks  Work on integrated data products  ?Conduct Data Assessment (prelim report March 2010?) Iterate with NASA over requirements IceBridge Data Management. September

Challenges Timeliness Of Data Delivery Understanding And Responding To User Community Needs Complexity Of Data Streams, Especially Experimental Instrumentation  Changing Formats Of Experimental Data  Dataset Co-registration Appropriate Data Set And System Integration  Develop Tools And Services To Fit Complex Data Or  Adapt Complex Data To Known Tools And Services Distributed (Virtual?) Data Management Systems  Who Holds What Data? Who Decides? How Does Interoperability Get Accomplished? Note That Archive Volume Is Not On This List IceBridge Data Management. September

Principles of Data Management IceBridge Data Management. September 2009 Preservation without access is pointless; access without preservation is impossible Keep it simple and flexible: it’s about data, not systems! Think about long-term archiving when planning for data collection Document uncertainty Involve scientists and users in data management, and data managers in science 16

BACKUP

It will use NSIDC’s Searchlight USERS JAZ Struts 2 Action Objects (controller) Struts 2 Action Objects (controller) WCS Service External Database External Database Metadata Database Metadata Database JPA Hibernate JPA Hibernate ECS File System ECS File System ECHO Metadata ECHO Metadata NSIDC File System NSIDC File System Service Objects Repository Objects Domain Objects Brokered Database Brokered Database Workflow Objects WMS Service WFS Service Analysis Objects CCAWS UI G-CAP OGC UI External Metadata External Metadata SearchLight JSP Pages (view) SearchLight JSP Pages (view) OPeNDAP Server Other Systems AJAX Libraries (DOJO, YUI) SearchLight CCAWS G-CAP User Interface Layer Service Layer Domain (model) Layer Infrastructure Layer Other Systems Core Software Architecture IceBridge Data Management. September

Searchlight Components IceBridge Data Management. September

Conceptual Flow Users: cryo Ice sheets Ice sheets Sea Ice Sea Ice Outlet Glaciers Outlet Glaciers Snow on Ice Snow on Ice Users: Gravity Users: Vegetation? Users: synergistic Modelers Modelers Sea Level Sea Level Instruments ICESat-1 GLAS LVIS UAVSAR ATM PARIS Test instruments Searchlight ECHO Primary Data Route Metadata Route Instr. Team to User ECS Other Data Distribute To Cryo. Community Platforms and Instruments Processing By Instr. team Data Management By Instr. Team or Facility Distribute To Other communities Synergistic Tools Between Communities IceBridge Data Management. September