November 7th, 2012 Mike Daniels EOL Data Services update: Legacy Data, EarthCube and the Earth Science Information Partners (ESIP)

Legacy Data

From the 2011 EOL Science Review Team Report “The EOL effort to rescue and digitally archive historical field data that are presently stored in older paper, chart, and tape formats will be an extremely valuable information asset for research requiring long-term records.” “Continue progress toward and maintain a priority for making current and historical field data easily accessible”

(2003!) Data Lessons Learned Climate Science 1.0: Observational data is very similar to model data Time Value Obs data Model data Climate Science 2.0: Observational data is very different from model data Integrated Parameter Engineering Units Parameter Derived Parameter Metadata QC

Legacy Data Activities, 1 st proposed in Spring 2010 (FY11 Annual Budget Review) Data and Metadata Rescue Project Documentation - Collect, digitize and webify - Improve documentation Digitize Aircraft Videos Media and Format Migration (e.g. GENPro to NetCDF) NCAR Iron Mountain Archives Embarked on two pilot projects: Sounding Legacy Project (CSU) T-28 Facility Archive (SDSMT) 1974

Protecting the legacy of current and future datasets: the EOL Data Audit What is the data flow (from sensor to archive) of all EOL datasets, including individual instruments and HAIS? Who is responsible for key aspects of DM for these datasets? Do we have backup copies of recently acquired data? Does each dataset have the necessary metadata in a recognized format? Are these metadata sufficient? Are all EOL data secured in a common archive that is linked to other portals? Are the data formats and access methods we use for these datasets today adequate for future communities?

New Programmer/Technician Position One-time funds of.40 FTE allocated in FY13: C. Brooks Snyder, a student assistant in EOL, has accepted the position!

EarthCube earthcube.ning.com

EarthCube: What is it?

“Charette” #1 November 2011, 106 whitepapers received which created a list of common E 3 capabilities (Mike Daniels participated virtually) EarthCube Stakeholder Alignment Survey (sent to EOL Scientists) Early 2012: “Community Groups” formed to gather user requirements “Concept Award Groups” formed to develop prototypes Less formal “Special Interest Groups” address EarthCube-related issues not covered by the existing Community and Concept Groups. Charette #2 June 2012 (KateYoung and Mike Daniels in attendance, other EOL staff participated virtually) Defined roadmaps of next steps for moving groups forward Vignette’s (use cases) from EOL submitted to EarthCube’s Workflow Community Group E^3 moving toward domain-specific workshops and community events Governance Group has plans to be in place by Spring 2013 What has happened thus far

EarthCube Groups and Awards Concept Awards – Develop Prototypes  Brokering  Layered Architecture  Web Services  Earth Systems Modeling  Interoperability Special Interest Groups  Education and Workforce Development  Software Engineering Community of Practice  Web site and Collaboration Environment  Early Career  Data integration for Biology and Earth Science  Physical Samples Cyberinfrastructure  Data Discovery, Access and Mining  Workflow  Governance  Semantics and Ontology Working Groups – Form Requirements

Set out concrete example cases, showcasing how real-time data can be leveragedand addressed within EarthCube.Set out concrete example cases, showcasing how real-time data can be leveraged and addressed within EarthCube. Bring together those in the real-time data community.Bring together those in the real-time data community. Focus on modeling efforts that require data quickly to aid in emergencymanagement, forecasting, decision-making, risk assessment, deployment ofresources, etc.Focus on modeling efforts that require data quickly to aid in emergency management, forecasting, decision-making, risk assessment, deployment of resources, etc. Determine how the needs of real-time data researchers, developers, data providersand stakeholders affect the overall EarthCube system architecture.Determine how the needs of real-time data researchers, developers, data providers and stakeholders affect the overall EarthCube system architecture. Discuss framework architectures and ideas relating to the distributed, online, highlyinteractive and time critical real-time operations.Discuss framework architectures and ideas relating to the distributed, online, highly interactive and time critical real-time operations. Discussions of adaptive sensing networks, un-tethered sensing systems anddistributed real-time observations occurring around the globe.Discussions of adaptive sensing networks, un-tethered sensing systems and distributed real-time observations occurring around the globe. Explore concepts related to real-time quality control and monitoring. PI: M. Daniels, Co-PIs: Dr. V. Chandrasekar (CSU), Dr. S. Graves (University of Alabama), Dr. B. Kerkez (University of Michigan), Dr. F. Vernon (UCSD) Real-time data E 3 workshop

Established to bring NASA groups together as an outcome of a National Academy report in 1998, UNIDATA was a Charter Member Globe, NOAA, NSIDC accepted as members in 2004 but no NCAR(!) I led an organization-wide effort for NCAR to become a member, we were accepted in January 2012 ESIP Vision Pillars Provides the Earth science informatics intellectual commons to drive innovation Is the trusted community authority that supports the integration of science and data into mainstream use Leads the development of the science data information profession Achieves sustainability through diversification, global partnerships and partner recognition Members are recognized as leaders in Informatics at AGU and other venues Represents a forum to allow EOL and NCAR to develop and expand our data services tools and technologies History and Purpose

Organic Growth of Topics Cluster Working Group Standing Committee (10) Very Informal, Monthly Telecons: e.g. Cluster on Climate Forecasting Conventions (6) Fixed Duration, Task Oriented: e.g. Working Group on Data Management Training (4) Formal, Chair is member of ESIP Assembly: e.g. Committee on Data Preservation and Stewardship

Annual Meetings: Winter Business and Summer Technical Semantic Web: Deborah McGuinness Open Source Summit Comparison of EarthCube and ESIP

FUNding Friday! This year, all proposers were winners of a $5K award!

Thank you! NCAREOL Mike Daniels, EOL/CDS