Caro-COOPS Data and Information Management in Support of Integrated Ocean Observing System Efforts Part I: Where we are Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

What is Caro-COOPS? A wholly-integrated system for coastal observations and their application to user needs An extensive array of instrumented moorings A comprehensive data and information management system An advanced suite of integrated models Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 IOOS DMAC High-priority Recommendations Data suppliers wishing to participate in IOOS immediately should adopt initial DMAC recommendations including (IOOS DMAC, 2003): -Creating FGDC-compliant metadata -Enabling data discovery -Making data available through the OpeNDAP data access protocol -Making data browsable on-line through the LAS -Ensuring long-term archival of contributed data

A comprehensive data management system

A comprehensive data and information management system consisting of three primary components: 1. Reception, management and transfer of primary and ancillary observation data 2. Delivery of QC’d data, metadata and data products to a Web-based data and information portal 3. Provision of user-friendly tools and information products for model output Data and Information Management Subsystem Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Specifically, our system includes: 1. Offshore and shore-based near real-time data transmission via the low-orbit Iridium satellite cellular communications system, the Orbcomm low-orbit satellite communication system, and NOAA’s GOES 2. Data assimilation from observing systems and ancillary data providers (e.g. NDBC) to facilitate data parameterization for modeling and product generation 3. Archival within a relational database system for near real- time display and permanent storage in netCDF format Data and Information Management Subsystem Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Specifically, our system includes (cont.): 4. Access to data via DODS/OPenDAP 5. Access to data and derived products available to the public and research community via a web-based portal 6. Enhanced FGDC metadata development, search and query capabilities 7. Management and visualization (e.g. GIS, LAS) tools allowing users to generate products over the web Data and Information Management Subsystem Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Current activities focus on: - Maintain and enhance techniques for data assimilation processing and product development - Support models with improved input source data - Maintain and enhance QA/QC procedures - Enhance data documentation tools and techniques - Develop products and web access for user communities Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Maintain and enhance techniques for data assimilation processing and product development - IOOS DMAC-compliant but not complacent! - Using relational databases to address diverse dataset types and formats - Working closely with SEA-COOS, CORMP, GoMOOS, SURA, NWLON, NDBC, NCDDC to be cognizant of and accommodate diversity of data types Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Maintain and enhance techniques … (cont.) - Assimilating real-time observations via dataAssimilating telemetry, and using a variety of techniques (e.g. screen scraping, ftp) for ancillary data receipt - Using web functionality and Internet-based GIS tools for data ingest, data browsing, product development, and data and product distribution Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Support models with improved input source data - We recognize the importance of quality input source data for modeling and applications - We understand the limits and implications of our source data and products - Plans to discuss collaboration with interested groups (e.g. USCoE, SCDNR, county governments) for mutually important data development activities Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data issues of spatial and temporal concern

Data and Information Management Subsystem Maintain and enhance QA/QC procedures Relevant QA/QC issues: - On-instrument QA/AC - QA/QC documentation detailing instrument calibration and maintenance procedures - Organizational QA/QC (I.e. the QA/QC link with ancillary data providers) - Regional QA/QC (I.e. data are checked against known or expected environmental range for geographic region) - Feedback QA/QC (I.e. data are checked against immediate past and future readings) Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Maintain and enhance QA/QC procedures (cont.) - Obviously, the quality of our models and products are dependent upon the quality of our data - Goal is to minimize turnaround time for release of QA/QC’d data - Initial QA/QC efforts involve: 1.) Raw data checked for transmission errors 2.) Instrument range checks 3. Time continuity checks Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Enhance data documentation tools and techniques - Importance of metadata development within Caro-COOPS cannot be overstated - Metadata standards include: 1. FGDC 2. EML 3. SensorML - Key operational areas supported by metadata standards include observational data, instrumentation, methods and QA/QC Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Enhance data documentation tools … (cont.) - Caro-COOPS is developing a metadata editing tool to be incorporated with a data upload wizard - Initially this tool will be a web-enabled metadata editor based on the model developed by the SURA- sponsored Cast-Net initiative. Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Develop products and web access for user communities - Multiple user communities (e.g. researchers, managers, fisheries) requiring different products -Website provides access not only to data but also information concerning the observational array, instrumentation, products, news and updates - IOOS-supported data access tools (DODS, LAS) as well as GIS functionality have been implemented Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem

A Hypothetical Cat. 3 hurricane (MCP=960 mb) moving 50 km offshore along the coast (from SW to NE) An advanced suite of integrated models Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Data and Information Management Subsystem

Products with specific audiences in mind (e.g. emergency planners) Data and Information Management Subsystem Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

(Future) potential products with specific audiences in mind (e.g. recreational fishermen; courtesy NOAA website) Data and Information Management Subsystem Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Data and Information Management Subsystem Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Visitors and users of the system are on the increase!

Caro-COOPS Data and Information Management in Support of Integrated Ocean Observing System Efforts Part II: Where we are going Scientific Advisory Board Meeting Columbia, SC16-18 December 2003

Caro-COOPS data and information management efforts provide for … ‘developing in situ and remote sensing techniques for near real-time provision of data on key biological and chemical variables’ (Malone 2003). ‘designing and implementing an enhanced, distributed data and information management system that links all observational and data management systems (across agencies and programs) to all data users’ (Ocean.US 2002). ‘providing an effective and sustained program for enhanced public awareness and education’ (Ocean.US 2003) Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Data and Information Management Subsystem

‘enhancing existing federal networks for in situ measurements … and to improve spatial and temporal resolution … to include physical, chemical and biological variables’ (Ocean.US 2002) ‘adding to the national program of … surveys to establish … multidisciplinary observations (assess the effects of land-based sources of pollution and to improve stock assessments of exploitable fish stocks)’ (Ocean.US 2002). Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Caro-COOPS data and information management efforts provide for … Data and Information Management Subsystem

Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Current and proposed Caro-COOPS data and information management efforts are … - Improving data assimilation, delivery and documentation - Taking information and information products to the user communities - Putting the “I” in IOOS! Data and Information Management Subsystem

Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Improving data assimilation, delivery and documentation - Program enhancements include:  establish true 24/7/365 connectivity to moorings with two-hour routine transmissions and on- demand data dumps  improvements in mechanisms for data delivery from Federal backbone data providers  enhance GIS capabilities focusing on both Internet-based GIS tools and stand-alone GIS applications  develop more rigorous QA/QC processes and evaluate appropriate mix of automated versus human-intervention approaches Data and Information Management Subsystem

Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Taking information and information products to the user communities - Efforts will focus on developing products based on needs identified by specific user communities. Examples include:  working with state and local level emergency preparedness organizations to enhance GIS-based inundation tools  establish data kiosks at public venues including the visitors’ center at Hunting Island, and on the Sunset Pier Data and Information Management Subsystem  Courtesy of

Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Putting the “I” in IOOS! - A major thrust will be the development of Metador, which is:  an enhanced data discovery and documentation tool  focuses on the use of an XML schema (EML) and SensorML while providing for FGDC-compliant metadata  will benefit all IOOS partners, including ocean, terrestrial and estuarine-based systems Data and Information Management Subsystem

Scientific Advisory Board Meeting Columbia, SC16-18 December 2003 Putting the “I” in IOOS! Data and Information Management Subsystem - A second major thrust will be in providing leadership via interoperability projects in designing and demonstrating how local, regional and national systems can successfully integrate Example of an integrated data product developed within the Caro-COOPS, SEA-COOS and SURA/Inter- operability framework