1. IOOS Data Management Activities for Ecosystem Observations
Russ Beard, Ecosystem and Liaison Directorate, National Coastal Data Development Center
23 Aug, 2005
(228)

2. Prelude to EGT/EOP DMA Supporting IOOS
As indicated by the Ecosystem Goal data management assessment chart, PM’s identified numerous challenges in data management. The result of the data management survey are a valuable tool for the DMC as they develop and implement a comprehensive and integrated NOAA data management plan
The plan will need to address the interoperability of legacy operating systems and the implementation must support the IOOS DMAC architecture
Overall, the EGT collects high quality data, provides analysis, and develops high quality products. At the insular level, certain programs have shown good to mixed success in meeting data management requirements.
This is due in part to expanding or new mandates
The current slide reflects the NOAA wide survey results conducted for the 2005 Report to Congress on NOAA’s data management infrastructure. This is a composite chart highlighting the EGT as a whole, and EOP. The colored format illustrates end-to-end DM functions. Green represents meeting 100% of requirement w/ current resource, yellow and red mean incremental increases needed to meet 100% requirement, and substantial resource needed to meet 100% req. respectively. This summary includes IOOS related data management activities as well. The chart doesn’t to justice to a number of success stories within the EGT programs working in IOOS related activities. The next several slides will highlight some of those activities. Integral to understanding the summary chart is the realization that it also reflects the status of DM functions that are affected by legacy systems hindering interoperability with current or emerging technology and expanding or new mandates w/o the resource to meet requirements.
2005 Environmental Data Management Report Assessment Summary*
* Draft Currently in Review

3. Highlights of Current EGT IOOS Activities – Environmental Research Division/SWFSC
Provide consistent, easy to use formats complying with recognized standards (IOOS)
1. One of the successful efforts within EGT as a whole is the work being done at Environmental Research Division at the NMFS SW Fisheries Science Center. Lab products focus on data integration to support ecosystem based management and supports the policy and spirit of the IOOS DMAC Plan guidelines to provide consistent, easy to use formats complying with recognized standards w/in IOOS.
IOOS
Data integration to support ecosystem based management
Fisheries Management, CRMP, Habitat, EOP, ERP, etc

4. Integration of Environmental/Biological Data for Ecosystem Observation and Management
1. The SWFSC efforts at data access, transport and delivery reflect their commitment to IOOS mechanisms, i.e., OPenDAP accessible data servers providing services to their stakeholders through OPenDAP protocol to Matlab, Ferret, and ArcGIS applications, and LAS for web discovery and browsing, providing one stop access to distributed multi-disciplinary datasets. Supports an IOOS compliant Web and OPeNDAP access to distributed multi-disciplinary datasets

5. Coral Reef Information System (CoRIS) Data Management
FGDC metadata are the heart of the Coral Reef Information System. Over 1000 metadata records providing access to over 13K data products. Number has doubled over the last 3 years. CoRIS data discovery tools- keyword search, GIS map search, browse lists. MARC records provide access to over 900 publications.
Data Transport CoRIS links data products maintained on distributed NOAA servers.
Data Archive and Access – CoRIS acts as an agent to ensure that mature data sets are securely stored in the appropriate national data center- those data stored in a national center, CoRIS provides links to the archives to directly access the data.
User outreach- 17 outreach workshops (P.I.’s, Puerto Rico, and the U.S. Virgin Islands) to inform local scientists, students, coral managers and users with CoRIS content and capabilities and to identify opportunities for improved communication of coral data and information. 3 user evaluation workshops Miami and SS to obtain feedback on functionality of the CoRIS Web site and data discovery tools.
CoRIS is a mature example of a NOAA IOOS activity that fully incorporates metadata and data discovery, data transport, data archive and access, user outreach, system engineering, technology refreshment, and the biological data, i.e., the DMAC Appendices (2005)

6. Voluntary Observing Ship Program
EOP DM Support Across Mission Goals – Local Warnings and Forecast Program, National Data Buoy Center
DM System replaces Oracle DB
WWW Browser Client
Models PMO activities- more actively tracks ship visits, met equipment,& access controls facilitating multiple users (one to many ships)
More accurate reporting
Integrates ship observation metadata (formerly separate process)
Integrates data feeds USCG and Lloyds of London
Open source SW components
Requires only a browser client and can be accessed world-wide where www is available
It models the PMO’s activities in workflow, allowing for a more intuitive process in which to manage tasks that occur in the course of tracking ships participation in the program. It more accurately tracks ship visits which are a performance measure for PMO’s
Provides strong access controls to safely facilitate multiple users tracking one to many ships
Models ship status in workflow, providing for a more accurate tracking process and therefore, more accurate reporting
Integrates the tracking of ship observation metadata that was previously managed by a separate system and time consuming to produce
Integrates data feeds from the USCG and LOL providing PMO’s with a comprehensive vessel database- provides a more accurate means of tracking ships cal sign/names
Built using open source software eliminating costly licensing and maintenance
Voluntary Observing Ship Program
Local Warnings and Forecast

7. Data Access Technology
Middleware Technology provides:
- Data download capability from source provider to user’s desktop including data translation (format, units)
- Provides access to a virtual database of coastal data sets
- Data aggregation capability provides transform and transport of data in a single file
Additional Gateways
EO-1 Hyperion Imagery
LOSCO (La Oil Spill Coordinators Office)
GoMOOS
TOWAN (Tactical Oceanography Wide Area Network- Naval Research Lab)
SEAMAP (SE Mapping Assessment Program)
COMPS (Coastal Ocean Monitoring and Prediction System)
Belle Baruch (USC)
NCDDC middleware is a client-server architecture that allows data providers to distribute their data, without having to change the way they store and make their data available to accommodate various customer needs. NCDDC develops a custom interface to existing data providers and has a common data representation model that data is transformed into. This allows the user to access data, do translations and download in common formats without requiring the data provider to make extensive changes to the way they provide data.
NCDDC has a prototype data aggregator running that combines observational data from multiple sources into a single download file, while converting units as required to provide that data in a common format with consistent units.
TOWAN Bathymetry and GDEM (salinity, temps, sound); SEAMAP Fish Trawl Data; MADIS weather observational data, buoy land and air; Belle Baruch CREEK Project's Internal Creek Habitat Survey for Eight Creeks in the North Inlet Estuary, South Carolina: January (comma delimited digital data and spreadsheet) CREEK Project's Microzooplankton Seasonal Monitoring Database for Eight Creeks in the North Inlet Estuary, South Carolina: (comma delimited digital data and Microsoft Excel spreadsheet) CREEK Project's Nekton Database for Eight Creeks in the North Inlet Estuary, South Carolina: (comma delimited digital data and spreadsheet) CREEK Project: RUI: the Role of Oyster Reefs in the Structure and Function of Tidal Creeks. A Project Overview. (NSF proposal) Long-Term Ecological Research (LTER) Macrobenthos Data for the North Inlet Estuary, Georgetown, South Carolina: (comma delimited digital data and Microsoft Excel spreadsheet) Long-Term Low Tide Monitoring Data for Fishes, Shrimps, & Crabs in Oyster Landing Creek, North Inlet Estuary, Georgetown, South Carolina: (comma delimited digital text data and Microsoft Excel spreadsheets. Documentation is in Microsoft Word and text formats.)

8. NMSP West Coast Observation (WCO) Project
Partners: PISCO, SIMoN, NODC, and NCDDC
Observational data collected at 4 of the 5 sanctuary sites located on the west coast.
Project focuses of two data streams (T, and ADCP) collected at 23 instrument moorings
The data Webapp allows a user the option of taking a subset of the original data set by specifying a valid date range, time period, average (hourly or daily), or with the ADCP, instrument depth
Future is to add 5th sanctuary and allow the data access portal to sub-set data across multiple datasets rather than just one.
The internet mapping groups the observational data into two categories- near-real time (MADIS) and long term observations (sanctuary environmental assessment (SEA) station moorings- 3 month period
Automated process by which native ASCII data is converted to NetCDF (COARDS) profile. EML metadata records converted to FGDC and transferred to NODC and posted to multiple search locations.
Data Access Webapp searches the available data on a OPeNDAP server. Create statistics, plots, and download data
CRMP, Habitat, EOP, ERP, Corals, & Fisheries Management

9. OPeNDAP Data Sets Implemented by NODC
Objective is to increase the number of IOOS relevant data sets available via IOOS compatible formats and server technologies on NODC/NCDDC servers
The selection of which data sets to host on NODC/NCDDC Internet sites is driven by:
(1) Expressed need by observing system operators; (2) IOOS ‘data assembly center’ functions not well covered by other programs/offices (3) Critical nature of the respective data sets in documenting long-term trends of key parameters for ecosystem and climate needs.

10. Interagency EOP DM Support – NESDIS MOA with Mississippi DMR
Web and GIS design
MODIS imagery
S gradients correlated with turbidity (250M)
Vibrio model forecast (T, S, turbidity)
Solar radiation, cloud cover, T, S, precip, and obs for fecal coliform die-off rate
ID and classify sediment fallout areas
Vegetative health index
1. Within the EOP, data management activities are being conducted across federal, state, and academic entities on a regional basis- this slide represents a partnership bwt state agencies (MS DMR which acts as conduit to other Mississippi agencies DEQ, PH et al) and federal partners at Stennis Space Center. Funded by NASA, research transition to ops at NRL, distributed and maintained by NOAA for the regional stakeholder. Provides a suite of observational data tailored for and evaluated by the state agencies involved. Blueprint for successful partnership aligning with the goals of the Gulf State Accord.
Partners: NASA ESD, NRL, NOAA, EPA GOMP, and MS DMR
Habitat, Fisheries Management, CRMP, Corals, EOP, & ERP

11. EOP NMFS Hypoxia Website 05 Season
ArcIMS upgrade from FY04
Links to SEAMAP, NMFS, NCDDC, CoastWatch, & RV Oregon II, LUMCON
MODIS 7 Day composite
Extract ship flow-through data, e.g., turbidity, S, & Chl
DO contour maps
1. 4th year supporting NMFS in Hypoxia surveys working in conjunction with Dr Nancy Rabalais at LUMCON
Habitat, ERP, EOP, CRMP, & Fisheries Management

12. EOP DMA for Modeling – Global Navy Coastal Ocean Model (G-NCOM) Dissemination
NCOM Surface temperature and current
NCOM surface current speed and direction
NCOM surface current and elevation
1. Another example of interagency cooperation resulting from a NESDIS MOU with the Naval Oceanographic Office to provide environmental products via NOAA to the general public, and aid in the transition from research to ops
Under evaluation by modelers at UM. Provides boundary conditions for coastal models SE USA and Gulf of Alaska (UW, TAMU, UNC, & WH)
Supports USCG Search and Rescue Ops (SAROP). Surface current fields are input to the Environmental Data Server (EDS) at USCG SAROPS
Fisheries Management, Habitat, Corals, CRMP, ERP, and EOP

13. NESDIS EOP personnel assigned to IOOS Related Data Management Activities
NESDIS IOOS IPA- Director of USM Hydrographic Research Center, Governance Board of the Gulf of Mexico Coastal Ocean Observing System (GCOOS)- Regional Association
DMIT Committee
NOAA Enterprise GIS Working Group
Co-Chair IOOS DMAC Expert Team for Metadata
EPA National Water Quality Monitoring Network Report
(US Ocean Action Plan)
QARTOD
NOAA DMAC Coordinator
Ocean.US DMAC Specialist

14. EOP FY2008 PBA Best Program Alternative – Integrated Ecosystem Data Management
“This proposal recommends that the GOM serve as a regional demonstration for a Large Marine Ecosystem (LME) in which to implement and build off existing integration efforts.” FY08 EOP PBA
Substantial challenge to combine data sets into more consolidated DM architectures, and/or systems, which are interoperable and of a single, consistent format, and standards
Well-developed DM plan for LO within Programs includes an end-to-end process addressing: acquisition of data, quality control, validation, processing, storage, discovery and retrieval, geospatial display, dissemination, the long-term archival of the data, and consistent information data management across programs needs to be addressed. This approach will integrate well with the Gulf State Accords White Paper issues, is an extension of the Stennis interagency work in Mississippi and will be expanded to La and Ala followed by TX and Fl.
Information access & dissemination
S&T IT architecture upgrade and maintenance
Standard Metadata system needs
Regional/National Data Archive System
Technology Refreshment
Data Information System Interoperability

15. EOP IOOS Data Management Challenges
Integrate and implement the IOOS DMAC Plan into a “NOAA Comprehensive Data Management Plan” (2005 Environmental Data Management Report- in draft review)
Regional Associations (IOOS-RA) are still organizing
- Loosely defined
- Funding is inconsistent
- Data management efforts are not focused on national level archival, access, and retrieval
Few centralized IOOS data access points- still individual agency, laboratory, project, and/or investigators
Expand interoperable access of data from client applications (OPenDAP)
Coordinate data management activities within the 9 EOP “Observing Systems/Observations” in the GOM LME (regional approach)
Substantial challenge to combine data sets into more consolidated DM architectures, and/or systems, which are interoperable and of a single, consistent format, and standards
Well-developed DM plan for LO within Programs includes an end-to-end process addressing: acquisition of data, quality control, validation, processing, storage, discovery and retrieval, geospatial display, dissemination, the long-term archival of the data, and consistent information data management across programs needs to be addressed