1. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 1 Information Management How SEACOOS IM Fits into the Regional/National System (MF) Managing Distributed Data – A Multi-Faceted Challenge (MF) Data Standards and Data Aggregation (DP) Data Products and Information Visualization (DP) The Website (CE) Partnerships – Leveraging People & Dollars (MF) Madilyn Fletcher, Dwayne Porter, Claire Eager

2. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 2 Includes PI representation from the five primary university participants. Members: USCUNCSKIOUMUSF Madilyn Fletcher (Chair) Harvey Seim Jim Nelson Ed Kearns Mark Luther Dwayne Porter Nick Shay

3. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 3 Goals of Data Management: 1.Coordinate sound data management practices including data assimilation, QA/QC, archival, documentation and availability. 2.Coordinate timely delivery of appropriate data and metadata to modelers, forecasting centers, emergency management personnel, and others. 3.Demonstrate a national model for observing system data management via coordination of a regional capacity for exchange of data, metadata and derived products in accordance with emerging national IOOS standards.

4. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 4 Identification of IM processing & innovations that must be tailored to support other WG milestones and IOOS compliant Delivery Interfaces Data Sources: Obs RS Model output Functions: Aggregation Analysis Visualization Prediction Assimilation Products: Documentation Data User-defined tools & applications Standards QA/QC

5. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 5 SEACOOS Partner Additions of in-situ Buoys/Towers/Coastal/+

6. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 6 Brief Summary of Progress to Date 1.Networking and problem-solving: DMCC continues to deal with ongoing challenges. 2.Standards development. a.Standard vocabulary (data dictionary) established. b.Federal Geographic Data Committee (FGDC) metadata generated. c.Meta-Door implementation 3.Data formatting and transfer processes. a.Initial assessment of DODS/OPeNDAP completed. b.MapServer Internet mapping capabilities utilized for integrated data discovery and display -- additional features, e.g. multiple data layer overlays, good technical c.Both netCDF and Relational Data Base (RDB) DODS servers have been established at each institution. The DODS RDB server has been shown to be technically feasible and is in place for use when needed.

7. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 7 Progress to Date (Continued) 4.Documentation: Web postings on tools and data aggregation efforts & issues. See (http://caro-coops.org/bb/viewtopic.php?t=249), internal Celoxis site, & SURA Wiki.http://caro-coops.org/bb/viewtopic.php?t=249 5.Development of initial Internet mapping interfaces and data workflows for in-situ observations, merged model output, and remote sensing observations. a.OpenIOOS: SEACOOS provided map images of in-situ wind speed/direction and SST via the OpenGIS Consortium (OGC) Web Mapping Service (WMS) protocol. b.All SEACOOS data are OGC-compliant. c.Interface required for merging real-time data with model products completed. 6.Enhancement of Web portals for dissemination of data and metadata. 7. Access to external data streams to support modeling and outreach efforts, e.g. UC Santa Cruz' in-situ SST data, external satellite data from JPL, & external in situ data from NOAA NBDC, NWLON/CO-OPS, and NWS.

8. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 8 SEACOOS Information Management Provides Essential Elements for IOOS and SECOORA Lessons learned and standards/products/protocols developed during evolution of SEACOOS can provide guidance for development of information management systems on a local/regional/national scale. Standards/products/protocols developed by SEACOOS can feed directly into IOOS-scale activities, e.g. OpenIOOS. Communications structure and support evolved by SEACOOS can provide a fundamental element for developing SECOORA network. The information management interactive “culture” is a model for the networking community essential for IOOS and SECOORA.

9. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 9 The SEACOOS “Experience” Identified Important IM Requirements: Development of an effective communications structure and networked problem-solving information management team, which must also interface with data providers and users, and key institutional personnel. Provision of sufficient resources, e.g. personnel, infrastructure, travel funds. Establishment of sound practices, including standards, common protocols, and documentation. Sufficient redundancy and backup. Utilization and testing of multiple technologies. Multiple lines of access to data and derived products. Web publication process that minimizes bottlenecks.

10. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 10 Managing Distributed Data Is A Multi-Faceted Challenge Four primary areas that need continuing attention and improved solutions: –The role of communication Essential Increased complexity as RAs and R-COOSs develop –Maintaining project identity Individual success vs. contribution to the “commons” –Resource investment and allocation Information management = the “glue” Contribution and requirements often underestimated, hence underfunded –Preserving work products and critical infrastructure presents difficult challenges, e.g. archival decisions, redundancy issues.

11. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 11

12. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 12 The Nuts & Bolts Hardware & Software Application server –Apache 2.x –MapServer –Perl, PHP, misc. Database A –In-situ –RS Database B –In-situ –Model output Data scout –In-situ –RS

13. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 13 The Nuts & Bolts System Administration Databases backed up nightly –requires some downtime (~ 2 hours) Databases cleaned up and optimized nightly –significant overhead (~ 4 hours) Some backend data massaging (mainly for model output aggregation) Archives: distributed, centralized, and/or with federal data repositories System redundancy (including personnel) and documentation– critical!

14. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 14 Data Standards and Data Aggregation Metadata development – Master FGDC metadata record – Contributing projects responsible for individual metadata development – Meta-Door information management and documentation tool under development in conjunction with SEACOOS QA/QC is the responsibility of contributing projects but adheres to consistent and national standards when feasible. Data dictionary under development

15. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 15 Data Standards and Data Aggregation Aggregation formats –netCDF (in situ and remotely-sensed data) –PNG (RS imagery) Storage –Relational database (in situ, model output, some RS) –Files (RS imagery) Technology –Perl –PostgreSQL & PostGIS

16. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 16 Normalization issues include: Reporting time varies –In-situ data E.g. daily, hourly, half-hourly, every 10 minutes –Remotely-sensed data E.g. twice daily Reporting area varies –Remotely-sensed data passes Round-the-clock updating is resource intensive –Balance the server load Data Standards and Data Aggregation

17. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 17 OPeNDAP (DODS) data access OGC-friendly: WMS, WFS –pick a layer, any layer http://nautilus.baruch.sc.edu/seacoos_misc/show_sea_coos_obs_time_ranges.phppick a layer, any layer Data and Products Visualization and Dissemination

18. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 18 Data and Products Visualization and Dissemination Ad-hoc time-series graphsAd-hoc animations

19. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 19 Data and Products Visualization and Dissemination Production site (http://www.seacoos.org)

20. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 20 Development site (http://nautilus.baruch.sc.edu/rs) Main engines –PHP PHP-MapScript –Perl –MapServer Data and Products Visualization and Dissemination

21. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 21 Integrated Ocean Observing System hurricane demoIntegrated Ocean Observing System hurricane demo (http://openioos.org) Data and Products Visualization and Dissemination

22. Southeast Atlantic Coastal Ocean Observing System Program Evaluation Jan 17-19, 2005 Information Management Workgroup 22 Partnerships: Leveraging People and Dollars Leveraging Examples: All partners provide in-kind effort through participation of local personnel and resources, particularly computing infrastructure. NOAA Grant. Carolinas Coastal Ocean Observing System (Caro-COOPS). 2002-2004: $4,995,200. The NSF EPSCOR project, “A Regional Framework for Interconnectivity of Coastal Ocean Observing Systems” laid a foundation for some of the metadata management and documentation required for SEACOOS IM. 2000-2003: $729,461. NOAA Grant. “Integrating the NERRS System-wide Monitoring Program with Ocean.US Integrated Ocean Observing System Efforts”. 2004-2006: $124,960. Research personnel and infrastructure at USC provide enhancements for observing system activities, although these are not separately budgeted nor accounted for. We estimate that these activities are equivalent to ~$60,000/year. Identified Primary Super-Users – Some Data Contributions NDBC – NC-COOS, COMPS, SABSOON, Explorer of the Sea, Caro-COOPS CO-OPS – COMPS, Caro-COOPS NWS – Explorer of the Sea, COMPS, Caro-COOPS