1. Integration through Data Management The “SEACOOS experience” Accomplishments and Obstacles SEACOOS approach to data integration Accomplishments and contributions to IOOS interoperability Some “lessons learned” Coordination with nascent Regional Association

2. SEACOOS was initiated in 2002 with ONR funding to develop a coastal ocean information system for FL, GA, SC and NC. SEACOOS Over-Arching Goal: To significantly increase the quantity and quality of environmental information from the coastal ocean of the SE U.S. and facilitate its use in a wide range of societal, scientific, and educational applications.

3. Observational Platforms Contributed by SEACOOS Partners Observational Platforms Contributed by SEACOOS Partners

4. Real-Time Observations: SE Coastal Ocean

5. SEACOOS Modeling Coordinating model simulations for the entire region

6. SEACOOS Information Management Data Management Coordinating Committee (DMCC) Coordination Madilyn Fletcher (USC), Dwayne Porter (USC), P.I. representatives Ed Kearns (U Miami), Mark Luther (USF), Harvey Seim (UNC), Nick Shay (U Miami), Jim Nelson (SkIO) People who get it done Charlton Purvis (USC), Jeremy Cothran (USC), Vembu Subramanian (USF), Jeff Donovan (USF), Sara Haines (UNC), Jesse Cleary (UNC), Liz Williams (U Miami), Tom Cook (U Miami) Web site documentation /portal Chris Calloway (UNC), Claire Eager (UNC)

8. SEACOOS achievements in regional Data Management “Data commons” -- Protocols for data providers to make their information available. A set of standards for use with a given file format Methods for data aggregation and display Methods for data aggregation and display Together – a demonstration of how an RCOOS can function to aggregate and display information

9. Data “commons” netCDF adopted as the common data language (“SEACOOS CDL”, currently version 2) -- well supported (UCAR, Unidata, Boulder, CO) -- supported under DODS/OPeNDAP -- interfaces to programming and analysis packages (e.g., Perl, Matlab) -- a flexible system for a variety of platforms/sensor types (scalar time series, vector time series, vector profiler time series, scalar gridded maps, vector unstructured maps)

10. Data “commons” (cont.) “Data Dictionary” -- table that registers known standards with each other -- provides an English language description -- “down-the-road” advantages: options for data providers; flexibility in presentation and interpretation of information (i.e., if format is registered, the providers’ data can be represented)

11. Data Management: the Nuts & Bolts Data is aggregated and stored Data is aggregated and stored Data is normalized Data is normalized Data is visualized Data is visualized Data is disseminated Data is disseminated

12. Data Aggregation & Storage Aggregation format flavors Aggregation format flavors netCDF (in situ & RS data; model output) netCDF (in situ & RS data; model output) PNG (RS imagery) PNG (RS imagery) Storage Storage Relational database (in situ, model output, some RS) Relational database (in situ, model output, some RS) Files (RS imagery) Files (RS imagery) Technology Technology Perl Perl PostgreSQL & PostGIS PostgreSQL & PostGIS

13. SEACOOS Data Train A central aggregation site as opposed to distributed network (e.g., LAS) Link to software community developing GIS-type applications Powerful visualization tool without limits to numbers of layers

14. Normalization Reporting time varies Reporting time varies In-situ data In-situ data E.g. daily, hourly, half-hourly, every 10 minutes E.g. daily, hourly, half-hourly, every 10 minutes Remotely-sensed data Remotely-sensed data E.g. twice daily E.g. twice daily Reporting area varies Reporting area varies Remotely-sensed data passes Remotely-sensed data passes Round-the-clock updating is resource intensive Round-the-clock updating is resource intensive Balance the server load Balance the server load

15. Visualization : Example 1 Production site http://www.seacoos.org Production site http://www.seacoos.org Production site Production site

16. Visualization : Example 2 Development site http://nautilus.baruch.sc.edu/rs Development site http://nautilus.baruch.sc.edu/rs Development site Development site Main engines Main engines PHP PHP-MapScript PHP PHP-MapScript Perl Perl MapServer MapServer

17. Visualization Fun : Graphs & Animations Ad-hoc time-series graphs Ad-hoc time-series graphs Ad-hoc animations Ad-hoc animations Ad-hoc animations Ad-hoc animations

18. Hurricane/Tropical Storm Jeanne 25 - 27 September, 2004 Merged wind observations and shore radar

19. Dissemination OPeNDAP (DODS) data access OPeNDAP (DODS) data access OGC-friendly: WMS, WFS OGC-friendly: WMS, WFS pick a layer, any layer http://nautilus.baruch.sc.edu/seacoos_misc/show_sea_coos_obs_time_ranges.php pick a layer, any layer http://nautilus.baruch.sc.edu/seacoos_misc/show_sea_coos_obs_time_ranges.php pick a layer, any layer pick a layer, any layer

20. Dissemination : example 1 NC OneMap Viewer http://gisdata.usgs.net/ website/NC_OneMap/ viewer.asp NC OneMap Viewer http://gisdata.usgs.net/ website/NC_OneMap/ viewer.asp NC OneMap Viewer NC OneMap Viewer

21. SEACOOS Remotely-Sensed Layers WMS-Enabled for Hurricane Charley (see OpenIOOS web site -- http://twiki.sura.org/bin/view/Main/CharleyUpgrade) * modis_ergb_low (USF) * modis_rgb_low (USF) * modis_rgb_hi_chesapeake_bay (USF) * modis_rgb_hi_florida_bay (USF) * modis_rgb_hi_miss_plume (USF) * modis_rgb_hi_suwannee_river (USF) * modis_rgb_hi_tampa_bay (USF) * oi_sst (USF) * modis_sst (USF) * avhrr_sst (USF) * modis_chl (USF) * quikscat_wind (JPL) * met_wind (UNC) * met_air_pressure_contour (UNC) * tamu_level_III (TAMU Mesonet) Any questions, please contact Charlton Purvis, cpurvis@sc.edu.

22. Dissemination : example 2 Integrated Ocean Observing System hurricane demo http://www.openioos.org Integrated Ocean Observing System hurricane demo http://www.openioos.org Integrated Ocean Observing System hurricane demo Integrated Ocean Observing System hurricane demo

23. Some Lessons : Interoperability Demonstrations Communication is key : an open source attitude Mailing lists, bulletin boards, and wiki’s proved invaluable across internal SEACOOS projects and external projects, e.g. Interoperability Demonstrations Successfully accomplished (capabilities were demonstrated), but had “fire-drill” aspects from perspective of technical personnel Better coordinated in Summer 2004 than previously, a more mature effort Better adaptation of OGC WMS and WFS technologies Additional layers A better product “under the hood” However, the push to get out products in the short-term was done at the expense of thoroughness Process to add new products, identify sources seen as somewhat haphazard Not the way to build an operational system

24. Other Obstacles / Issues QA/QC of real-time data and archives Needs further development; compliant with national standards Resource availability and allocation Many personnel have multiple roles Crucial expertise in a small number of people Time for documentation Need for redundancy in comm./processing streams Data from national providers not always in a readily accessible form (often requires “screen scrapes”) Engaging users / getting feedback “In-reach” within the regional program as well as with external users Track users and map resultsmap results

25. Other SEACOOS regional DM activities Coordination with NDBC, NWS e.g., data push from COMPS, SABSOON Stations part of NDBC network Coordination with developing Regional Association (SECOORA) Sensor/equipment inventory Data dictionary (documentation) Data dictionarydocumentation Development of metadata tool/metadata system for the RCOOS (“MetaDoor” & documentation, USC)MetaDoordocumentation Coordination with State Agencies

26. Equipment Inventory Full Search Query Returns: DB records Query Returns: Map User Interface Components Quick Search – by variable measured

27. Equipment Inventory Administrative Interface 1.Institution: Institution Name, Abbreviation, Observing Group URL, Affiliation 2.Contacts: Contact Name, Phone Number, Email Address, Contact level 3.Stations: Institutional Station ID, Station Name, Lat/Long 4.Equipment: Package Description, Manufacturer, Model, Equipment Type, Vertical Position, Power Requirement, Communication Type, Near Real-time Status, Variables Measured, Change/Calibration History, Active Status, Comments This interface provides database access to equipment managers to add and edit the following information.

28. Data input and involvement from federal data providers for a more accurate picture of the regional observation system (assets, gaps, performance) Data input and involvement from federal data providers for a more accurate picture of the regional observation system (assets, gaps, performance) Connecting equipment data as metadata to support observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool developed at USC). Connecting equipment data as metadata to support observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool developed at USC). Improved data intake capability: automated batch processing for new equipment Improved data intake capability: automated batch processing for new equipment Expanded interactive map functionality: Zoom/Pan Expanded interactive map functionality: Zoom/Pan Equipment inventory Next Steps

29. Metadoor : metadata creation

30. Metadoor : entry context features - Collapsible forms organized into tabbed sections - Fields color coded by requirement need -Help context link for each field -Listbox selections and calendars - Form presents only subfields needed based on listbox selection

31. Metadoor : planned metadata support for marineXML marineXML sensorML sensorML Platforms and Sensors Platforms and Sensors Observations and Measurements Observations and Measurements OGC Sensor Web OGC Sensor Web Other metadata harvesters Other metadata harvesters OGC Catalog services OGC Catalog services geodata.gov geodata.gov

32. A friend in the business Development site: http://nautilus.baruch.sc.edu/rs Development site: http://nautilus.baruch.sc.edu/rs Production site: http://www.seacoos.org Production site: http://www.seacoos.org Very active listserv dealing with mainly remote-sensing issues: remotesensing@caro-coops.org Very active listserv dealing with mainly remote-sensing issues: remotesensing@caro-coops.org mailto: majordomo@caro-coops.org mailto: majordomo@caro-coops.org message text: subscribe remotesensing message text: subscribe remotesensing Who am I? Who am I? Charlton Purvis, University of South Carolina, SEACOOS Charlton Purvis, University of South Carolina, SEACOOS cpurvis@sc.edu cpurvis@sc.edu Happy to help and share. Happy to help and share.