Integration through Data Management The “SEACOOS experience” Accomplishments and Obstacles SEACOOS approach to data integration Accomplishments and contributions to IOOS interoperability Some “lessons learned” Coordination with nascent Regional Association

SEACOOS was initiated in 2002 with ONR funding to develop a coastal ocean information system for FL, GA, SC and NC. SEACOOS Over-Arching Goal: To significantly increase the quantity and quality of environmental information from the coastal ocean of the SE U.S. and facilitate its use in a wide range of societal, scientific, and educational applications.

Observational Platforms Contributed by SEACOOS Partners Observational Platforms Contributed by SEACOOS Partners

Real-Time Observations: SE Coastal Ocean

SEACOOS Modeling Coordinating model simulations for the entire region

SEACOOS Information Management Data Management Coordinating Committee (DMCC) Coordination Madilyn Fletcher (USC), Dwayne Porter (USC), P.I. representatives Ed Kearns (U Miami), Mark Luther (USF), Harvey Seim (UNC), Nick Shay (U Miami), Jim Nelson (SkIO) People who get it done Charlton Purvis (USC), Jeremy Cothran (USC), Vembu Subramanian (USF), Jeff Donovan (USF), Sara Haines (UNC), Jesse Cleary (UNC), Liz Williams (U Miami), Tom Cook (U Miami) Web site documentation /portal Chris Calloway (UNC), Claire Eager (UNC)

SEACOOS achievements in regional Data Management “Data commons” -- Protocols for data providers to make their information available. A set of standards for use with a given file format Methods for data aggregation and display Methods for data aggregation and display Together – a demonstration of how an RCOOS can function to aggregate and display information

Data “commons” netCDF adopted as the common data language (“SEACOOS CDL”, currently version 2) -- well supported (UCAR, Unidata, Boulder, CO) -- supported under DODS/OPeNDAP -- interfaces to programming and analysis packages (e.g., Perl, Matlab) -- a flexible system for a variety of platforms/sensor types (scalar time series, vector time series, vector profiler time series, scalar gridded maps, vector unstructured maps)

Data “commons” (cont.) “Data Dictionary” -- table that registers known standards with each other -- provides an English language description -- “down-the-road” advantages: options for data providers; flexibility in presentation and interpretation of information (i.e., if format is registered, the providers’ data can be represented)

Data Management: the Nuts & Bolts Data is aggregated and stored Data is aggregated and stored Data is normalized Data is normalized Data is visualized Data is visualized Data is disseminated Data is disseminated

Data Aggregation & Storage Aggregation format flavors Aggregation format flavors netCDF (in situ & RS data; model output) netCDF (in situ & RS data; model output) PNG (RS imagery) PNG (RS imagery) Storage Storage Relational database (in situ, model output, some RS) Relational database (in situ, model output, some RS) Files (RS imagery) Files (RS imagery) Technology Technology Perl Perl PostgreSQL & PostGIS PostgreSQL & PostGIS

SEACOOS Data Train A central aggregation site as opposed to distributed network (e.g., LAS) Link to software community developing GIS-type applications Powerful visualization tool without limits to numbers of layers

Normalization Reporting time varies Reporting time varies In-situ data In-situ data E.g. daily, hourly, half-hourly, every 10 minutes E.g. daily, hourly, half-hourly, every 10 minutes Remotely-sensed data Remotely-sensed data E.g. twice daily E.g. twice daily Reporting area varies Reporting area varies Remotely-sensed data passes Remotely-sensed data passes Round-the-clock updating is resource intensive Round-the-clock updating is resource intensive Balance the server load Balance the server load

Visualization : Example 1 Production site Production site Production site Production site

Visualization : Example 2 Development site Development site Development site Development site Main engines Main engines PHP PHP-MapScript PHP PHP-MapScript Perl Perl MapServer MapServer

Visualization Fun : Graphs & Animations Ad-hoc time-series graphs Ad-hoc time-series graphs Ad-hoc animations Ad-hoc animations Ad-hoc animations Ad-hoc animations

Hurricane/Tropical Storm Jeanne September, 2004 Merged wind observations and shore radar

Dissemination OPeNDAP (DODS) data access OPeNDAP (DODS) data access OGC-friendly: WMS, WFS OGC-friendly: WMS, WFS pick a layer, any layer pick a layer, any layer pick a layer, any layer pick a layer, any layer

Dissemination : example 1 NC OneMap Viewer website/NC_OneMap/ viewer.asp NC OneMap Viewer website/NC_OneMap/ viewer.asp NC OneMap Viewer NC OneMap Viewer

SEACOOS Remotely-Sensed Layers WMS-Enabled for Hurricane Charley (see OpenIOOS web site -- * modis_ergb_low (USF) * modis_rgb_low (USF) * modis_rgb_hi_chesapeake_bay (USF) * modis_rgb_hi_florida_bay (USF) * modis_rgb_hi_miss_plume (USF) * modis_rgb_hi_suwannee_river (USF) * modis_rgb_hi_tampa_bay (USF) * oi_sst (USF) * modis_sst (USF) * avhrr_sst (USF) * modis_chl (USF) * quikscat_wind (JPL) * met_wind (UNC) * met_air_pressure_contour (UNC) * tamu_level_III (TAMU Mesonet) Any questions, please contact Charlton Purvis,

Dissemination : example 2 Integrated Ocean Observing System hurricane demo Integrated Ocean Observing System hurricane demo Integrated Ocean Observing System hurricane demo Integrated Ocean Observing System hurricane demo

Some Lessons : Interoperability Demonstrations Communication is key : an open source attitude Mailing lists, bulletin boards, and wiki’s proved invaluable across internal SEACOOS projects and external projects, e.g. Interoperability Demonstrations Successfully accomplished (capabilities were demonstrated), but had “fire-drill” aspects from perspective of technical personnel Better coordinated in Summer 2004 than previously, a more mature effort Better adaptation of OGC WMS and WFS technologies Additional layers A better product “under the hood” However, the push to get out products in the short-term was done at the expense of thoroughness Process to add new products, identify sources seen as somewhat haphazard Not the way to build an operational system

Other Obstacles / Issues QA/QC of real-time data and archives Needs further development; compliant with national standards Resource availability and allocation Many personnel have multiple roles Crucial expertise in a small number of people Time for documentation Need for redundancy in comm./processing streams Data from national providers not always in a readily accessible form (often requires “screen scrapes”) Engaging users / getting feedback “In-reach” within the regional program as well as with external users Track users and map resultsmap results

Other SEACOOS regional DM activities Coordination with NDBC, NWS e.g., data push from COMPS, SABSOON Stations part of NDBC network Coordination with developing Regional Association (SECOORA) Sensor/equipment inventory Data dictionary (documentation) Data dictionarydocumentation Development of metadata tool/metadata system for the RCOOS (“MetaDoor” & documentation, USC)MetaDoordocumentation Coordination with State Agencies

Equipment Inventory Full Search Query Returns: DB records Query Returns: Map User Interface Components Quick Search – by variable measured

Equipment Inventory Administrative Interface 1.Institution: Institution Name, Abbreviation, Observing Group URL, Affiliation 2.Contacts: Contact Name, Phone Number, Address, Contact level 3.Stations: Institutional Station ID, Station Name, Lat/Long 4.Equipment: Package Description, Manufacturer, Model, Equipment Type, Vertical Position, Power Requirement, Communication Type, Near Real-time Status, Variables Measured, Change/Calibration History, Active Status, Comments This interface provides database access to equipment managers to add and edit the following information.

Data input and involvement from federal data providers for a more accurate picture of the regional observation system (assets, gaps, performance) Data input and involvement from federal data providers for a more accurate picture of the regional observation system (assets, gaps, performance) Connecting equipment data as metadata to support observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool developed at USC). Connecting equipment data as metadata to support observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool developed at USC). Improved data intake capability: automated batch processing for new equipment Improved data intake capability: automated batch processing for new equipment Expanded interactive map functionality: Zoom/Pan Expanded interactive map functionality: Zoom/Pan Equipment inventory Next Steps

Metadoor : metadata creation

Metadoor : entry context features - Collapsible forms organized into tabbed sections - Fields color coded by requirement need -Help context link for each field -Listbox selections and calendars - Form presents only subfields needed based on listbox selection

Metadoor : planned metadata support for marineXML marineXML sensorML sensorML Platforms and Sensors Platforms and Sensors Observations and Measurements Observations and Measurements OGC Sensor Web OGC Sensor Web Other metadata harvesters Other metadata harvesters OGC Catalog services OGC Catalog services geodata.gov geodata.gov

A friend in the business Development site: Development site: Production site: Production site: Very active listserv dealing with mainly remote-sensing issues: Very active listserv dealing with mainly remote-sensing issues: mailto: mailto: message text: subscribe remotesensing message text: subscribe remotesensing Who am I? Who am I? Charlton Purvis, University of South Carolina, SEACOOS Charlton Purvis, University of South Carolina, SEACOOS Happy to help and share. Happy to help and share.