Open Solutions to Regional Observing Systems

Outline Recent near real-time in-situ observations are aggregated to a ‘Xenia’ schema relational database(RDB) via its import XML format ‘ObsKML’ schema. From this database of recent observations a variety of file formats, web services and applications can be driven. By suggesting a minimally common observation oriented XML and RDB schema, developed scripts and products can benefit from and build around these shared schema. For more info contact Keywords Scripting languages Perl PHP Relational Database Sqlite Postgres Mapping Engine/Interface Mapserver OpenLayers TileCache OpenGeospatial Consortium(OGC) services WMS/WFS/SOS Google Earth/Maps KML/KMZ Google Code/Analytics

Problems Xenia intended to address Grants for research instrumentation which will be collecting observation data while lacking a data management/sharing component beyond basic file storage Low-volume data(< 100,000 records per hour) in-situ observational platforms or system arrays (e.g. 1 to 1000 platforms collecting observations per hour) collecting data at any geographic scale (local,regional,national,etc) Bridging the gap between raw data collection and the organization and sharing of data using previously developed products, services and standards(leveraging earlier work against new data providers) Fostering a standardization of products and services via a common openly shared technical infrastructure(common database schema and product support scripts)

Problems Xenia not intended to address High-volume data (millions of records per hour) such as gridded model outputs, hf radar, etc. High-volume data problems at this time are better addressed using traditional file processing techniques where data management can suggest output file formats(such as images, shapefiles, etc) and metadata that are conducive to search and usage needs.

Xenia Relational Database Schema Hierarchy = organization->platform->sensor->multi_obs(observation data)

ObsKML XML schema for import/export

Data flows – general

Data flows – ObsKML ObsKML (Observations KML) A simple XML encoding of observation metadata associated with a KML Placemark. Default XML import/export format for Xenia database Instances. Postgres or Sqlite database paths

Xenia data organized by time interval Latest past several hours New Data Recent 0-6 weeks Archival 3+ weeks to 1-2 years Possibly table separated by year,month,etc Archival file 1-2+ years Files separated by product/year/month Julian weekly divided database files(sqlite) of all observations are available

Xenia aggregation, replication, redundancy Xenia A,B,C,D,E,F Xenia A,B,C Xenia D,E,F Xenia A Xenia B Xenia C Xenia D Xenia E Xenia F Xenia Backup A,B,C,D,E,F Xenia Backup D,E,F Overlapping/redundant systems could aggregate/replicate via SQL or XML

Flow/event monitoring Around 10,000 observations are collected each hour from a variety of data providers. The data flows are monitored for both successful import and towards provided report metrics or notifications when problems or events occur. Sensor count graph showing the hourly number of successful sensor imports per provider More detailed platform/observation table showing daily counts color coded to highlight specific sensor issues

Applications – National Weather Service Latest regional platform observations from Xenia database are styled to html tables and presented in fixed map interface alongside NWS forecast warnings/advisories and other map layers of interest

Applications – SECOORA interactive map Latest regional observation maps are accessed via WMS(Web Mapping Service) and merged into an OpenLayers(browser javascript) map interface. Observation database requests are supported via WMS GFI(Get Feature Info)

Applications - CarolinasRCOOS Map image layers supported via MapServer WMS with increased map response time via TileCache which tiles and caches image requests for repeated reference. Site feature info supported via Xenia database instance styled output html table accessed via WMS GFI.

KML/KMZ – Latest obs by platform KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer Latest observation data organized by all observations per platform

KML/KMZ – latest obs by obsType KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer Latest observation data organized by observation type for all platforms All observations carry * observation type(obsType) * unit of measure(uom) * measurement value obsType+uom = measurement type(m_type) Color styled low to high value blue/green/red

Development/documentation support Google Code provides very simple, free browser based access to the basic project support tools such as a subversion(svn) code/documentation sharing/versioning, wiki documentation and issue tickets. Google Analytics provides free browser based access to website usage statistics via a javascript project token included in webpage access.

Lessons Learned Web GIS interface: php/mapscript -> openlayers/javascript/ajax, tilecache Latest in-situ obs: RDB – sensor/obs hard-coded/table approach -> obs non-specific infrastructure(xenia,obskml) Sensor/observation specific metadata: loosely link a variety of more specific metadata files/schemas to common infrastructure Hourly gridded data: RDB quickscat, model layers -> ogr/sql+shapefiles Full RDB to file-based RDB: postgresql+postgis -> sqlite+ogr, sqlitegeo? Self hosted to third party hosted: third party website hosting for static content/context, common processing/storage. Google code/analytics for basic shared development support/documentation. ROA(Resource Oriented Architecture)/REST style simpler declarative alternatives to complex query SOA(Service Oriented Architecture)

File formats and web services File formats (CSV,shapefile,netCDF,KML/KMZ,Sqlite DB) Free desktop analysis tools MapWindows(shapefile/GIS) ncBrowse(netCDF) ODV(ctd/csv) Sqlite DB Web services (OGC WMS/WFS/SOS)

Credits 2 C.Calloway, 2 J.Cleary, 1 J.Cothran, 4 J.Donovan, 3 J.Dorton, 1 M.Fletcher,C.Galvarino, 1 S.King, 2 S.Haines, 3 L.Leonard, 1 D.Porter, 3 X.Qi, 1 D.Ramage, 2 H.Seim, 4 V.Subramanian, 1 S.Walker, 4 R.Weisberg 1 University of South Carolina, 2 University of North Carolina at Chapel Hill, 3 University of North Carolina at Wilmington, 4 University of South Florida 2 C.Calloway, 2 J.Cleary, 1 J.Cothran, 4 J.Donovan, 3 J.Dorton, 1 M.Fletcher,C.Galvarino, 1 S.King, 2 S.Haines, 3 L.Leonard, 1 D.Porter, 3 X.Qi, 1 D.Ramage, 2 H.Seim, 4 V.Subramanian, 1 S.Walker, 4 R.Weisberg 1 University of South Carolina, 2 University of North Carolina at Chapel Hill, 3 University of North Carolina at Wilmington, 4 University of South Florida