Part I DataFed An Agile Distributed Air Quality Data System Rudolf B. Husar Washington University, St. Louis Seminar Presented at University of Alabama, Dept. of Atmospheric Sciences December 6, 2006

Revolution in Sensing (1990s) & Delivery (2000+) Real-time Air Pollution Sensing and Reporting High Resolution Satellite DataSurface PM25 and Ozone Data Smoke Plumes

Changes in Air Quality Management Command & Control Weight of Evidence Flexible NAAMS Rigid Monitoring

The data life cycle consists of the acquisition and the usage parts Usage ActivitiesData Acquisition Data Acquisition and Usage Activities (Select View Show, click to step through PPT) The acquisition part processes the sensory data by firmly linked procedures The focus is on data usage activities The usage activities are more iterative, dynamic procedures The collected and cleaned data are stored in the repository Data Repository The usage cycle transform data into knowledge for decision making Decisions

Stages of AQ Data Flow and Value-Adding Processes Domain ProcessingData Sharing Std. Interface Gen. Processing Std. Interface Data Control Reports Reporting Obs. & ModelsDecision Support System Analyzing Filter/Integrate Aggregate/Fuse Custom Analysis Organizing Document Structure/Format Interfacing Characterizing Display/Browse Compare/Fuse Characterize Value-Adding Processes Reporting Inclusiveness Iterative/Agile Dynamic Report

Generic Decision Support for Air Quality Decisions Global Earth Observing System of Systems GEOSS Architecture Framework Knowledge into the Minds of Regulatory Analysts Knowledge into the Minds of Technical Analysts Observations Reports: Model Forecasts, Obs. Evidence Models Decisions Knowledge into the Minds of Decision- making managers Decision Support System

Characteristics of System of Systems (SoS) Autonomous constituents managed/operated independently Independent evolution of each constituent Displays emergent behavior Must recognize, manage, exploit the characteristics: No stakeholder has complete SoS insight Central control is limited; distributed control is essential Users, must be involved throughout the life of a SoS

Differentiating System Engineering and System of Systems Engineering “System of Systems Engineering for Air Force Capability Development”, 2005

Air Quality Cluster Air Quality Cluster TechTrack Earth Science Information Partners Partners NASA NOAA EPA (?) USGS DOE NSF Industry… Air Quality Information System Architecture Flow of Data Flow of Control Air Quality Data Meteorology Data Emissions Data Informing Public AQ Compliance Status and Trends Network Assess. Tracking Progress Data to Knowledge Transformation Data ProductsKnowledge ProductsMediators

Loosely Coupled Data Access through Standard Protocols The next three slides describe the key technologies used in the creation of an adaptable and responsive air quality information system. OGC data access protocols and standard formats facilitate loose coupling between data on the internet and processing services. For air quality, the Web Coverage Service (WCS), provides a universal simple query language for requesting data as where, when, what. That is: geographic (3D bounding box), time range and parameter. The Web Map Service (WMS) and Web Feature Service (WFS) are also useful. The use of standard data physical data formats and naming conventions elevates the syntactic and semantic interoperability. Within DataFed all data access services are implemented as WCS or WMS and optionally WFS. General format adapter components permit data request in a variety of standard formats. GetCapabilities GetData Capabilities, ‘Profile’ Data Where? When? What? Which Format? Server Back End Std. Interface Client Front End Std. Interface QueryGetData Standards Where?BBOXOGC, ISO When?TimeOGC, ISO What?TemperatureCF FormatnetCDF, HDF..CF, EOS, OGC T2T1 Domain ProcessingData Sharing Std. Interface Gen. Processing Std. Interface Data Control Reports Reporting Obs. & ModelsDecision Support System

Web Services and Workflow for Loose Coupling Service Broker Service Provider Publish Find Bind Service User Web Service Interaction Service Chaining & Workflow Domain ProcessingData Sharing Std. Interface Gen. Processing Std. Interface Data Control Reports Reporting Obs. & ModelsDecision Support System Web Services Triad: Publish – Find – Bind Workflow Software: Dynamic Programming

Collaborative Reporting and Dynamic Delivery Co Writing - Wiki ScreenCast Analysis Reports: Information supplied by many Needs continuous program feedback Report needs many authors Wiki technologies are for collaborative writing Dynamic Delivery: Much of the content is dynamic Animated presentations are compelling Movies and screencasts are for dynamic delivery Domain ProcessingData Sharing Std. Interface Gen. Processing Std. Interface Data Control Reports Reporting Obs. & ModelsDecision Support System

Providers NASA DAACs EPA R&D Model EPA AIRNow others Public Manager Scientist Users other The info system transforms the data into info products for each user In the first stage the heterogeneous data are prepared for uniform access Uniform Access Agile Information System: Data Access, Processing and Products The second stage performs filtering, aggregation, fusion and other operations Data Processing Web Service Chain Custom Processing SciFlo DataFed Info Products Reports, Websites Forecasting Compliance Other Sci. Reports The third stage prepares and delivers the needed info products

Approach: Mediation Between Users and Data Providers DataFed assumes spontaneous, autonomous data providers Non-intrusively wraps datasets for access by web services Mediates, homogenizes data views. e.g. geo-spatial, time...

Scientist Science DAACs Current info systems are project/program oriented and provide end-to-end solutions Info UsersData ProvidersInfo System AIRNow Public AIRNow Model Compliance Manager ‘Stovepipe’ and Federated Usage Architectures Landscape Part of the data resources of any project can be shared for re-use through DataFed Through the Federation, the data are homogenized into multi-dimensional cubes Data processing and rendering can then be performed through web services Each project/program can be augmented by Federation data and services

DataFed User Tools –Data Catalog –Data Browser –PlumeSim, Animator –Combined Aerosol Trajectory Tool (CATT) Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets

Federated Datasets Data are accessed from autonomous, distributed providers DataFed ‘wrappers’ provide uniform geo-time referencing Tools allow space/time overlay, comparisons and fusion Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOWO3, PM25 ASOS_STIVisibility, 300 sites METARVisibility, 1200 sites VIEWS_OL40+ Aerosol Parameters Satellite MODIS_AOTAOT, Idea Project GASPReflectance, AOT TOMSAbsorption Indx, Refl. SEAW_USReflectance, AOT Model Output NAAPSDust, Smoke, Sulfate, AOT WRFSulfate Fire Data HMS_FireFire Pixels MODIS_FireFire Pixels Surface Meteorology RADARNEXTRAD SURF_METTemp, Dewp, Humidity… SURF_WINDWind vectors ATADTrajectory, VIEWS locs.

A Sample of Datasets Accessible through ESIP Mediation Near Real Time (~ day) It has been demonstrated (project FASTNET) that these and other datasets can be accessed, repackaged and delivered by AIRNow through ‘Consoles’ MODIS Reflectance MODIS AOT TOMS Index GOES AOT GOES 1km Reflec NEXTRAD Radar MODIS Fire Pix NRL MODEL NWS Surf Wind, Bext

Web Services: Building Blocks of DataFed Programming Access, Process, Render Data by Service Chaining NASA SeaWiFS Satellite NOAA ATAD Trajectory OGC Map Boundary RPO VIEWS Chemistry Data Access Data Processing Layer Overlay LAYERS Web Service Composition

Interoperability Wrappers and Adapters