Federated PM and Haze Data Warehouse Project a sub- project of (enter your sticker & logo here ) Nov 20, 2001, RBH St. Louis Midwest Supersite Project Regional Planning Organization RPO EPA Supersites SupSite NARSTO PM NARSTO EPA Division1, Division2, Division2 EPA Me and my dog for our aerosol project Me

PM/Haze Data Flow in Support of AQ Management There are numerous organizations in need of data relevant to PM/Haze Most interested parties (stakeholders) are both producers and consumers of PM and haze data There is a general willingness to share data but the resistances to data flow and processing are too high RPO Regional Planning Orgs FLM Federal Land Managers EPA EPA Regul. & Research Industry Academic NARSTO Other: Private, Academic SuperSite Shared PM/Haze Data PM and haze data are used for may parts of AQ management, mostly in form of Reports The variety of pertinent (ambient, emission) data come from many different sources To produce relevant reports, the data need to be ‘processed’ (integrated, filtered aggregated)

Scientific and Administrative Rationale for Resource Sharing Scientific Rationale: Regional haze and its precursors have a km airshed. (Smoke, Dust, Haze) – Data integration Substantial fraction of haze originates from natural sources or from out-of- jurisdiction man-made sources Cross-RPO data and knowledge sharing yields better operational and science support to AQ management Management Rationale: Haze control within some RPOs cannot yield Data sharing saves money and ….

A Strategy for the Federated PM/Haze Data Warehouse Negotiate with the data providers ‘open up’ their data servers for limited, controlled, access in accordance with clear ‘access contract’ with the Federated Warehouse Design an interface to the warehoused datasets that has simple data access and satisfies the data needs of most integrating users.(oxymoron ????) Facilitate the the development of shared value-adding processes (analysis tools, methods) that refine the raw data to useful knowledge

Three-Tier Federated Data Warehouse Architecture (Note: In this context, ‘Federated’ differs from ‘Federal’ in the direction of the driving force. Federated meant to indicate a driving force for sharing from ‘bottom up’ i.e. from the members, not dictated from ‘above’, by the Feds) 1.Provider Tier: Back-end servers containing heterogeneous data, maintained by the federation members 2.Proxy Tier: Retrieves designated Provider data and homogenizes it into common, uniform Datasets 3.User Tier: Accesses the Proxy Server and uses the uniform data for presentation, integration or processing Provider Tier Heterogeneous data in distributed SQL Servers Proxy Tier Data homogenization, transformation Federated Data Warehouse User Tier Data presentation, processing

Federated Data Warehouse Interactions The Provider servers interact only with the Proxy Server in accordance with the Federation Contract –The contract sets the rules of interaction (accessible data subsets, types of queries) –Strong server security measures enforced, e.g. through Secure Socket layer The data User interacts only with the generic Proxy Server using flexible Web Services interface –Generic data queries, applicable to all data in the Warehouse (e.g. data sub-cube by space, time, parameter) –The data query is addressed to the Web Service provided by the Proxy Server –Uniform, self-describing data packages are passed to the user for presentation or further processing SQLDataAdapter1 CustomDataAdapter SQLDataAdapter2 SQLServer1 SQLServer2 LegacyServer Presentation Data Access & Use Provider Tier Heterogeneous Data Proxy Tier Data Homogenization, etc. Member Servers Proxy Server User Tier Data Consumption Processing Integration Federated Data Warehouse Fire Wall, Federation Contract Web Service, Uniform Query & Data

Live Demo of the Data Warehouse Prototype Uniform Data Query regardless of the native schema: Query by parameter, location, time, method Currently online data are accessible from the CIRA (IMPROVE) and CAPITA SQL servers The hidden DataAdopter - accepts the uniform query - accesses the data server - transforms the original to uniform data - delivers uniforms DataSets A rudimentary viewer displays the data in a table for browsing.

‘Global’ and ‘Local’ AQ Analysis AQ data analysis needs to be performed at both global and local levels The ‘global’ refers to regional national, and global analysis. It establishes the larger- scale context. ‘Local’ analysis focuses on the specific and detailed local features Both global and local analyses are needed for for full understanding. Global-local interaction (information flow) needs to be established for effective management. National and Local AQ Analysis

Integration for Global-Local Activities Global Activity Local Benefit Global data, tools => Improved local productivity Global data analysis => Spatial context; initial analysis Analysis guidance => Standardized analysis, reporting Local Activity Global Benefit Local data, tools => Improved global productivity Local data analysis => Elucidate, expand initial analysis Identify relevant issues => Responsive, relevant global work Global and local activities are both needed – e.g. ‘think global, act local’ ‘Global’ and ‘Local’ here refers to relative, not absolute scale

Data Re-Use and Synergy Data producers maintain their own workspace and resources (data, reports, comments). Part of the resources are shared by creating a common virtual resources. Web-based integration of the resources can be across several dimensions: Spatial scale:Local – global data sharing Data content:Combination of data generated internally and externally The main benefits of sharing are data re-use, data complementing and synergy. The goal of the system is to have the benefits of sharing outweigh the costs. Content User Local Global Virtual Shared Resources Data, Knowledge Tools, Methods User Shared part of resources

Federated Data Warehouse Features Data reside in their respective home environment where it can mature. ‘Uprooted’ data in separated databases are not easily updated, maintained, enriched. Abstract (universal) query/retrieval facilitates integration and comparison along the key dimensions (space, time, parameter, method) The open data query based on Web Services promotes the building of further value chains: Data Viewers, Data Integration Programs, Automatic Report Generators etc..Web Services The data access through the Proxy server protects the data providers and the data users from security breaches, excessive detail