1. Application of NASA ESE Data and Tools to Particulate Air Quality Management A proposal to NASA Earth Science REASoN Solicitation CAN-02-OES-01 REASoN: A Information Network for Earth Science Enterprise (ESE) Science, Applications and Education Stefan Falke and Rudolf Husar (Co-PIs) Washington University in St. Louis

2. NASA ESE and REASoN Programs As expressed in the NASA Earth Science Enterprise (ESE) Strategic Vision, the overarching goal for the ESE Applications Program (2001-2012) is to bridge the gap between Earth system science research results and the adoption of data and prediction capabilities for reliable and sustained use in decision supportEarth Science Enterprise (ESE) Strategic Vision As part of ESE, REASoN, (Research, Education and Applications Solutions Network) is a Distributed Network of Data and Information Providers For ESE Science, Applications and Education The data distribution network provide data products and tools for resource management and policy decision support in applications of national importance (including Air Quality Management), and provide decision makers with interactive access to dynamically updated knowledge of the Earth system REASoN emphasizes principles from the Strategic Evolution of ESE Data Systems (SEEDS) regarding community involvement and standards & interfaces for interoperability and exchange of data & information

3. Strategic Evolution of ESE Data Systems (SEEDS) SEEDS is not a large data system development effort with centrally controlled requirements, design, implementation and operation. Rather, the vision of SEEDS is: to engage the data user/data producer community in establishing and maintaining a unifying framework and management guidelines; to allow new applications projects to implement their own data systems and services while maintaining an aggregated system-wide interoperability. Based on these principles, we propose the application of ESE data to PM management through a ‘node’ within the larger ESE network. This node will: provide infrastructure for data access, network connectivity, and interoperability; process data to derive PM relevant products; provide value-adding tools for viewing, manipulating, analyzing, and presenting data.

4. Proposed Project: Application of NASA ESE Data and Tools to Particulate Air Quality Management REASoN applications projects must identify a user organization that will ultimately benefit from the project and partner with them. Projects are encouraged to establish partnerships with federal, regional and state agencies. (This is one of the reasons we’ve contacted you) We expect the proposed REASoN project to be useful for three levels of PM air quality management: PM Policy decisions Is intercontinental long range transport (LRT) of PM significant? PM Regulator decisions What are the PM concentrations; role of LRT; how to control? PM Implementation and Operation decisions Specific local and distant source attribution; State Implementation Plan (SIP) for PM; PM forecasting; health alerts Implementation & Operation Regulatory Policy Highly reduced, filtered, aggregated ‘knowledge’ Analyzed quantitative data on PM pattern, exceedances Considerable raw data, model input, verification Long decision time frame Intermediate to long decision time frame Intermediate and short decision time frame

5. Project Overview Rationale: Satellite, lidar and and other NASA data provide a unique view of particulate air pollution NASA ESE data and tools allow internet access to these resources The research community has begun using the satellite data However, these ESE data have not been incorporated in the PM management processes. Approach: Identify the ESE datasets and tools suitable for PM management Build infrastructure to support distributed data access and connectivity with existing networks Conduct data processing to convert raw ESE data to PM relevant data and combine with surface PM data Develop web tools to support decision making through data access, visualization, and analysis Apply SEEDS principles in working with the PM management community to incorporate the developed resources in the decision making process Benefits: Augment surface PM data with satellite-derived aerosol parameters, e.g. optical depth Provide broader and more detailed spatial coverage Implement an advanced infrastructure for resource sharing Improve both PM management and NASA data usage Products: True color, geo-referenced images of aerosol events Derived aerosol optical depth Fused satellite and surface data Animations of imagery and surface data

6. Use of Satellite Data within the PM Community Providing boundary conditions for PM models Helping assess inter-continental and regional transport Verifying and improving emissions inventories Spatial and temporal analysis of PM trends Source apportionment for SIPS Determination impact of LRT during PM standards violations Forecasting PM concentrations Satellite derived data are useful supplements to surface PM measurements and PM models. Some example uses of satellite data to aid particulate matter (PM) management include:

7. ESE Data Delivery In our proposed project, NASA ESE satellite data products relevant to air quality management will be accessible through web interfaces developed by Washington University as well as through Geospatial One-Stop and other environmental information networks using established standards and protocols, such as Open GIS specifications and FGDC metadata standards. ESE Data Decision Support EPA RPOs States Wash U Geospatial One-stop EPA and RPOs NASA Federated ESE A key objective of REASoN projects is to provide data distribution systems and information products and services that serve national applications by enabling the integration of ESE observations and predictions into decision support systems operated by national organizations and agencies.

8. Proposed Infrastructure Wrappers (middleware that translates heterogeneous data) provides access to data through a uniform front-end. The retrieval of specific data ‘slices’ required by viewing and processing services is controlled by the user. The user also decides which wrapped datasets to browse and how to stack and display the layers. Because of the flexibility and adaptability in a user-driven infrastructure, we expect the proposed infrastructure to support decision-making for a variety of policy, regulatory, and implementation needs.

9. Potential ESE Data Products MODIS Resolution 250-1000m Daily Derived Aerosol TOMS Resolution: ~100km Daily Upper Troposphere Aerosol SeaWiFS Resolution 1km-4km Daily Derived Aerosol GOES Resolution 1 km Hourly Aerosol Movement Future satellite sensors: ??

10. Integration with EPA and other surface data ESE data will be processed and combined with EPA particulate matter data to generate data products useful for PM management. For surface data, relevant EPA metadata and XML standards will be used. Web-based access, visualization and analysis tools will be developed and available for PM community use. An example is a spatio-temporal data explorer that provides interactive and dynamic access and browsing capabilities to distributed data sets. A prototype data browser can be accessed here: SeaWiFS Reflectance, PM2.5 in Idaho (Aug 2000)Aerosol Optical Depth, Fire Locations

11. Idaho Fires Example (August 2000) Processed SeaWiFS image showing aerosol optical depth Aerosol Optical Depth, Fire Locations SeaWiFS Reflectance, PM2.5 ASTR Fire Locations Smoke Aerosol

12. Community Input Because you are a potential user of the proposed NASA REASoN application, we would like your input in defining the scope of the project. Please let us know your interests or needs in using NASA data and products. Please send comments or questions to: stefan@me.wustl.edu and rhusar@me.wustl.edu or call 314-935-6099 Thank you!

13. ‘Global’ and ‘Local’ AQ Analysis AQ data analysis needs to be performed at both global and local levels The ‘global’ refers to regional national, and global analysis. It establishes the larger-scale context. ‘Local’ analysis focuses on the specific and detailed local features Both global and local analyses are needed for for full understanding. Global-local interaction (information flow) needs to be established for effective management. National and Local AQ Analysis

14. General Approach to SRDS Design Based on consensus, adopt a uniform relational data structure, suitable for regional and cross-Supersite data integration and analysis. We propose a star schema with spatial, temporal, parameter and method dimensions. The ‘original’ data are to be maintained at the respective providers or custodians (Supersites, CIRA, CAPITA...). We propose the creation of flexible ‘adapters’ and web-submission forms for the transfer of data subsets into the uniformly formatted ‘Federated Data Warehouse’. Data users would access the data warehouse manually or through software. We propose data access using modern ‘web services’ protocol, suitable for adding data viewers, processors (filtering, aggregation and fusion) and other value-adding processes.

15. The RDMS Schema: ‘Minimal’ Star Schema‘Minimal’ Star Schema The minimal Sites table includes SiteID, Name and Lat/Lon. The minimal Parameter table consists of ParamterID, Description and Unit The time dimensional table is usually skipped since the time code is self-describing The minimal Fact (Data) table consists of the Obs_Value and the three dimensional codes for Obs_DateTime, Site_ID and Parameter_ID Additional dimension tables may include Method and Data Quality. For integrative, cross-Supersite analysis, the database has to have, at the minimum, a ‘fact table’ and associated time, location, parameter and method tables as dimensions The CAPITA data exploration software, Voyager uses this minimal schema. Voyager has been in use for the past 12 years successfully encoding and browsing 1000+ datasets worldwide.Voyager The state of California still formats and distributes their AQ data on CDs using Voyager.

16. From Heterogeneous to Homogeneous Schema Individual SQL databases have varied designs, usually following a more elaborate ‘snowflake’ pattern (see Database Schema Design for the Federated Data Warehouse ). Database Schema Design for the Federated Data Warehouse Though they have more complicated schemata, these Supersite SQL servers can be queried along spatial, temporal, parameter, method dimensions. However, the query to retrieve the same information depends on the particular database schema. A way to homogenize the distributed data is by accessing all the data through a Data Adapter which accesses only a subset of the tables/fields from any particular database (shown red in schemata below). The proposed extracted uniform (abstract) schema is the Global Star Schema The final form of the uniformly extracted data schema will be arrived at by Federation consensus. Subset used Uniform Schema Fact Table Data Adapter Extraction of homogeneous data from heterogeneous sources

17. Live Demo of the Data Warehouse Prototype Uniform Data Query regardless of the native schema: Query by parameter, location, time, method Currently online data are accessible from the CIRA (IMPROVE) and CAPITA SQL servers The hidden Data Adapter - accepts the uniform query - translates the uniform to server-specific query - returns DataSet in uniform schema Data Returned in uniform schema A rudimentary viewer displays the data in a table for browsing. http://capita.wustl.edu/DSViewer/DSviewer.aspx

18. Federated Data Warehouse Federated Data Warehouse Architecture Three-tier architecture consisting of –Provider Tier: Back-end servers containing heterogeneous data, maintained by the federation members –Proxy Tier: Retrieves Provider data and homogenizes it into common, uniform schema and format –User Tier: Accesses the Proxy Server and uses the uniform data for presentation, integration or further processing The Provider servers interact only with the Proxy Server in accordance with the Federation Contract –The contract sets the rules of interaction (accessible data subsets; types of queries submitted by the Proxy) –The Proxy layer allows strong security measures, e.g. through Secure Socket layer The data User interacts only with the generic Proxy Server using flexible Web Services interface –Generic data queries, applicable to all data in the Federation (e.g. space, time, parameter data sub-cube) –The data query is addressed to a Web Service provided by the Proxy Server of the Federation –Uniform self-describing SOAP-wrapped XML data packages are passed to the user for presentation or further machine processing SQLDataAdapter1 CustomDataAdapter ImageDataAdapter2 SQLServer1 ImageServer2 LegacyServer Presentation Data Access & Use Provider Tier Heterogeneous Data Proxy Tier Data Homogenization, etc. Member Servers Proxy Server User Tier Data Consumption Processing Integration Federated Data System Fire Wall, Federation Contract Web Service, Uniform Query & Data

19. Content Integration for Multiple Uses (Reports) Data from multiple measurements are shared by their providers or custodians Data are integrated, filtered, aggregated and fused in the process of analysis Reports use the analysis for Status and Trends; Exposure Assessment; Compliance … The creation of the needed reports requires data sharing and integration from multiple sources.

20. Other REASoN Facts Proposal deadline: Nov. 26, 2002 Project Term: 5 years, beginning mid-2003 Funding Mechanism : Cooperative Agreement with NASA’s Office of Earth Science Ours will be an Application proposal Note: the NASA REASoN solicitation is for three types of proposals: Research: Improve accessibility by the NASA science community to, and accuracy of: a) data and data products, including selected geophysical parameters of Earth observations constructed from multiple sources; and, b) efforts that more effectively integrate and fuse sources for geophysical parameters that may not be directly observed; Application: Provide data products and tools for resource management and policy decision support in applications of national importance (including Air Quality Management), and provide decision makers with interactive access to dynamically updated knowledge of the Earth system Education: Address needs of the educational community particularly with respect to timely and ready access to Earth and environmental data to promote math, science and geography in K-12 education, and earth system science in undergraduate, graduate and post graduate education.

21. Federated Data System Features As much as possible, data should reside in their respective home environment. ‘Uprooted’ data in decoupled databases tend to decay i.e. can not be easily updated, maintained, enriched. Data Providers would need to ‘open up’ their data servers for specified data subsets and queries, in accordance with a ‘federation contract’. However, in general, the data structures of the Providers will not need to be changed. The global data schema of the federated data system is a multidimensional data cube, represented as ‘star schema’ relational Retrieval of homogenized multidimensional data from the federation facilitates integration and comparison along the key dimensions (e.g. space and time) The open architecture data warehouse (see Web Services) promotes the building of further value chains: Data Viewers, Data Integration Programs, Automatic Report Generators etc..Web Services

22. SRDS and Federated Data Warehouse Technologies Server hardware: 2 identical Dell PowerEdge 4400 servers, (SQL server and Web Server); dual processor Xeon, 1 GB memory, 260 GB RAID drives SQL Software: Microsoft SQL 2000 Enterprize Development Server Web Server: Microsoft IIS 2000, including Data Transformation Services for data ingestion. Programming Environment: Microsoft.Net languages (VB, C#) for creating and programming web objects and ASP.NET to create the distributed web pages. Note: The rapid development of distributed applications was recently made possible by the ubiquity of SOAP/XML as a data transport protocol, and Web Services/.Net as the distributed programming environment. In fact,.NET is still in version Beta2.Web Services

23. Related CAPITA Projects EPA Network Design Project (~$150K/yr –April 2003). Development of novel quantitative methods of network optimization. The network performance evaluation is conducted using the complete PM FRM data set in AIRS which will be available for input into the SRDS. EPA WebVis Project (~$120K/yr - April 2003). Delivery current visibility data to the public through a web-based system. The surface met data are being transferred into the SQL database (Since March 2001) and will be available to SRDS. NSF Collaboration Support Project (~$140K/yr – Dec 2004). Continuing development of interactive web sites for community discussions and for web-based data sharing; (directly applicable to this project) NOAA ASOS Analysis Project (~$50K/yr - May 2002). Evaluate the potential utility of the ASOS visibility sensors (900 sites, one minute resolution) as PM surrogate. Data now available for April-October 2001 – can be incorporated into to the Supersite Relational Data System. St. Louis Supersite Project website (~$50K/yr – Dec 2003). The CAPITA group maintains the St. Louis Supersite website and some auxiliary data. It will also be used for this project

24. Integration for Global-Local Activities Global Activity Local Benefit Global data & analysisSpatial context; initial analysis Analysis guidance Standardized analysis, reporting Local Activity Global Benefit Local data & analysisElucidate, expand initial analysis Identify relevant issuesResponsive, relevant global analysis Global and local activities are both needed – e.g. ‘think global, act local’ ‘Global’ and ‘Local’ here refers to relative, not absolute spatial scale

25. Federated Information Architecture Data producers maintain their own workspace and resources (data, reports, comments). However, part of the resources are shared through a Federated Information System. Web-based integration of the shared resources can be across several dimensions: Spatial scale:Local – global data sharing Data content:Combination of data generated internally and externally The main benefits of sharing are data re-use, data complementing and synergy. The goal of the system is to have the benefits of sharing outweigh the costs. User Local Global Federated Information System Data, Knowledge Tools, Methods User Shared Private

26. VOYAGER Data Explorer: Architecture and Technologies Built and used Used by a Virtual Community on AerosolsVirtual Community Layered Map Time Chart ProvidersUsers Vector GIS Data XDim Data SQL Tables Web Images Voyager Web Services Publish, Find, Bind Data & Tool Catalog Uniform Access Scatter Chart S u p p o r tCoord./Cooperation T e c h n o l o g i e s Select, Overlay, Explore; Multidimensional data Maintain Distributed Data; Heterogeneous coding, access Connect providers to users; Homogenize data access

27. Web Publish HTTP, FTP Web Services Now: Data Access though a Web Service Adapter Service Broker Publish Service Consumer Find Access Ordinary web content can be delivered as a Web Service through a Proxy Server. The Adapter Service converts HTTP/FTP service to XML Web service The Adopter Service publishes the web service to the Broker The User finds the data from the broker and accesses the Adapter to get distributed data Service Adapter Web Server Service User Chain

28. Example Data Adapter: Daily TOMS Aerosol Index Map TOMS Image Metadata Geo-rectangle (65, -180; -65, 180) Image Size (640, 480) Image Margins (40, 40, 30,30) Transparent Colors (0,0,255 ) Image Access Metadata StartDate EndDate Incement DataType URL template: :ftp://jwocky.gsfc.nasa.gov/pub/eptoms/images/aerosol/YYYYY/eaYYMMDD.gif

29. Quebec Fires, July 6, 2002 SeaWiFS, METAR and TOMS Index superimposed SeaWiFS satellite and METAR surface haze shown in the Voyager distributed data browser Satellite data are fetched from NASA GSFC; surface data from NWS/CAPITA servers

30. Browsing of Distributed Data from HTTP/FTP Servers Land Reflectance from SeaWiFS Project, NASA GSFC AVHRR Oceanic Aerosol CAPITA, WashU, STL Fire Pixels, Jan 1997, ESA Ionia Project

31. Trans-Atlantic Transport of Quebec Smoke July 11: Smoke approaching Europe July 10: Quebec smoke over Mid- Atlantic SeaWiFS Reflectance TOMS Absorbing Aerosol SeaWiFS Reflectance TOMS Absorbing Aerosol Spain E. US

32. NRL Forecast Model for Dust, Smoke and Sulfate METAR Surface Haze Real-time model and surface observations are compared spatially and temporally Dust Sulfate Smoke METAR Haze Time Selector

33. Vertical Pattern of Global Aerosol Windblown Dust (crustal elements) Biomass Smoke (organics, H 2 0 ) Sea H 2 0 salt (NaCl. H 2 0) Stratospheric (Volcanic) (H2SO4) Biogenic (Non-sea salt sulfate, org) Urban-Industrial Haze (SO4, org. H 2 0) Dust, smoke, volcanic aerosol and industrial haze originate from land The global aerosol concentration is highest over land and near the continents over the oceans (coastal regions) Sea salt is significant over some of the windy oceanic regions and biogenic sulfate and organic aerosols also occur …

34. Megatrends Related to PM2.5 and Ozone From SO 2 and TSP to ozone and fine particulates. –Recent health and environmental effects studies implicate ozone and fine particulates as two of the most serious current air quality problems in north America. From primary to secondary pollutants. –Ozone as fine particles are not primary (emitted) but formed in the atmosphere from complex mixtures of precursor gases. There are no direct ways of identifying the impact of specific sources. From short range to long range impact. –The atmospheric lifetime of O3 and PM2.5 is several days, so the winds carries them over 1,000 km from their source. The result is "long-range transport" across state and international boundaries. From command and control to ‘weight of evidence.” –The new AQ management style strives to include stakeholders in the policy development; Encourages market-based resource allocations and applies 'weight of evidence' - to compliance management.

35. Environmental Megatrends Short to long range transport. Pollutants (e.g. ozone, PM2,5, POPs) travel across state and national boundaries. New regulatory approach. Compliance evaluation is now based on ‘weight of evidence’ and the effectiveness of controls need to be tracked. From command & control to participatory management. The participating stakeholders now include federal, state, local, industry and international members

36. The Air Quality Manager’s Challenge Broader user community. The information systems need to be extended to reach all the stakeholders (federal, state, local, industry, international) A richer set of data and analysis. Establishing causality, ‘weight of evidence’, emissions tracking requires the analysis of air quality, meteorology emissions and effects data. Secondary pollutants along with more open environmental management style require broader and more detailed data analysis.Increasing demand for analysis. Secondary pollutants along with more open environmental management style require broader and more detailed data analysis.

37. The Researcher/Analyst’s Challenge The researcher cannot get access to the data; if he can, he cannot read them; if he can read them, he does not know how good they are; and if he finds them good he cannot merge them with other data. Information Technology and the Conduct of Research: The Users view National Academy Press, 1989 Air Quality Data Integration and Living Data Inventory

38. Opportunities Rich AQ data availability. Abundant high-grade routine and research monitoring data from EPA and other agencies are now available. New information technologies. Effective data management along with distributed analysis, exploration and communication tools allows cooperation (sharing) and coordination among diverse groups. More Cooperative Spirit. The stakeholders increasingly recognize the need and the benefits of collaboration(sharing) and coordination.

39. Air Quality Management: Sensory Data to Action Air Quality Assessment Compare to Goals Plan Reductions Track Progress Controls (Actions) Monitoring (Sensing) Set Goals CAAA NAAQS Assessment performs data analysis to turn data into useful knowledge for decision making and actions Multi-sensory data are collected through Monitoring and delivered for Assessment

40. Analysis: From Raw Data to Refined Knowledge Data Refinery: Data analysis can be viewed as a refinery that transforms raw sensory data into knowledge usable for management Multi-step processing. The data refining has many parallel and sequential steps, usually performed by different analysts. Value-Adding Chain. Each step in the analysis is part of a value-adding chain. Example data to knowledge refining: Environmental Status Report 1.Primary data are gathered from providers of sensory data 2.Data are filtered, aggregated and fused into secondary data, figures, tables 3.Report describes pollutant pattern and possibly causality

41. Environmental Data and Use Features Multidimensional. The key data dimensions are space (x,y,z) and time (t). Need for current and historical data. Daily (hourly) as well as long-term strategic management decisions need to be supported. Data from many sources. For full context, data from multiple sources need to be combined and analyzed, e.g. –Air quality data (collected by many federal, state and local agencies) –Weather data (from the National Weather Service) –Possibly satellite data (from NASA or NOAA)

42. Distributed Environmental Data Analysis System Distributed Environmental Data Analysis System DEDAS Specifications: Specifications:  Use standardized form of data, metadata and access protocols  Support distributed data archives, each run by its own providers  Provide tools for data exploration, analysis and presentation Features: Features:  The data are organized as multidimensional data cubes  The dimensional data cubes are distributed but shared  Analysis is supported by built-in and user functions

43. A Possible Architecture of DEDAS There are four types of nodes in the system: Data Providers, Organizers, Transformers and Users. The Users receive data on demand from the Providers through DEDAS

44. Sensory-Motor Response to Changes Regardless whether the Earth is considered ‘healthy’ or ‘sick’, the inevitable and unforeseeable environmental changes require response to these changes: The response includes the following major steps: The above three steps are the necessary conditions for sustainable development. This is logical since all living organisms use this type of sensory-motor feedback to maintain their existence. Sensing and recognition (monitoring) Reasoning and explaining (sciences) Decision making, action (management)

45. Information Value Chain: Data Providers, Organizers, Transformers and Users. Data Providers supply primary data to system, through SQL or other data servers. Data Organizers populate the data cubes with ‘primary data’ from the Providers Transformers add value to the primary data by processing (e.g. filtering, aggregation, fusion). They produce secondary data in ‘virtual data cubes’ accessible to the users Users are the analysts who access the DEDAS and produce knowledge from the data

46. Benefits of DEDAS Access to data. Data in DEAS can be easily found, accessed, processed and presented. Recycling data. Data are costly resource. The system can help managing, accessing and documenting one's own data, and sharing it with others for re-use. Saving time and money. The data, tools and other resources in the shared system could be leveraging the dollars and time available for specific projects.