Improving an Air Quality Decision Support System through the Integration of Satellite Data with Ground-based, Modeled, and Emissions Data NASA ROSES 2007: Decision Support through Earth Science Research Results Project Update 1/7/2010: Cooperative Institute for Research in the Atmosphere

Project overview NASA ROSES proposal to enhance VIEWS/TSS with NASA satellite data Based on the NASA ROSES omnibus solicitation of 2007 We submitted the original proposal in 2007 or 2008 (?) Proposal was awarded in 2008 Work began in 2009 after NASA funding was received Uma Shankar (UNC) and Shawn McClure (CIRA/CSU) are the co-PIs Many collaborators, including Dr. Bret Schichtel (NPS) and Tom Moore (WRAP) at CIRA Two people from CIRA working part time on the project Multiple people from UNC working part time on the project Period of performance ends in May 2011

Project goals Overall Project Goal To enhance and add value to a currently operational air quality decision support system (VIEWS/TSS) by integrating and utilizing satellite data from NASA satellites Aura, Terra, Aqua, and CALIPSO. Specific Project Goals Develop routine capture, analysis, and processing algorithms with high temporal and spatial resolution to provide land use/land cover data as inputs to emissions and air quality modeling Achieve more complete temporal and spatial resolution of activity data and emission rates from natural and anthropogenic emission sources in remote areas and from individual sources and source clusters Obtain multiple-dimensional vertical profiles and column measurements of pollutants to improve model inputs and provide evaluation data for gridded chemistry-transport models such as CMAQ Develop advanced analysis tools to better understand the relevant atmospheric processes and their representation in the CTMs Visualize and analyze satellite data in combination with existing monitoring, emissions, and modeling data within a unified decision support platform

Introduction to VIEWS/TSS The Visibility Information Exchange Web System is a database system and set of online tools originally designed to support the Regional Haze Rule enacted by the EPA to reduce regional haze in national parks and wilderness areas. What is VIEWS? Serve as the primary online source for IMPROVE aerosol and RHR datasets Provide easy online access to a wide variety of ground-based air quality data Provide online tools for exploring, analyzing, and downloading this data Maintain a catalog of relevant air quality-related resources Facilitate the research and understanding of global air quality issues What its primary goals? An extension to VIEWS developed by the Western Regional Air Partnership (WRAP) to provide a single portal to technical data and analytical results prepared by WRAP Forums and Workgroups, and to serve as a decision support system for aiding the development of state and tribal implementation plans. What is the TSS?

Simplified, CIRA-specific project goal Make it easier to… find visualize query download …satellite and modeled data in conjunction with ground-based data

Fulfilling our goal: Getting the data

Fulfilling our goal: Importing and integrating the data

Fulfilling our goal: Enhancing our system to use the data

Leveraging the VIEWS Architecture

Project Overview: What we’ve done on the back end Imported a large collection of satellite and modeled air quality data Implemented an extensive software library for querying, filtering, managing, and reformatting HDF and netCDF files, both programmatically and remotely Acquired expertise in manipulating, managing, and analyzing satellite data for use in air quality applications Designed an experimental spatial relational database schema for managing gridded satellite and modeled data Developed an automated suite of utilities to dynamically generate images from HDF and netCDF files Designed a folder and file naming convention for facilitating the programmatic management, exploration, and parsing of data files Implemented a suite of temporal aggregation utilities that can operate on native satellite and modeled data files Designed strategies and methods for performing meaningful spatial interpolations of gridded and point-based data Developed uniform KML/KMZ schemas for representing point data

Project Overview: What we’ve done on the front end Designed a 2D mapping tool using Google Maps that can display satellite, modeled, and ground based datasets Developed an interface for querying (“drilling down” through) a set of spatial layers to retrieve and display data values at a selected point on a map Developed a timeline tool for visualizing the temporal availability and coverage of our inventory of satellite, modeled, and ground based datasets Developed a general purpose file “explorer” (similar to Windows Explorer) for browsing and retrieving documents and data files Developed a tool for browsing, selecting, and animating the large collection of static images generated from the source data files Implemented a very loosely-coupled integration of UNC Tools (AMET, VPM) with the VIEWS website Designed access points and client-server communication mechanisms to the new tools from existing pages on VIEWS Developed a suite of visual, interactive test harnesses for exercising and refining the various software components developed for this project.

Project Overview: What we haven’t done Achieved a more thorough and pervasive integration of the new tools with the existing tools and web pages Developed end-to-end, real world case studies that “mash up” individual tools, components, and services into polished end user analyses and reports Realized a more comprehensive integration with remote data systems such as RSIG and Giovanni Integrated the satellite and modeled data more thoroughly and transparently into the spatial/relational database, and utilize this integration from the tools. (NOTE: This “lack” of a fuller RDBMS integration was partially due to both intentional design decisions and also unexpected complexities.) Achieved a tighter, more seamless integration of the UNC tools (AMET, VPM) with VIEWS website and database Made as much progress as hoped/expected on the visualization and analysis tools

Challenges and issues encountered Key personnel resources were lost mid-project (details and ramifications?) –A key satellite scientist had to be released due to unexpected budgetary constraints (CIRA) –Two lead developers took different positions during intensive development phases (UNC) An unexpectedly high degree of heterogeneity was encountered in satellite data formats and conventions The quality and completeness of available satellite data documentation was very non-uniform and often difficult to find and understand Many satellite measurements were found to have limited direct relevance to meaningful air quality studies Low-level data management of satellite data was found to be a more complicated and “one-off” exercise than hoped Relational database capabilities were found to be lacking with regard to handling densely- gridded data Data import, management, and post-processing, as well as software development, was more complicated and took longer than hoped

Import: Organize the source data files on a file server

Import: Apply a strict file and folder naming convention Aura_OMI_L3 ColumnAmountO3 CloudFraction World netCDF nc ← _ _ ← Individual components of the folder path and file name: Resulting file name: Aura_OMI_L3!ColumnAmountO3!CloudFraction!World!netCDF! nc

Import: Understand the structure and contents of the data

Import: Understand the context of the data (i.e. metadata) Source o Satellite platform o Atmospheric model o Data provider Collection Method(s) Processing Method(s) Value interpretation o Concentration o Integrated column amount o Radiance measure o Etc… Units, precision, significant digits

Import: Extract relevant subsets of the data

Import: Cross reference parameters across datasets

Derived data products Uniformly formatted netCDF files –Optimized for long term storage –More easily consumed by end users –Facilitates interoperability and standardization –Better enables derived product generation Static graphical images –Can be used by a wide variety of visualization tools –Transparency of *.PNG images is ideal for Google and Bing Maps –Enables quick browsing and identification of events –Enables client-side animations KML and KMZ files –Makes it possible to store feature and topology data –Can be used by current and popular visualization tools –Enables direct data and metadata query –Facilitates sharing and interoperability

Accessing project data and tools from the main menu

File Explorer tool

Image Browser and Animation Tool

Dataset Timeline and Parameter Inventory tool

Map Viewer: OMI Level 3 Ozone Column

Map Viewer: OMI Level 3 Radiative Cloud Fraction

Map Viewer: IMPROVE Ammonium Sulfate

Map Viewer: Three layers combined

Viewing data products in Google Earth

Test harnesses and alpha tools

Exceptional event metadata (such as fires) can be stored and dynamically associated with data, photos, and imagery at run time in order to provide quick cross-referencing and discovery of related data. Thus, satellite and modeled data can be quickly associated with aerosol events once these linkages are complete. “Bisquit” fire impacting Crater Lake in 2002 Roadmap: Linking data to relevant photos and imagery

Roadmap: Specific case studies using component tools Comparing CMAQ AOD with MODIS AOD during specific events

Roadmap: Supplemental charts and analyses 2*cmaq aod CMAQ AOD X 2 JJA 2001 MODIS Avg. AOD JJA 2001

Roadmap: Speciated contribution to air quality case studies Fractional SO 4 AOD Fractional NO 3 AOD Fractional NH 4 AOD Fractional OC AOD Fractional BC AOD Fractional A25 AOD Fractional CM AOD August 2001 Fract. AOD Sulfate contributes ~ 40%

Roadmap: Overlaying HYSPLIT back trajectories

“Interoperability”: The ability of diverse systems and organizations to work together to exchange and utilize information. To facilitate this, VIEWS will offer the following services and features: Discovery, retrieval, and exchange of data and metadata On-the-fly transformation and formatting of data and metadata Upload and management of user-supplied data and metadata Generation of visualization and analysis products Availability of “embeddable” components and tools Support of OGC web standards for spatial data exchange VIEWS/TSS Interoperability: Services and standards

Earth Science Information Partners (ESIP) Federation Collaborate on interoperability efforts for data/metadata discovery and exchange Contribute data and services to the Earth Information Exchange ESIP website: GEO/GEOSS Architecture Implementation Pilot (AIP) Contribute components and services to the GEOSS registry Participate in efforts to standardize and streamline data discovery and exchange Develop VIEWS/TSS as an “persistent operational exemplar” of the GEOSS architecture GEOSS AIP 2 link: CyAir (cyberinfrastructure for the air quality community) Contribute to the planning, development, maintenance and coordination of systems to help the air quality community better utilize air quality data and information. Assess the current state of air quality data providers Investigate and assess data user needs and issues, including types of users, types of data, data access methods, data usage and transformation, and software usage Identify and assess existing or needed standards and conventions Generate a community vision for air quality cyberinfrastructure Interoperability and Collaborations

VIEWS/TSS Data Value Chain Diagram

Roadmap: From now until the project ends January 2011 – Beta testing phase begins January 10-11: Beta tester’s meeting at UNC January-May, 2011: –Organize and prioritize feedback –Incorporate ideas and requests –Resolve problems and issues –Import additional data –Generate additional derived products –Implement key case studies with component tools –Push changes and additions to testers and users –Repeat the above steps in various combinations Solicit NASA for a possible project extension May 2011 – Period of performance for the project ends Publish final versions of data products, website tools, and project reports Organize and conduct any final training and/or project status meetings Announce enhancements via VIEWS and TSS websites and user newsletter Design scope of work and strategy for project extension, if forthcoming

Roadmap: Keys issues, operation and maintenance VIEWS/TSS funding has expired, and no renewals have been tendered All work on the project, including ongoing data import, will cease after May 2011 Responses to feedback and support requests will cease or be intermittent and/or infrequent No hardware repairs will be made No software upgrades or troubleshooting can be done The websites will “piggyback” on existing hardware and infrastructure (electricity) as long as possible NPS is funding the Federal Environmental Database (FED) Leverages key components and infrastructure from VIEWS Will continue the import of ground-based datasets Will improve and refine various tools But no funding for satellite-related data or development Possible extension to NASA ROSES project Will help us finish key functionality and polish existing features Will extend the time over which new satellite and modeled data will be imported But it’s not a sure thing

The End… Thanks!