Inter-RPO Emissions Inventory Warehouse System Update Megan Schuster, MARAMA RPO Technical Meeting June 10, 2005

Technical Oversight Committee WRAP – Tom Moore CENRAP – Kathy Pendleton, Annette Sharp VISTAS – Pat Brewer, Greg Stella MRPO – Mark Janssen MANE-VU – Megan Schuster, Jung-Hun Woo, Serpil Kayin

Solution requirements Design of warehouse solution to meet RPO needs Outline Introduction Web-based solution Solution requirements Design of warehouse solution to meet RPO needs

Introduction and Background Inventory Development/Sharing Source types: point, area, nonroad, onroad, and biogenic sources RPOs have developed 2002 Baseline Photochemical Modeling Inventory PM2.5, Regional Haze and Ozone State Implementation Plans (SIPs) RPOs also developing future year and typical year inventories A lot of information Each RPO has develop an inventory for 2002 and will be developing future year inventories too.

Inventories are continually updated Challenge Inventories are continually updated Each RPO has several versions of each inventory Making the most up-to-date, quality assured inventory available to large user community is difficult Main issue – each RPO has several versions of each inventory that has been/ will be developed Difficult to get our data out to users in a timely fashion

Inter-RPO Emissions Inventory Warehouse Warehouse Solution Inter-RPO Emissions Inventory Warehouse Web-based application to store and share inventory and associated data

Warehouse Requirements Easy and accessible data upload and download Allow for several acceptable formats Versioning - Track versions of inventories Security - Secure Data Storage and Transmittal Maintenance - Low system operating and maintenance costs Database Storage Supporting Data Listserv Reports Low costs for system to survive for extended period of time

Required Data Formats and Files NIF Records (all sources) RPO Data Exchange Protocol Files Mobile emissions files Continuous emissions monitoring (CEM) files Temporal allocation and profile assignment files Spatial surrogate files Chemical speciation files Growth and control factor files Basically there a lot of data types that need to shared across the RPOs and with the general public, and we need a system that is large enough to handle all the file types necessary

Required Data Formats and Files Nonroad model inputs MOBILE6 model inputs NMIM inputs System is also required to handle Nonroad, M6 and NMIM inputs

Web Site Map ERG and Environ helped the Inter-RPO TOC design a system to meet all of our requirements. The system design is outlined in this website map. There will be 1 main homepage, with 4 pages under that. I will show screen shots of each page to give a taste

Web Site Functionality Four Main Components of Site Importing Data RPO assigned importer responsible for: Verifying the content of the files Determining if and when higher quality data are available and should replace Designating which files and info are associated with a specific version of the inventory Determining when a set of inventory files is complete and of sufficient quality to share with the public Follow naming convention for inventory files to be parsed into the database Required by EPA Content and Format Checker

Web Site Functionality Downloading Data Public users can search for and download files stored and managed in the system

Web Site Functionality 3. Reporting Public users Summarize and retrieve inventory data stored in the database Reports include Emissions summaries Top 10 sources Threshold Reports: sources that exceed pollutant and stack thresholds. Mobile VMT and speed data reports Reports can be generate for a subset of an inventory based on source type, geographic location, SCC level 1 or SCC level 2, pollutant, and stack parameters Emission summaries - user can choose inventory name, category – typical actual, source type – point, version, geographic location, (RPO, state, tribe, county) , SCC level, pollutant Top 10 sources – Point source reports will be at facility level, non-point at SCC level. Threshold report – inv version, category, year, source type, geo location, pollutant, pollutant threshold criteria, stack threshold criteria (for point source only), stack height threshold, stack diameter threshold, stack flowrate threshold. Point – at facility level, non-point at SCC level (at the level reported) Mobile vmt and speed data reports – inv version, category, year, geo location, roadway type, vehicle clssification, report type, vmt summary or avg speed, and time period: hourly or daily

Web Site Functionality Maintenance Manage users and modify limited portions of the site content

Web Site Functionality Three databases will be supported Working database – inventories actively being worked on by RPOs Public Database – publicly accessible data Archive Database – store older versions

Few clips of how the system is designed to work. Users will be required to enter a user name and password. Each user must have a valid email address. Listserv will be created based on login email addresses. Updates will be sent via a listserv.

As said earlier, a requiremetn of this system was that it need to track version of the inventory. One of the greatest challenges to the RPO community was managing and tracking inventory versions. The Inter-RPO Emissions Inventory Warehouse will automate the process of assigning versions to inventory data. Specific inventory version ‘names’ will be added to the database. Each RPO will have access to an upload area for their data files only. As RPO users upload data, they will be required to designate the inventory version for the file. Each RPO will be able to upload as many updates to an inventory as desired until a predetermined number of days before the inventory release date.

Public will be able to specify what geographic area, year, inventory version, source and file type they require. Depedning on size of the file – it may be placed on an FTP site or emailed to the user

As described earlier one of the requiremetns of this system was that it store a great amount of data. Uploading, storing and providing access to, and allowing users to download several versions of a nationwide inventory and ancillary modeling files each year requires a tremendous amount of server storage and database resources. To reduce the requirements for database storage, only those files and data needed for reporting purposes will be stored in the database. All other files will be stored in their native format, compressed if appropriate, on a file management server. To speed upload and download times, file transfer via http and ftp will be enabled. One of the challenges of using an open-source database tool is the potential impacts on the speed of real-time, on-line reporting when working with extremely large data sets such as those that will be stored in the Inter-RPO Emissions Inventory Warehouse. The RPO TOC decided to address this challenge in two ways: 1) after users select report criteria, the report will be scheduled and generated off-line; the user will be able to download the completed report at a later time, and 2) the speed of off-line report generation will be improved through the use of pre-aggregated data tables that can support most reporting requirements. Stored procedures will be used to populate these pre-aggregated tables after each version of an inventory is finalized and designated as ready for release

Open Source Solution Database: PostgreSQL Web Site Architecture Open Source Solution Database: PostgreSQL www.postgresql.org Procedural Language: PL/pgSQL Web interface: Java Server Pages (JSP) Servers: Linux, Apache Considering the requirement to provide Internet access to the data and files managed by the Inter-RPO Emissions Inventory Warehouse and the significant file/database server resource requirements, the licensing fees, hosting fees, and the costs of staff with specialized expertise required to prepare and maintain proprietary database technology such as Oracle or SQL Server could be prohibitive. To balance these competing needs, the RPO TOC chose to implement the website using reliable, proven, open-source technologies that are capable of handling large volumes of data. Additional details on the selected open source technologies and site architecture are provided later in this paper.

Draft Design document complete Final Deign Document due June Status and Next Steps Draft Design document complete Final Deign Document due June Require RPO TOC approval for implementation Begin Implementation June/July Testing in Fall Scheduled Final Implementation Date January 2006

Project Website: http://www2.ergweb.com/projects/interrpo/rpo/index.htm user name: rpo password: warehouse