1. Inter-RPO Emissions Inventory Warehouse System Update
Megan Schuster, MARAMA
RPO Technical Meeting
June 10, 2005

2. Technical Oversight Committee
WRAP – Tom Moore
CENRAP – Kathy Pendleton, Annette Sharp
VISTAS – Pat Brewer, Greg Stella
MRPO – Mark Janssen
MANE-VU – Megan Schuster, Jung-Hun Woo, Serpil Kayin

3. Solution requirements Design of warehouse solution to meet RPO needs
Outline
Introduction
Web-based solution
Solution requirements
Design of warehouse solution to meet RPO needs

4. Introduction and Background
Inventory Development/Sharing
Source types: point, area, nonroad, onroad, and biogenic sources
RPOs have developed 2002 Baseline Photochemical Modeling Inventory
PM2.5, Regional Haze and Ozone State Implementation Plans (SIPs)
RPOs also developing future year and typical year inventories
A lot of information
Each RPO has develop an inventory for 2002 and will be developing future year inventories too.

5. Inventories are continually updated
Challenge
Inventories are continually updated
Each RPO has several versions of each inventory
Making the most up-to-date, quality assured inventory available to large user community is difficult
Main issue – each RPO has several versions of each inventory that has been/ will be developed
Difficult to get our data out to users in a timely fashion

6. Inter-RPO Emissions Inventory Warehouse
Warehouse Solution
Inter-RPO Emissions Inventory Warehouse
Web-based application to store and share inventory and associated data

7. Warehouse Requirements
Easy and accessible data upload and download
Allow for several acceptable formats
Versioning - Track versions of inventories
Security - Secure Data Storage and Transmittal
Maintenance - Low system operating and maintenance costs
Database Storage
Supporting Data
Listserv
Reports
Low costs for system to survive for extended period of time

8. Required Data Formats and Files
NIF Records (all sources)
RPO Data Exchange Protocol Files
Mobile emissions files
Continuous emissions monitoring (CEM) files
Temporal allocation and profile assignment files
Spatial surrogate files
Chemical speciation files
Growth and control factor files
Basically there a lot of data types that need to shared across the RPOs and with the general public, and we need a system that is large enough to handle all the file types necessary

9. Required Data Formats and Files
Nonroad model inputs
MOBILE6 model inputs
NMIM inputs
System is also required to handle Nonroad, M6 and NMIM inputs

10. Web Site Map
ERG and Environ helped the Inter-RPO TOC design a system to meet all of our requirements.
The system design is outlined in this website map. There will be 1 main homepage, with 4 pages under that. I will show screen shots of each page to give a taste

11. Web Site Functionality
Four Main Components of Site
Importing Data
RPO assigned importer responsible for:
Verifying the content of the files
Determining if and when higher quality data are available and should replace
Designating which files and info are associated with a specific version of the inventory
Determining when a set of inventory files is complete and of sufficient quality to share with the public
Follow naming convention for inventory files to be parsed into the database
Required by EPA Content and Format Checker

12. Web Site Functionality
Downloading Data
Public users can search for and download files stored and managed in the system

13. Web Site Functionality
3. Reporting
Public users
Summarize and retrieve inventory data stored in the database
Reports include
Emissions summaries
Top 10 sources
Threshold Reports: sources that exceed pollutant and stack thresholds.
Mobile VMT and speed data reports
Reports can be generate for a subset of an inventory based on source type, geographic location, SCC level 1 or SCC level 2, pollutant, and stack parameters
Emission summaries - user can choose inventory name, category – typical actual, source type – point, version, geographic location, (RPO, state, tribe, county) , SCC level, pollutant
Top 10 sources – Point source reports will be at facility level, non-point at SCC level.
Threshold report – inv version, category, year, source type, geo location, pollutant, pollutant threshold criteria, stack threshold criteria (for point source only), stack height threshold, stack diameter threshold, stack flowrate threshold.
Point – at facility level, non-point at SCC level (at the level reported)
Mobile vmt and speed data reports – inv version, category, year, geo location, roadway type, vehicle clssification, report type, vmt summary or avg speed, and time period: hourly or daily

14. Web Site Functionality
Maintenance
Manage users and modify limited portions of the site content

15. Web Site Functionality
Three databases will be supported
Working database – inventories actively being worked on by RPOs
Public Database – publicly accessible data
Archive Database – store older versions

16. Few clips of how the system is designed to work.
Users will be required to enter a user name and password. Each user must have a valid address.
Listserv will be created based on login addresses.
Updates will be sent via a listserv.

19. As said earlier, a requiremetn of this system was that it need to track version of the inventory.
One of the greatest challenges to the RPO community was managing and tracking inventory versions. The Inter-RPO Emissions Inventory Warehouse will automate the process of assigning versions to inventory data. Specific inventory version ‘names’ will be added to the database. Each RPO will have access to an upload area for their data files only. As RPO users upload data, they will be required to designate the inventory version for the file. Each RPO will be able to upload as many updates to an inventory as desired until a predetermined number of days before the inventory release date.

20. Public will be able to specify what geographic area, year, inventory version, source and file type they require.
Depedning on size of the file – it may be placed on an FTP site or ed to the user

22. As described earlier one of the requiremetns of this system was that it store a great amount of data.
Uploading, storing and providing access to, and allowing users to download several versions of a nationwide inventory and ancillary modeling files each year requires a tremendous amount of server storage and database resources. To reduce the requirements for database storage, only those files and data needed for reporting purposes will be stored in the database. All other files will be stored in their native format, compressed if appropriate, on a file management server. To speed upload and download times, file transfer via http and ftp will be enabled.
One of the challenges of using an open-source database tool is the potential impacts on the speed of real-time, on-line reporting when working with extremely large data sets such as those that will be stored in the Inter-RPO Emissions Inventory Warehouse. The RPO TOC decided to address this challenge in two ways: 1) after users select report criteria, the report will be scheduled and generated off-line; the user will be able to download the completed report at a later time, and 2) the speed of off-line report generation will be improved through the use of pre-aggregated data tables that can support most reporting requirements. Stored procedures will be used to populate these pre-aggregated tables after each version of an inventory is finalized and designated as ready for release

23. Open Source Solution Database: PostgreSQL
Web Site Architecture
Open Source Solution
Database: PostgreSQL
Procedural Language: PL/pgSQL
Web interface: Java Server Pages (JSP)
Servers: Linux, Apache
Considering the requirement to provide Internet access to the data and files managed by the Inter-RPO Emissions Inventory Warehouse and the significant file/database server resource requirements, the licensing fees, hosting fees, and the costs of staff with specialized expertise required to prepare and maintain proprietary database technology such as Oracle or SQL Server could be prohibitive. To balance these competing needs, the RPO TOC chose to implement the website using reliable, proven, open-source technologies that are capable of handling large volumes of data. Additional details on the selected open source technologies and site architecture are provided later in this paper.

24. Draft Design document complete Final Deign Document due June
Status and Next Steps
Draft Design document complete
Final Deign Document due June
Require RPO TOC approval for implementation
Begin Implementation June/July
Testing in Fall
Scheduled Final Implementation Date
January 2006

25. Project Website:
user name: rpo
password: warehouse