ICIS-NPDES Plugin Design Preview Webinar ICIS-NPDES Full Batch OpenNode2 Plugin Project Presented by Bill Rensmith Windsor Solutions, Inc. 3/15/2012

Agenda Project Background About OpenNode2, flows, and plugins About this Project Project Timeline ICIS-NPDES Full Batch Plugin Design Staging Tables Stage 1: Data Preparation Stage 2: Submission Stage 3: Result Processing Implementer Responsibilities Options for non-OpenNode2 States Q & A

About OpenNode2 Open source Exchange Network node software OpenNode2 can: send data to other Network partners make data available for others to query Most widely used node software on the Network Available at

About Flows and Plugins Each type of data on the network is a flow There is a flow for each regulatory area (air, waste, etc…) ICIS-NPDES is one of many available flows EPA defined the rules for sending data via the ICIS-NPDES flow Each flow is implemented as a Plugin in OpenNode2 Plugins are the software that contain the functionality needed to support a specific flow The OpenNode2 Google code site has plugins for most major regulatory flows to EPA

About this Project 15 ICIS-NPDES Full Batch states Will submit their NPDES data to EPA via the Exchange Network The ICIS-NPDES Full Batch flow is complex 46 different data families 149 tables (complex data types) 1195 fields (simple elements) 963 business rules To reduce implementation challenge, EPA and ECOS wanted to make available tools to simplify flow implementation for states.

About this Project (contd) In September 2011, EPA and ECOS engaged Windsor to develop full batch data flow plugin for OpenNode2 Ubiquity of OpenNode2 made it a good choice to reach the widest potential audience of states To reduce cost, only developing for the.NET version of OpenNode2 (most commonly used version) Pilot with Washington Dept. of Ecology Summer 2012 Does not include implementation of plugin at other states

Project Timeline 1/5/2012 – Plugin design completed 2/8/2012 – Staging tables released to Google Code 3/13/2012 – Beta Plugin release to Google Code 6-8/2012 – Test/Implement in WA 8/23/2012 – Final Plugin release to Google Code 12/2012 – Release of ICIS v4 Plugin Adds support for compliance and enforcement modules

Overall Submission Workflow

Workflow Lifecycle 1. Data Preparation Stage a. State-specific Extract, Transform and Load (ETL) b. Change Detection Process 2. Submission Stage 3. Result Processing Stage a. Retrieve Accept/Reject Report and Parse/Store Results b. Store Accepted Records Full lifecycle must complete before repeating Checks in place to prevent out-of-sequence execution

Workflow Lifecycle Tracking Workflow Tracking (ICS_SUBM_TRACK) Submission Result Processing Overall Status Data Preparation

ICIS-NPDES Staging Tables Two sets of staging tables: ICS_FLOW_LOCAL – Agencys NPDES data to send to ICIS ICS_FLOW_ICIS – Copy of data successfully sent to ICIS

Stage 1: Data Preparation ETL Step Refresh local tables with latest data from agency NPDES database

Stage 1: Data Preparation (contd) Three Ways to Load Agency Data: 1. Full Data Synchronization Transfer ALL agency NPDES data to Local staging database. Just keep it up-to-date with a regular refresh. Could be implemented as a full purge/rebuild or incremental refresh. Let the plugin figure out what is new, changed, or deleted and therefore what needs to be sent.

Stage 1: Data Preparation (contd) 2. Incremental Data with Automatic Change Detection Agency only populates data it wishes to send to ICIS. Requires that agency can track what data is new or changed since last successful submission to ICIS. Must turn of Auto generate deletes in ICS_PAYLOAD staging table. Let the plugin figure out what to send. 3. Incremental Data with Manual Change Detection Same as #2 but agency sets Transaction Codes in ETL. Does not rely on the plugin to figure out what to send. Does not leverage any of the plugins built-in change detection Fewer database components required, but much more complicated to implement for the agency.

Stage 1: Data Preparation (contd) Detect Changes Step Database routine compares Local data with ICIS data to determine what needs to get sent. Sets Transaction Codes (N, C, R). Leave Transaction Code blank if data is already in sync with ICIS. Inserts records into Local for Deletes (D, X).

Stage 2: Submission Plugin builds payload for all modules/records that have a Transaction Code set.

Stage 2: Submission (contd) Submission Settings in ICIS-NPDES plugin: Organization, Contact Info, and Author Gets inserted into XML header ICIS User ID The ICIS user performing the submission Notification Addresses Semicolon separated list Added to XML header, instructs EPA to send processing s Validate XML Yes/no

Stage 3: Result Processing Download, Parse and Store Results Step: When processing is complete, Node downloads and parses Accepted and Rejected Transactions into a Result Tracking Table.

Stage 3: Result Processing (contd) Store Accepted Transactions Step: Accepted Records are copied from Local to ICIS. Performed by a stored procedure executed by the plugin.

Stage 3: Result Processing (contd) Submission Settings in ICIS-NPDES plugin: Notification Addresses Semicolon separated list Notifications sent from OpenNode2 upon successful parsing/storing.

Stage 3: Result Processing (contd) How do I audit successes/failures? All feedback from ICIS is stored in result tracking table (ICS_SUBM_RESULTS) Only stores the accepted transactions from the most recent submission Stores all errors received for a given business key

What Can I Download?

What Can I Download? (contd) Plugin Plugin.zip – this is what you upload to OpenNode2 Database scripts – for SQL Server and Oracle Creates needed tables, procedures, views… Documentation Plugin Implementation Guide PDF Just the facts. Describes how to install and configure the plugin and database components. Plugin Design Specification PDF All the nitty-gritty details of the plugin design. Very useful to understand the details of how the plugin components work.

Implementer Responsibilities 1. Set up staging tables in state environment 2. Map source systems to staging tables Mapping to ICIS-NPDES lookup values 3. Develop data extraction and transformation routines to copy data from state database to staging tables 4. Set up node and plugin 5. Flow data to CDX Test environment 6. Monitor for errors and refine transformation logic 7. Migrate to production

Questions and Answers