Virtual Exchange Services VES 11/01/2017

Background The VES is a cloud-based platform designed to minimize the burden of building data exchanges by using standard templates that define much of the processing It has been in service for more than two years now Partners can use VES to supplement their existing node or to replace their node The exchanges (e.g. SDWIS) are configured not installed Partners can get started quickly using the VES web application wizards and powerful dataflow import functions All new components and features such as dataflow definitions and services are shareable (from VES or between partners)

Why VES? Provides a low cost option for partners to exchange data Eliminates the need for partner node HW/SW Streamlines development and maintenance of dataflows Updates to templates are made centrally Minimizes the support staff needed for a node You do still need a database person

What are the Virtual Exchange Services? A “data exchange as a service” platform that hosts partner services for State, Tribe, Region, Agency, Industry, etc. This means that all of the functionality of an application server is provided as standard reusable components. Partners no longer deal with the complexities of development, data formatting , dataflow logistics, notifications… Partners import exchanges (pre-filled forms) from a template for each dataflow they deploy The system generates REST and SOAP services based on the imported templates/forms Data submissions and data publishing flows are supported

What’s New in VES? Upgraded cloud infrastructure - Tripled the processing power Added new Q/A service to speed testing of dataflows Optimized XML creation processes to speed throughput of large submissions Upgraded to .Net workflow v4.5 Added support for ORACLE stored procedures in the workflow process              

What’s New in VES ? Added Custom Header storage support Implemented Owner Messaging (notifications/messages to all owners) Enhanced security policy module (access control managed by owners) Added supported for tasks with dynamic parameters (rolling dates or date ranges)  Implemented workflow persistence and pending transaction recovery  Added automatic REST URL generation for Query services              

Loads Staging data to either VES Partner Environment Azure Cloud Exchange Network/ CDX/ Other Users REST/ SOAP/ FTP .. Virtual Exchange Services Admin Configures dataflow based on Template Submissions Establishes connectivity Local Staging Tables Internet Service Bus Or VPN Reports Notifications Loads Staging data to either Local or cloud DB Cloud-based Staging Tables Production Database

SDWIS Dataflow Tasks Owner: VES SDWIS Prime State Create SDWIS Workflow in VES Establish SS Database Connectivity and Data source (if using local SS database) or VES establishes Cloud SS Database Scheduled VES workflow calls SDWIS Service for a specific state with parameters State Date Range Stylesheet Name(assuming you are supporting state specific data) SDWIS Service Extract data for the state (SQL query ) Transform raw data from SQL query into 1 SDWIS XML Schema (Stylesheet mapping) Return SDWIS State XML document as a Blob in the response to VES service call VES Workflow continues – Logs, Notifies, Processes Response Connect to SDWIS State Db in state (with established VPN or Service Bus) and inserts Blob or Connect to Cloud SDWIS State Db and inserts Blob Execute Load Process on SDWIS State Db Reviews processing reports

Potential SDWIS Dataflow Design Azure Cloud Partner Environment SDWIS Prime NCC Virtual Exchange Services State Admin Tasks Configures Data Source PRIME Run your DB scripts to establish Local database (pointer to your database) Create standard dataflow Create schedulers Create State Dataflows Establishes VPN or Service Bus Connection (if necessary) Extract Single States Data Service Parameters: Service, State, Date Range Stylesheet … Stylesheet SS XML Doc Local State Specific Database Blob State blob data inserted to either Local or cloud DB Solicit Execute State specific Stored Procedure Download Standard EN Asynchronous SOAP Request Response Exchange Network Node SDWIS Prime creates custom DBs and procedures to load data You create a SOAP service On your node to support Asynchronous extracts from SDWIS Prime Execute State Specific Stored Procedure Cloud State Specific Database Note: Supports any file size without timeout issues Potential SDWIS Dataflow Design

Home | About | FAQs | Help Configuring a Service in the VES Admin Tool Test User | Profile | Logout Home | About | FAQs | Help Virtual Node Administration Platform (VNAP) Node Dashboard Services Setup Wizard Steps Service Settings Database Service Template Finish Method Type Explanation of Fields Review these requirements to ensure your node is working. Solicit V Workflow Select a workflow… V Node ID / Description: CWANJ - Clean Water Act NJ Waterways Dataflow: CWANJ Service Status Enabled V Service Description Display all water transaction data Data Source Please select an existing data source and provide SQL statement to test cgi.oracle.test V SQL Statement SELECT * FROM {TABLE} WHERE {FIELD}.{TABLE} TEST QUERY

REST Data Services Virtual Exchange Services support REST type services for publishing data GetCountyCode – actual REST service on my Chris Clark Production VES: This is what the REST service URLs look like for my VES REST services JSON format: https://virtualnode.cloudapp.net/nodedataservice?node=ChrisClarkProductionNode&Dataflow=RESTServices&request=GetCustomerList&CustomerId=B&format=json   XML format: https://virtualnode.cloudapp.net/nodedataservice?node=ChrisClarkProductionNode&Dataflow=RESTServices&request=GetCustomerList&CustomerId=B&format=XML

Partners and Dataflows Hualapai Tribe:  WQX production flow San Joaquin Valley Air District:  ICIS AIR Jefferson County Health Dept: ICIS Air Klamath Tribe: WQX Alabama: ICIS-AIR Georgia: ICIS-AIR dataflows , EIS Region 1: Storm 3 Buoy Flow, SPCC Tanks St. Regis Mohawk Tribe Gila River Tribe - AQS Oregon Tennessee: ICIS-AIR, ICIS NPDES Texas: AQS, Facility FRS Maine: ICIS data access (Export) Gold Systems: WQX, ICIS-AIR, AQWMS New Hampshire: Facility, AQS, etc.. South Dakota: TRI-OUT, ICIS-AIR, EIS Pacific States Marine Fisheries Commission: Fishery information publishing service Washington - Radon Other Organizations OR,CT,VT,Chesapeake Bay Program

Data Exchanges Supported AQS 3.0 ICIS-AIR FACID v3.0 (FRS) E-Beaches EIS EMTS v3.0 ENDS v2.0 ICIS-NPDES SDWIS SPCC  v1.1 (Region 1) Buoy Data (Region 1) TRI (OUT) UIC WQX v2.1 CROMERR OWIR_ATT RCRA 5.2,5.3 Radon

Questions Contact Information: nodehelpdesk@epacdx.net Clark.chris@epa.gov

More Advantages Decreased Plug-in development time No Plug-in deployment Streamlined setup No administrative staff Simplified testing and QA