A State Perspective Mentoring Conference New Orleans, LA 2/28/2005 RCRAInfo Network Exchange
State Perspective - Overview New process – everyone involved is learning – New technologies – New business processes – New relationships Multiple parties – no true Project Lead or Authority – Competing priorities – Resource restrictions Beta states should expect more changes/rework as schema is tested, modified, and finalized Positive Note - this is progress. Future projects should benefit from our efforts.
RCRA Implementation Challenges – the Flow Implementer of Record (IOR) Primary Keys - Staying in Synch Business Data and Composite Keys Look-Up Code Maintenance Data Conversion Gaps in Data Validation (for historical data) Developing additional RCRA exchange process flows – Data sharing and synchronization services – Get Handler ID; Get x by Handler
RCRA Implementation Challenges – the Project Transfer of Knowledge – Business team and Technical team – Contractors and Internal Resources – EPA and State team Project Phases with schema fluctuations require flexible relationships with resources (internal and external staff) Grant constraints create tension with project goals – Satisfying time and financial constraints could lead to tension between broad goals of network exchange and immediate scope decisions Disparate Parties – Identify appropriate expert resources / decision makers – Communication standards are evolving
RCRA Implementation Challenges – the Project What are the affects of Schema changes? – Optional fields, element names, namespace – Structural modifications and mandatory fields Identify phases that cause schema changes – determine navigation path – Drafting schema, Translator Design and Build, State Mapping, State/Pilot Testing, Production – Timing of activities and parties involved in the activities can affect schema stability Other Factors – Source and Target system modifications need to be monitored
MDEQ TEMPO-RCRAInfo Data Extraction
RCRAInfo Mapping Challenges Corresponding Events to Units Permitting Actions – time to complete Event Capture Tracking Primary Keys Implementing Transaction Codes The Payload Operation Decision
Corresponding Events to Units
Permitting Actions – Time to Complete Permits require long periods of time to complete. Interim periods required data submissions Traditional method of permit “complete” could not be used to indicate submissions. What would be the trigger? What would be the data?
Permitting Actions – Time to Complete Basics Trigger: Completion of RCRA Event Data: All data available linked to the event – …with conditions Conditions Event Capture: Data added or changed events Date criteria: – Last date file generation executed – Current date – Comfort Zone
Permitting Actions – Time to Complete Comfort Zone Current Date: 10/22/2004 Last Run Date: 10/15/2004 Comfort Zone: 3 days
MDEQ RCRAInfo Event Capture Track data events in source system – Insert new data – Update previously existing data – Delete previously existing data Event, table, primary keys and “data over time” Components – Database Triggers – Event Capture table structure – Event Capture interpreter
Event Capture Table Structure
MDEQ RCRAInfo Event Capture - Interpreter Identify Event as occurring on table. Group table and column events since last execution Determine “net” value for data element(s) Store “conclusion” in memory to generate file Loop Clear Event Capture tables using comfort zone
Waste Codes and Event Capture Some Primary Keys of RCRAInfo tables require distinct event logic Tracking the change of data and composing the correct net transaction can be difficult Example: RCRAInfo Waste Codes PK – Waste Code, Waste Code Owner, Handler ID, Unit Sequence Number, Unit Detail Sequence Number – No waste code sequence number
Waste Codes and Event Capture C t1 B t1 A t1 A t2 Q t2
Tracking Primary Keys Method of tracking/generating correct Primary Key (PK) for data in RCRAInfo Capture PK structure of RCRAInfo and map to PK structure of source system Challenge: – Learning the RCRAInfo model – Keeping in synch – sequence numbers – Similar challenges and structure in look-up code context
Implementing Transaction Codes
Valid Transaction Codes – A (Add/Update) – D (Delete) – X (Submitted for Context) Add and Update are handled in the same manner - the translator will interpret an Add as an Update when necessary – If PK already exists, add is interpreted as update – Record is replaced/updated, not the element (send all data for record) – Potential lack of feedback on incorrect data synchronization
Implementing Transaction Codes Nesting a delete under update transaction codes Cascade Delete ramifications – Sending delete on the parent table row only – “Child” deletes will trigger critical file submission errors Cannot delete co-implemented records (transactional or full-replace) – Keep in mind with cascade deletions
Implementing Transaction Codes: De-Normalized Data
How does one delete a link between an event and a unit? How does one completely delete a unit? Applying the Delete transaction at the child and root levels has different effects
Miscellaneous Challenges Establish SOPs - new source system features Data configuration and conversion resources Facilities that exist in multiple counties Address data – multiple lines to single elements Converting geographic coordinate data to Decimal Degrees Owner Operator Data – Polling entity types – Active/Inactive – Type Codes
Discussion