1. Becoming a Millionaire: Lessons Learned While Submitting 30 Years of Monitoring Data to WQX James Hudson Wisconsin Department of Natural Resources

2. Hudson, 11/20/2007 2 What We Did Started testing WQX in January 2007 Started testing WQX in January 2007 Started production submittals in March Started production submittals in March Submitted 1 M results by June Submitted 1 M results by June Identified analytic methods for other samples (including state-specific ones) Identified analytic methods for other samples (including state-specific ones) Submitted another 600K results in October Submitted another 600K results in October Monthly submittals of new/changed data Monthly submittals of new/changed data

3. Hudson, 11/20/2007 3 What Came Before Nothing submitted to Modernized STORET for over 10 years Nothing submitted to Modernized STORET for over 10 years Extreme pressure to submit data (Grant requirements) Extreme pressure to submit data (Grant requirements) Major redesign of our internal systems, 2005-2007 Major redesign of our internal systems, 2005-2007 Major effort for data migration and data quality review Major effort for data migration and data quality review

4. Hudson, 11/20/2007 4 Other Success Factors Management wanted this to succeed Management wanted this to succeed Exchange Network experience Exchange Network experience –FRS –NEI –Drinking Water –AQS All data in Oracle, with experience converting Oracle to XML All data in Oracle, with experience converting Oracle to XML Motivated, engaged project team Motivated, engaged project team Extensive Preparation Extensive Preparation

5. Hudson, 11/20/2007 5 Exchange Network Challenges Obtaining the test and production NAAS accounts Obtaining the test and production NAAS accounts Getting them authorized for WQX Getting them authorized for WQX Identifying errors, especially at the CDX level in the Test environment Identifying errors, especially at the CDX level in the Test environment What to do when something goes wrong? What to do when something goes wrong? –Call Joe at the Helpdesk

6. Hudson, 11/20/2007 6 WQX Challenges Can be hard to understand some nuances, even with help Can be hard to understand some nuances, even with help –Example: took me 3 tries to get the organization- specific analytic methods right Split of responsibilities between WQX and the Helpdesk Split of responsibilities between WQX and the Helpdesk Problem analysis: is the error in the submission, CDX, or WQX? Problem analysis: is the error in the submission, CDX, or WQX? –Sometimes just resubmit

7. Hudson, 11/20/2007 7 Internal Database Challenges, The Techie Section Schema and Stylesheets Schema and Stylesheets Database views Database views Performance Performance Transformations Transformations Tracking changes Tracking changes

8. Hudson, 11/20/2007 8 Schemas and Stylesheets Issue: Oracle uses names such as ACTIVITY_START_DATE, Schema uses ActivityStartDate Issue: Oracle uses names such as ACTIVITY_START_DATE, Schema uses ActivityStartDate Solution, transform in Stylesheet Solution, transform in Stylesheet

9. Hudson, 11/20/2007 9 Database Views Used to Transform Internal Structures to WQX Used to Transform Internal Structures to WQX We use object-relational views We use object-relational views Code looks like: Code looks like: CREATE OR REPLACE VIEW EQ_XN_PROJECT_V CREATE OR REPLACE VIEW EQ_XN_PROJECT_V OF XN_WQX_PROJECT WITH OBJECT IDENTIFIER (xn_key) AS select to_char(plan_seq_no) as xn_key, null as xn_parent_key, null as xn_parent_key, plan_id as project_identifier, plan_id as project_identifier, asciistr(plan_name) as project_name, asciistr(plan_name) as project_name, pk_xn_wqx_utility.f_project_desc(plan_comment) pk_xn_wqx_utility.f_project_desc(plan_comment) as project_description_text, as project_description_text, cast (null as xn_wqx_attachment_t) as cast (null as xn_wqx_attachment_t) as associated_binary_object associated_binary_object from wt_swims_plan p One of these for each major portion of the schema One of these for each major portion of the schema

10. Hudson, 11/20/2007 10 Performance Complex queries can be slow and need to be tuned carefully Complex queries can be slow and need to be tuned carefully With Oracle, doing the DOM actions (like XSL transforms) is fastest in the database With Oracle, doing the DOM actions (like XSL transforms) is fastest in the database Also use the profiler in Oracle Also use the profiler in Oracle And using some fancy stuff, like global temp tables And using some fancy stuff, like global temp tables Typical monthly run for us takes about 30 minutes Typical monthly run for us takes about 30 minutes

11. Hudson, 11/20/2007 11 Transformations Your codes may not match WQX codes Your codes may not match WQX codes For example, our “parameters” are not the same as the WQX “characteristics” For example, our “parameters” are not the same as the WQX “characteristics” In our case, handled by PL/SQL functions and lookup tables In our case, handled by PL/SQL functions and lookup tables Some can get complex Some can get complex

12. Hudson, 11/20/2007 12 Tracking Changes What needs to be sent? What needs to be sent? What shouldn’t be sent? What shouldn’t be sent? –Non-reportable characteristic –Legacy –Macroinvertebrate (for now) –Anything that would cause a WQX error How to identify deletes? How to identify deletes? Transaction table for everything sent Transaction table for everything sent Compare the date last sent with the date last changed Compare the date last sent with the date last changed Deletes are “sent but no longer exist” Deletes are “sent but no longer exist”

13. Hudson, 11/20/2007 13 Submission Process Select Records to be Sent Select Records to be Sent Create the XML document Create the XML document Mark those Records as Sent Mark those Records as Sent Zip the document Zip the document Test with the Schematron site (optional) Test with the Schematron site (optional) Submit to CDX Submit to CDX Download/Review the Processing Report Download/Review the Processing Report

14. Hudson, 11/20/2007 14 What to Take Home It is possible to make this work, we’ve done it It is possible to make this work, we’ve done it It helps to be a little desperate since that makes resources available It helps to be a little desperate since that makes resources available It’s critical to know your own data well It’s critical to know your own data well It helps to have experience with your database product and with XML, but those can be learned or hired It helps to have experience with your database product and with XML, but those can be learned or hired I’m sorry to say that it will be a “learning experience”. I’m sorry to say that it will be a “learning experience”.