Development of the SMC Data Portal Presentation to the SMC Executive Committee June 7, 2017 Steve Steinberg - SCCWRP

Data Management Challenges The SMC produces a lot of data SMC data are not currently available in a single location Limited ability to accommodate new data sources/types and interact with other data systems Data queries and information request are handled manually The SMC program and member agencies spend a substantial time and money on data management Approximately 80% of SMC facilitation budget

SMC Data Portal needs Tools to improve data sharing, integration and dissemination are consistently ranked as a high priority by the Executive Committee More effective data uploads Data QA/QC and core computations on submission More efficient data retrieval Web-based data portal with data query, download and reporting Better, faster, cheaper Quality data in less time, at lower cost and effort Capability to link with other data systems using open data standards and web services CEDEN, TMDL data, BMP/LID, data from other agencies

Key Features Simple web-based interface No specialized software needed Single location to submit and retrieve data Requires little to no training Leverages existing, industry standard tools and data structures Reduced development costs Future updates are automatic and don’t disrupt workflow Automated tools QA/QC and data checking On-the-fly computations of commonly required values and indices Easy access to data products and reporting Open data, web service approach allows easy linkages to other online data systems, labs, etc. Includes tools many partners already have and use

Open data – becoming expected Government Technology: June 6, 2017 AB 1755: The Open and Transparent Water Data Act Sept. 23, 2016 http://www.govtech.com/data/Visualization-Takes-Open-Data-to-the-Next-Level.html

Recommendation at last meeting Develop the core system - $155,000 Data portal architecture Web-based data input/output (including QA/QC, query and retrieval) Incorporate regional stream monitoring data Requested add-on - $85,000 Automated data reporting and visualization CEDEN data submission - $30,000 One-time, manual loading Under consideration - $50,000 Wet weather monitoring data

CEDEN considerations At the March meeting concern raised about means to ensure SMC data is provided to CEDEN without requiring members to make two separate submissions. We CAN make the submission There are no technical barriers This decision is NOT straightforward CEDEN is not stable (changes on the horizon) Does not accept all data types The approach selected by SMC should account for future changes to CEDEN

CEDEN Interaction options Three options for loading SMC data to CEDEN: Format data into CEDEN templates and load to submission system (one-time manual push) – $30,000 Develop automated tools to structure and load SMC data into current CEDEN system (auto push – outdated soon) – $25,000 (+$30,000 initial manual push = $55,000) Delay automated tools until after updated CEDEN system is defined (auto push – stable with time) – $25,000 (+$30,000 initial manual push = $55,000; OR all the above = $80,000)

Main tasks Year 1: Development of Data Portal for all SMC data Task 1: Develop architecture for the data portal [core] Task 2: Develop data query tools [core] Task 3: Develop automated reporting tools [requested add-on] Year 2: Incorporate regional monitoring data in Portal Task 4: Incorporate regional stream monitoring data [core] Task 5: One time upload of regional monitoring data into CEDEN Task 6: Incorporate wet weather monitoring data [under consideration]

Recommendation at last meeting Develop the core system - $155,000 Data portal architecture Web-based data input/output (including QA/QC, query and retrieval) Incorporate regional stream monitoring data Requested add-on - $85,000 Automated data reporting and visualization CEDEN data submission - $30,000 One-time, manual loading Under consideration - $50,000 Wet weather monitoring data