1. Development of the SMC Data Portal
Presentation to the SMC Executive Committee
June 7, 2017
Steve Steinberg - SCCWRP

2. Data Management Challenges
The SMC produces a lot of data
SMC data are not currently available in a single location
Limited ability to accommodate new data sources/types and interact with other data systems
Data queries and information request are handled manually
The SMC program and member agencies spend a substantial time and money on data management
Approximately 80% of SMC facilitation budget

3. SMC Data Portal needs
Tools to improve data sharing, integration and dissemination are consistently ranked as a high priority by the Executive Committee
More effective data uploads
Data QA/QC and core computations on submission
More efficient data retrieval
Web-based data portal with data query, download and reporting
Better, faster, cheaper
Quality data in less time, at lower cost and effort
Capability to link with other data systems using open data standards and web services
CEDEN, TMDL data, BMP/LID, data from other agencies

4. Key Features Simple web-based interface
No specialized software needed
Single location to submit and retrieve data
Requires little to no training
Leverages existing, industry standard tools and data structures
Reduced development costs
Future updates are automatic and don’t disrupt workflow
Automated tools
QA/QC and data checking
On-the-fly computations of commonly required values and indices
Easy access to data products and reporting
Open data, web service approach allows easy linkages to other online data systems, labs, etc.
Includes tools many partners already have and use

5. Open data – becoming expected
Government Technology: June 6, 2017
AB 1755: The Open and Transparent Water Data Act
Sept. 23, 2016

6. Recommendation at last meeting
Develop the core system - $155,000
Data portal architecture
Web-based data input/output (including QA/QC, query and retrieval)
Incorporate regional stream monitoring data
Requested add-on - $85,000
Automated data reporting and visualization
CEDEN data submission - $30,000
One-time, manual loading
Under consideration - $50,000
Wet weather monitoring data

7. CEDEN considerations
At the March meeting concern raised about means to ensure SMC data is provided to CEDEN without requiring members to make two separate submissions.
We CAN make the submission
There are no technical barriers
This decision is NOT straightforward
CEDEN is not stable (changes on the horizon)
Does not accept all data types
The approach selected by SMC should account for future changes to CEDEN

8. CEDEN Interaction options
Three options for loading SMC data to CEDEN:
Format data into CEDEN templates and load to submission system (one-time manual push) – $30,000
Develop automated tools to structure and load SMC data into current CEDEN system (auto push – outdated soon) – $25,000 (+$30,000 initial manual push = $55,000)
Delay automated tools until after updated CEDEN system is defined (auto push – stable with time) – $25, (+$30,000 initial manual push = $55,000; OR all the above = $80,000)

9. Main tasks Year 1: Development of Data Portal for all SMC data
Task 1: Develop architecture for the data portal [core]
Task 2: Develop data query tools [core]
Task 3: Develop automated reporting tools [requested add-on]
Year 2: Incorporate regional monitoring data in Portal
Task 4: Incorporate regional stream monitoring data [core]
Task 5: One time upload of regional monitoring data into CEDEN
Task 6: Incorporate wet weather monitoring data [under consideration]

10. Recommendation at last meeting
Develop the core system - $155,000
Data portal architecture
Web-based data input/output (including QA/QC, query and retrieval)
Incorporate regional stream monitoring data
Requested add-on - $85,000
Automated data reporting and visualization
CEDEN data submission - $30,000
One-time, manual loading
Under consideration - $50,000
Wet weather monitoring data