NHD Update Process Improvement Project U.S. Department of the Interior U.S. Geological Survey Kevin McNinch, Dave Hughes, Paul Kimsey

Current NHD Tools NHD Geo Edit v3.3.2 –Official release 04/18/2010 –ArcGIS 9.3 sp1 & ArcGIS –NHD Model 1.06

Current NHD Tools NHD Geo Edit v3.3.3 –Official release planned 06/07/2010 –ArcGIS 9.3 sp1 & ArcGIS –Compatible with NHD Model 1.07 Permanent Identifiers & ComID (retired in 1 year) External Crosswalk table –Backward compatibility to Model 1.06 –Compatible with NHD Model 2.0 WBD replaces old 1:250k hydrologic units

NHD Update Process Improvements WHYWHATHOWBENEFITSMID TERM LONG TERM Short Term 0-1 Years

NHD Update Process Improvements

What do the Stewards Say: 1.Maintenance is too difficult. 2.Inconsistent results after editing. 3.Need “direct” editing of the NHD data. 4.Quick and simple edits need to be easy. 5.Software installation and performance is an issue.

0 Status of Stewardship Today (Sub basins)

The Net Result After five years of stewardship, the program is not as advanced as it should be.

NHD Update Process Improvements

NHD Update Process Improvement Project Charter –02/01/2010 Survey –Distributed to all partners –Compiled results from 23 respondents Requirements Document –Draft April, 2010

Process Enhancements, Requirements Specifications: Ranked in order of High to Low: Refactor code for NHD tools and utilities Single Installation Package for all NHD Update Process Tools and Utilities Merge all current NHD toolbars into one toolbar with logical progression of tasks

Requirements Specifications: Utilize ArcSDE Versioning to manage NHD Update Process data transactions Allow editing with NHD GeoEdit Tool in geographic coordinate system Create queue for QC checks Create queued edit tool

Requirements Specifications: Make all NHD Update Process tools compatible with ESRI Personal Geodatabases (PGDB) and ESRI File Geodatabases (FGDB) Refactor NHD Update Process tools so that they are not reliant on ESRI Task Assistant

Requirements Specifications: Incorporate NHD Plus build and refresh QC checks to support NHD Plus creation Update process workflow to use ESRI ArcGIS Job Tracking Extension (JTX) Improve Metadata Process

Requirements Specifications: Reach Code Allocation utility process updates Bug Fixes and enhancements from NHD development group list Update tools and installation package to work with ArcGIS 10.0 when released

NHD Update Process Current process (29 steps)

NHD Stewardship Web site Extract from database NHD75464.zip UnZip 1804_orig 1804_orig.mdb Save zip file in this folder 1804_alb 1804_alb.mdb All edits performed on this mdb 1804_qc 1804_qc1.mdb Copy of NHD75464.mdb with transactions applied 1804_geo 1804_geo.mdb 1804_alb.mdb reprojected to geographic 1804_geo_load 1804_geo_load.mdb Used for build flow, releveler and final XML extract Reproject XML Extract XML2pgdb Yes No Make copy of final 1804_geo.mdb and name it 1804_geo_load.mdb Flow Diagram for Current NHD Maintenance Process Corrections ?

NHD Stewardship Web site Extract from database NHD75464.zip UnZip 1804_orig 1804_orig.mdb Save zip file in this folder 1804_alb 1804_alb.mdb All edits performed on this mdb 1804_qc 1804_qc1.mdb Copy of NHD75464.mdb with transactions applied 1804_geo 1804_geo.mdb 1804_alb.mdb reprojected to geographic 1804_geo_load 1804_geo_load.mdb Used for build flow, releveler and final XML extract Reproject XML Extract XML2pgdb Yes No Make copy of final 1804_geo.mdb and name it 1804_geo_load.mdb Steps that can be eliminated from the current NHD Maintenance Process Corrections ?

National Map Replica Checkout Flow Diagram for NHD Maintenance Process after Improvements Replica Checkin Run QC suite against entire dataset Resolve QC errors (Queued Edit tool) Update using NHDGeoEdit tool Run QC suite against entire dataset Resolve QC errors (Queued Edit tool)

NHD Update Process Improvements

Agile Project Management - SCRUM Requirements are captured as items in a list of “product backlog” Iterative development in a series of four-week “sprints” Product is designed, coded, and tested during the sprint Responsive to changing needs NHD Update Process Improvements

Product Noise Level Simple Complex Anarchy Complicated Technology Requirements Far from Agreement Close to Agreement Close to Certainty Far from Certainty Source: Strategic Management and Organizational Dynamics by Ralph Stacey in Agile Software Development with Scrum by Ken Schwaber and Mike Beedle.

Putting it all together Image available at

No changes during a sprint Plan sprint durations around how long you can commit to keeping change out of the sprint Change

Scrum framework Product owner ScrumMaster Team Roles Sprint planning Sprint review Daily scrum meeting Ceremonies Product backlog Sprint backlog Burndown charts Artifacts

NHD Update Process Improvements

- Level of complexity allows for a much broader audience from “expert” to “knowledgeable users” - Reduces training time - Reduces need for constant technical support -Eliminates re-training

Where Does NHD Stewardship Go From Here? Looking beyond lifecycle of current NHD GeoEdit Tools and NHD Stewardship Processes –Current development will be as reusable as possible Begin planning and requirements gathering— create project charter to reflect this work –Completed NHD Architecture Planning Meeting May 13, 2010 Analysis of current process (On-going) Create NHD Update Process that: –Addresses a broader spectrum of NHD users –Simplifies

NHD User Base Huge number of Casual Users Many Knowledgeable Users Some Experts

NHD Stewardship Addressed Very Well with Edit Tool Complete Edit Capability Poorly Addressed No Intermediate Tool Basic Editing Not Yet Addressed Web Edit Tool Identify and Markup Edits In development Experts Casual Users Knowledgeable Users

NHD Update Process Web-Based Markup/Editing Tools –Alabama Web Edit Tool(WET) –Requirements gathering group ongoing for national web markup/editing tool Detailed requirements gathering and analysis Improve web services

NHD Update Process Evaluating other technologies, trends, and tools for future of Vector Data Stewardship –Open source or COTS tools –Open data standards –Geosyncronization –Volunteered Geographic Information (VGI)