1. Automation Using Python for WVDOT GIS Data Management Yueming Wu, Ph.D., GISP GIS Manager May 9, 2012

2. Agenda  Introduction  WVDOT Geospatial Data Repository  Automation Implementation  Key Issues & Solutions  Future Work  Questions & Answers

3. Introduction

4. History Historically the West Virginia Department of Transportation (WVDOT) has been utilizing geospatial (GIS, Remote Sensing, & GPS) technology for planning, analysis, and mapping purposes ever since the technology became available. In 2005 the WVDOT GIS Section was created to meet the growing needs for geospatial data and services in transportation. In 2007 the GIS Section was renamed as the GIS Unit and merged with the Highway Data Services (HDS) Unit, a program responsible for processing changes in roadway information, and the update and maintenance of the Roadway Inventory Log, to form the current Geospatial Transportation Information (GTI) Section.

5. -Roadway Inventory Log -Roadway Statistics -Public Certified Mileage - Geospatial Data Management -Geospatial Services Geospatial Transportation Information (GTI) Section Program Planning & Administration Division Highway Data Services Unit GIS Unit

6. GTI Mission & Vision Statements  Mission Statement To improve customer service to the citizens of West Virginia by supplying the latest geospatial transportation information.  Vision Statement The West Virginia Department of Transportation Geospatial Transportation Information (GTI) Section is committed to managing an enterprise Geospatial Information System that will improve the productivity of the West Virginia Department of Transportation, meet growing customer needs, and play a leading role in West Virginia’s geospatial community.

7. GIS Implementation  Esri Shop  Enterprise License Agreement (ELA)  Esri Technology  Staff  7 GIS Professionals  2 Cartographers  1 GIS Coordinator at District 7  Enterprise GIS  WVDOT Enterprise GIS Roadmap  Enterprise Resource Planning (ERP)/Asset Management System

8. Major GIS Projects in 2012  Enterprise GIS Implementation  New Transportation GIS Data Model  District Level GIS Implementation  Workflow Manager  Transportation GIS Data Collection (MS4, Railroad Crossings, Landslides, Outdoor Advertising, Right-of-Way, Trails, Meta Data, etc.)  Highway Map Reproduction  Web Mapping Applications (STIP, Park & Ride, Functional Classifications, etc.)  Mobile GIS Applications (MS4, etc.)  3D GIS Application Development  GIS Portal Upgrade  ProjectWise Implementation  ERP/Asset Management System Implementation

9. WVDOT Geospatial Data Repository

10. Export Reproject Upload Update Store Download &Add Metadata Version - Edit GTI_EDITING TRANSPORTATION GIS DATA Commissioner Orders Paper Maps Field Data Consultant Reports Etc. TRANSPORTATION GIS DATA Commissioner Orders Paper Maps Field Data Consultant Reports Etc. GTI_PUB_WM GTI_PUB_UTM Upload REFERENCE GIS DATA WV GIS Tech Center WVGES USGS Esri Etc. REFERENCE GIS DATA WV GIS Tech Center WVGES USGS Esri Etc. IMAGERY Aerial Photos Shade Relief Hillshade Topo Etc. IMAGERY Aerial Photos Shade Relief Hillshade Topo Etc. Editing Publication Source Data Shapefiles Imagery GIS Services Shapefiles for Download Store Download & Add Metadata Version - Edit Update Geospatial Analysis Map Production Web Apps Reproject Export WVDOT Geospatial Data Repository

11. Automation Implementation

12.  Four Automated Data Management Operations  GTI_EDITING GDB  GTI_PUB_UTM GDB  Shapefiles  GTI_PUB_UTM GDB  GTI_PUB_UTM GDB  GTI_PUB_WM GDB  GTI_PUB_UTM GDB  Shapefiles for Publications GP1 GP2 GPn GP … Data Model ArcGIS Model Builder Python Script Pythonwin Task Windows Scheduler ExportDeploy Improve

13. Key Issues & Solutions

14. Database Lock  When a GDB is updated, it’s been accessed by clients (ArcGIS Desktop, ArcGIS Server, etc.). Database locks are created accordingly. #Remove Database Locks on GTI_PUB_WM Import pyodbc cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=dotarcgis02;DATABASE=GTI_PUB_UTM;UID=xxxxx;PWD=xxx') cursor = cnxn.cursor() cursor.execute("delete FROM [GTI_PUB_UTM].[dbo].[SDE_process_information]" ) cnxn.commit()

15. Scheduled Task  After a Python script passes test, use Windows Scheduler to create a task so the script would run on a regular basis.  Make the task independent to individual’s login

16. Email Notification?  If a task fails, notify the data repository manager by email #Email import smtplib, MIMEText try: … except: sender = 'admin@wvdohwebserver.com‘ receivers = ['wuyueming@gmail.com', 'wu_yueming@yahoo.com'] msg = MIMEText("The GTI_PUB_UTM_2_Web_Shapefiles script failed. Please check!") msg['Subject'] = 'Python Script Failed' msg['From'] = "Admin@WVDOHGISWebServer.com" msg['Reply-to'] = "GTI Core Team" msg['To'] = "Admin@WVDOHGISWebServer.com" smtpObj = smtplib.SMTP('localhost') smtpObj.sendmail(sender, receivers, msg.as_string())

17. or Log  After a task is created, use a log file to track. #Log Import os, sys, time rootFolder = r"C:\Geoprocessing Scripts\\" # This is where the log file is located text_file = open(rootFolder + "PScriptLogFile.txt", "a") text_file.write("\n\n") text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py began at " + str(time.strftime('%X %x %Z')) + ".\n\n") try: … text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py successfully ended at " + str(time.strftime('%X %x %Z')) + ".") except: text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py failed at " + str(time.strftime('%X %x %Z')) + ".") text_file.close()

18. Future Work

19. Export Reproject Upload Update Store Download &Add Metadata Version - Edit GTI_EDITING TRANSPORTATION GIS DATA Commissioner Orders Paper Maps Field Data Consultant Reports Etc. TRANSPORTATION GIS DATA Commissioner Orders Paper Maps Field Data Consultant Reports Etc. GTI_PUB_WM GTI_PUB_UTM Upload REFERENCE GIS DATA WV GIS Tech Center WVGES USGS Esri Etc. REFERENCE GIS DATA WV GIS Tech Center WVGES USGS Esri Etc. IMAGERY Aerial Photos Shade Relief Hillshade Topo Etc. IMAGERY Aerial Photos Shade Relief Hillshade Topo Etc. Editing Publication Source Data Shapefiles Imagery GIS Services Shapefiles for Download Store Download &Add Metadata Version - Edit Update Geospatial Analysis Map Production Web Apps Reproject Export WVDOT Geospatial Data Repository Replicate

20. Reproject Upload Update GTI_EDITING GTI_PUB_WM GTI_PUB_UTM Upload Editing Publication Shapefiles Update Reproject Headquarters Replicate Districts 1- 10 GTI_PUB_UTM FGDB GTI_PUB_UTM FGDB Publication Replicate WVDOT Geospatial Data Repository Imagery

21. Yueming Wu GIS Manager WVDOT (304) 558-7437 yueming.wu@wv.gov