Tips and tricks for bridging platforms Python, Data and You! Tips and tricks for bridging platforms

AGENDA Python And the ERSI Platform Key Items in ESRI Platform Connection Files ArcPy Cursors Branching out with Python Libraries Pyodbc Cx_Oracle Sqlite – A hidden gem Code Focus – Example 1 Code Focus – Example 2

It’s All Connected… AGOL ArcServer / Portal Open Source Shapefiles Flat Files ArcSDE

Connection Files: Gateways Between Platforms External Data Oracle ArcSDE ArcServer/ Portal For ArcGIS AGOL MS SQL Server

Updates or deletes rows ArcPy Cursors Cursor Function Code arcpy.da.SearchCursor Read-only access arcpy.da.InsertCursor Inserts rows arcpy.da.UpdateCursor Updates or deletes rows

import arcpy fc = ‘C:/some.gdb/buildings’ fields = [ ‘OID@’, ‘SHAPE@XY’, ‘STRUCTURE’ ] # Other Options: SHAPE@WKT, SHAPE@WKB, SHAPE@JSON with arcpy.da.SearchCursor(fc, fields) as cursor: for row in cursor: # Perform some task

import arcpy row_values = [( ‘Barn’, (-84.005, 33.002)), ( ‘House’, (-84.899, 33.556))] # Open an InsertCursor cursor = arcpy.da.InsertCursor(‘C:/some.gdb/building', [ ‘STRUCTURE’, ‘SHAPE@XY’ ]) # Insert new rows that include point nane and a x,y coordinate # pair that represents the structure's center for row in row_values: cursor.insertRow(row) # Delete cursor object del cursor

import arcpy fc = ‘c:/some.gdb/buildings’ fields = [‘OID@’, ‘SHAPE@XY’, ‘STRUCTURE’] # Create update cursor for feature class with arcpy.da.UpdateCursor(fc, fields) as cursor: for row in cursor: row[2] = row[2].upper() # Update the cursor with the updated list cursor.updateRow(row)

Python Library - PyODBC Pyodbc is an open source Python module specifically designed for connecting to ODBC databases. Based on the DB API 2.0 specification, this library is also packed with even more Pythonic convenience. Pyodbc can connect to these Microsoft databases:

How to use PyODBC # Import python library import pyodbc # Use the specific connection below for each Microsoft database type. # MS Access 2000 connection DBfile = ‘C:/SomeMdb.mdb’ cnmdb = pyodbc.connect(‘‘‘DRIVER={Microsoft Access Driver (*.mdb)};DBQ=’’’ +Dbfile)

PyODBC continued… # Sql Server Express Connection. Use FQDN for remote server connection cnxpress = pyodbc.connect(‘‘‘Driver= {SQL Server Native Client 10.0}; Server=Instance Name; Database=DB Name; Uid=User; Pwd=Password;’’’)

PyODBC continued… # MS SQL Server Connection. Use FQDN for remote server connection cnmsql = pyodbc.connect(‘‘‘DRIVER={SQL Server}; SERVER=ServerName; DATABASE=DB Name; UID=User; PWD=Password’’’)

PyODBC continued… # MS SQL Server Connection cnazure = pyodbc.connect(‘‘‘Driver= {SQL Server Native Client 10.0}; Server= tcp:ServerName.database.windows.net; Database=DB NAME; Uid=[LoginForDb]@[serverName]; Pwd=myPassword;Encrypt=yes;’’’) cur = ConnChoice.cursor()

Python Library – cx_Oracle # Import python library import cx_Oracle # Create connection to Oracle DB. Versions 10g, 11g, and 12c are applicable. Use FQDN for remote server connection. con = cx_Oracle.connect(‘‘‘scott/tiger@server:port/instance’’’) # Create database cursor cur = con.cursor()

What is SQLite? SQLite is an open source transactional database that supports ANSI SQL 92. Why use SQLite with GIS? SQLIte can be use as a traditional database on disk or invoked solely in memory. Operations like joins are handled in a single transaction versus multiple steps.

How to use SQLite # Loads python library import arcpy # Execute CreateSQLiteDatabase for disk based database arcpy.CreateSQLiteDatabase_management(‘C:/data/example.sqlite’, ‘SPATIALITE’)

How to use SQLite # Loads python library import sqlite3 as lite # Creating a SQLite DB connection in RAM con = lite.connect(‘:memory:’) # Generate cursor cur = con.cursor()

“If you want to succeed you should strike out on new paths, rather than travel the worn paths of accepted success.” John D. Rockefeller

Code Focus Bridging Platforms ArcPy SQLite Python Resources Used Take the provided CSV file and create a point feature class using the coordinates provide. Be sure to validate all text attributes for spacing issues before creating feature class. The final schema should look like the following: ArcPy SQLite Python Resources Used Attribute Name Attribute Type Attribute Size State_Terr Text 75 Capital Latitude Double - Longitude Conus 3 Type 10

Case Example for Bridging Platforms

import arcpy import datetime import sqlite3 as lite # Scripting environment variables arcpy.env.workspace = r‘C:\Shrug2017’ arcpy.env.overwriteOutput = True csv = r‘C:\Shrug2017\Demo1\states_terr.csv’ fields = [‘State_Terr’,‘Capital’,‘Latitude’,‘Longitude’,‘Conus’, ‘Type’] # Optional: fields = [‘*’]

# Creating a SQLite DB in RAM con = lite.connect(‘:memory:’) # Generate cursor cur = con.cursor() print(‘Destination database created –’+ str(datetime.datetime.now())) # Creates a table structure for incoming data cur.execute(‘‘‘CREATE TABLE States(StateNme TEXT, Capital TEXT, Lat REAL, Long REAL, Conus TEXT, Typ TEXT);’’’)

print(‘Working table created –’+ str(datetime.datetime.now())) with arcpy.da.SearchCursor(csv, fields) as cursor: for row in cursor: # Inserting records to SQLite DB to_db = [row[0],row[1],row[2],row[3],row[4],row[5] ] cur.execute(‘‘‘INSERT INTO States(StateNme,Capital,Lat,Long, Conus,Typ) VALUES(?,?,?,?,?,?);’’’, to_db) con.commit()

print(‘CSV inserted into SQLite database table –’ + str(datetime print(‘CSV inserted into SQLite database table –’ + str(datetime.datetime.now())) #Removing white spacing from columns with text data type cur.execute(‘‘‘UPDATE States SET StateNme = TRIM(StateNme), Capital = TRIM(Capital), Conus = TRIM(Conus), Typ = TRIM(Typ);’’’) con.commit() print(‘Validation process completed – ’+ str(datetime.datetime.now()))

cur.execute(‘‘‘ALTER TABLE States ADD ShapeXY TEXT’’’) cur.execute(‘‘‘UPDATE States SET ShapeXY = ‘POINT’||‘(’ ||Long|| ‘ ’||Lat|| ‘)’ ’’’) con.commit() print(‘Spatial column created – ’+ str(datetime.datetime.now())) output_path = r‘C:\Shrug2017\Scratch.gdb’ out_name = ‘us_states_terr’ geometry_type = ‘Point’

wkt = ‘‘‘GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984', SPHEROID['WGS_1984',6378137.0,298.257223563]], PRIMEM['Greenwich',0.0], UNIT['Degree',0.0174532925199433]], VERTCS['WGS_1984',DATUM['D_WGS_1984', PARAMETER['Vertical_Shift',0.0], PARAMETER['Direction',1.0],UNIT['Meter',1.0]]; -400 -400 1000000000;-100000 10000;-100000 10000;8.98315284119522E-09; 0.001;0.001;IsHighPrecision’’’ sr = arcpy.SpatialReference()

sr.loadFromString(wkt) result = arcpy.CreateFeatureclass_management(output_path, out_name,geometry_type, ‘’, ‘DISABLED’,‘DISABLED’,sr) fcOut = result.getOutput(0) arcpy.AddField_management(fcOut,‘State_Terr’,‘TEXT’,‘’,‘’,75) arcpy.AddField_management(fcOut,‘Capital’,‘TEXT’,‘’,‘’,75) arcpy.AddField_management(fcOut,‘Latitude’,‘DOUBLE’, 8,‘’, ‘’, ‘’, ‘NULLABLE’)

arcpy.AddField_management(fcOut,‘Longitude’,‘DOUBLE’, 8,‘’,‘’,‘’, ‘NULLABLE’) arcpy.AddField_management(fcOut,‘Conus’,‘TEXT’,‘’,‘’, 3) arcpy.AddField_management(fcOut,'Type','TEXT',‘’,‘’, 10) print( ‘Shell featureclass created – ’+ str(datetime.datetime.now()) ) schema = [ ‘SHAPE@WKT’,‘State_Terr’,‘Capital’, ‘Latitude’,‘Longitude’,‘Conus’,‘Type’ ] curESRI = arcpy.da.InsertCursor(fcOut,schema)

# Query to extract data from SQLITE DB table rows = cur.execute(‘‘‘SELECT ShapeXY,StateNme,Capital, Lat,Long,Conus,Typ FROM States’’’) for row in rows: curESRI.insertRow(row) # close insert cursor del curESRI # close SQLite database connection con.close() print(‘SQLite table insertion completed –’+ str(datetime.datetime.now()))

The point data that is displayed in ArcGIS Pro was created in 0 The point data that is displayed in ArcGIS Pro was created in 0.004 of second. The table below depicts the processing times for three different configurations Arc Map Arc Pro SQLite 1.47 s 0.39 s 0.004 s

Prior Steps Before Starting Demo2 Putting It All Together… Code Focus Prior Steps Before Starting Demo2 Create a MS SQL Server Express database using the SQL scripts provided. Modified the previous script to add data to a MS SQL Server Express database table. Upload ‘states_update1.csv’ to the temp table in MSSQL Express PyODBC Python Resources Used

import pyodbc import os import datetime print(‘Starting Update Process- ’ + str(datetime.datetime.now())) # Sql Server connection cnmsql = pyodbc.connect(‘DRIVER={SQL Server}; Server=Server; Database=ShrugDemo; UID=ScrptKng;PWD=python!’) # Create cursor cur = cnmsql.cursor()

# Issue update commands for states counts = cur.execute(‘‘‘MERGE INTO dbo.States_Territories USING dbo.States_Delta ON dbo.States_Territories.State_Terr = dbo.States_Delta.State_Terr WHEN MATCHED THEN UPDATE SET Capital = dbo.States_Delta.Capital, Latitude = dbo.States_Delta.Latitude, Longitude = dbo.States_Delta.Longitude, Conus = dbo.States_Delta.Conus Type = dbo.States_Delta.Type

WHEN NOT MATCHED THEN INSERT(State_Terr,Capital,Latitude, Longitude,Conus,Type) VALUES(dbo.States_Delta.State_Terr, dbo.States_Delta.Capital, dbo.States_Delta.Latitude, dbo.States_Delta.Longitude, dbo.States_Delta.Conus, dbo.States_Delta.Type);’’’).rowcount print(‘Number of rows added or updated: ’+ str(counts)+ ‘-’ + str(datetime.datetime.now()))

# Commits transactions to database cur.commit() # Closing cursor and database connection cur.close() cnmsql.close()

Thank you for your attention! Daniel Thomas ThomasD@leoncountyfl.gov TLCGIS.org Goto http://shrug-gis.org/workshop/Workshop/Presentation-Resources for this presentation, scripts, code snippets, and more!