1. Tips and tricks for bridging platforms
Python, Data and You!
Tips and tricks for bridging platforms

2. AGENDA Python And the ERSI Platform Key Items in ESRI Platform
Connection Files
ArcPy Cursors
Branching out with Python Libraries
Pyodbc
Cx_Oracle
Sqlite – A hidden gem
Code Focus – Example 1
Code Focus – Example 2

3. It’s All Connected… AGOL ArcServer / Portal Open Source Shapefiles
Flat Files
ArcSDE

4. Connection Files: Gateways Between Platforms
External
Data
Oracle
ArcSDE
ArcServer/ Portal For ArcGIS
AGOL
MS SQL
Server

5. Updates or deletes rows
ArcPy Cursors
Cursor
Function
Code
arcpy.da.SearchCursor
Read-only access
arcpy.da.InsertCursor
Inserts rows
arcpy.da.UpdateCursor
Updates or deletes rows

6. import arcpy
fc = ‘C:/some.gdb/buildings’
fields = [ ‘STRUCTURE’ ]
# Other Options:
with arcpy.da.SearchCursor(fc, fields) as cursor:
for row in cursor:
# Perform some task

7. import arcpy
row_values = [( ‘Barn’, ( , )),
( ‘House’, ( , ))]
# Open an InsertCursor
cursor = arcpy.da.InsertCursor(‘C:/some.gdb/building',
[ ‘STRUCTURE’, ])
# Insert new rows that include point nane and a x,y coordinate
# pair that represents the structure's center
for row in row_values:
cursor.insertRow(row)
# Delete cursor object
del cursor

8. import arcpy
fc = ‘c:/some.gdb/buildings’
fields = ‘STRUCTURE’]
# Create update cursor for feature class
with arcpy.da.UpdateCursor(fc, fields) as cursor:
for row in cursor:
row[2] = row[2].upper()
# Update the cursor with the updated list
cursor.updateRow(row)

9. Python Library - PyODBC
Pyodbc is an open source Python module specifically designed for connecting to ODBC databases. Based on the DB API 2.0 specification, this library is also packed with even more Pythonic convenience.
Pyodbc can connect to these Microsoft databases:

10. How to use PyODBC # Import python library import pyodbc
# Use the specific connection below for each Microsoft
database type.
# MS Access 2000 connection
DBfile = ‘C:/SomeMdb.mdb’
cnmdb = pyodbc.connect(‘‘‘DRIVER={Microsoft Access Driver
(*.mdb)};DBQ=’’’ +Dbfile)

11. PyODBC continued…
# Sql Server Express Connection. Use FQDN for remote server connection
cnxpress = pyodbc.connect(‘‘‘Driver=
{SQL Server Native Client 10.0};
Server=Instance Name;
Database=DB Name;
Uid=User; Pwd=Password;’’’)

12. PyODBC continued…
# MS SQL Server Connection. Use FQDN for remote server connection
cnmsql = pyodbc.connect(‘‘‘DRIVER={SQL Server};
SERVER=ServerName;
DATABASE=DB Name;
UID=User;
PWD=Password’’’)

13. PyODBC continued… # MS SQL Server Connection
cnazure = pyodbc.connect(‘‘‘Driver=
{SQL Server Native Client 10.0};
Server=
tcp:ServerName.database.windows.net;
Database=DB NAME;
Pwd=myPassword;Encrypt=yes;’’’)
cur = ConnChoice.cursor()

14. Python Library – cx_Oracle
# Import python library
import cx_Oracle
# Create connection to Oracle DB. Versions 10g, 11g, and 12c are applicable. Use FQDN for remote server connection.
con =
# Create database cursor
cur = con.cursor()

15. What is SQLite?
SQLite is an open source transactional database that supports ANSI SQL 92.
Why use SQLite with GIS?
SQLIte can be use as a traditional database on disk or invoked solely in memory. Operations like joins are handled in a single transaction versus multiple steps.

16. How to use SQLite # Loads python library import arcpy
# Execute CreateSQLiteDatabase for disk based database
arcpy.CreateSQLiteDatabase_management(‘C:/data/example.sqlite’, ‘SPATIALITE’)

17. How to use SQLite # Loads python library import sqlite3 as lite
# Creating a SQLite DB connection in RAM
con = lite.connect(‘:memory:’)
# Generate cursor
cur = con.cursor()

18. “If you want to succeed you should strike out on new paths, rather than travel the worn paths of accepted success.”
John D. Rockefeller

19. Code Focus Bridging Platforms ArcPy SQLite Python Resources Used
Take the provided CSV file and create a point feature class using the coordinates provide.
Be sure to validate all text attributes for spacing issues before creating feature class.
The final schema should look like the following:
ArcPy
SQLite
Python Resources
Used
Attribute Name
Attribute Type
Attribute Size
State_Terr
Text 75
Capital
Latitude
Double -
Longitude
Conus 3
Type 10

20. Case Example for Bridging Platforms

21. import arcpy
import datetime
import sqlite3 as lite
# Scripting environment variables
arcpy.env.workspace = r‘C:\Shrug2017’
arcpy.env.overwriteOutput = True
csv = r‘C:\Shrug2017\Demo1\states_terr.csv’
fields = [‘State_Terr’,‘Capital’,‘Latitude’,‘Longitude’,‘Conus’,
‘Type’]
# Optional: fields = [‘*’]

22. # Creating a SQLite DB in RAM
con = lite.connect(‘:memory:’)
# Generate cursor
cur = con.cursor()
print(‘Destination database created –’+ str(datetime.datetime.now()))
# Creates a table structure for incoming data
cur.execute(‘‘‘CREATE TABLE States(StateNme TEXT, Capital TEXT, Lat REAL, Long REAL, Conus TEXT, Typ TEXT);’’’)

23. print(‘Working table created –’+ str(datetime.datetime.now()))
with arcpy.da.SearchCursor(csv, fields) as cursor:
for row in cursor:
# Inserting records to SQLite DB
to_db = [row[0],row[1],row[2],row[3],row[4],row[5] ]
cur.execute(‘‘‘INSERT INTO
States(StateNme,Capital,Lat,Long,
Conus,Typ)
VALUES(?,?,?,?,?,?);’’’, to_db)
con.commit()

24. print(‘CSV inserted into SQLite database table –’ + str(datetime
print(‘CSV inserted into SQLite database table –’ + str(datetime.datetime.now()))
#Removing white spacing from columns with text data type
cur.execute(‘‘‘UPDATE States
SET StateNme = TRIM(StateNme),
Capital = TRIM(Capital),
Conus = TRIM(Conus),
Typ = TRIM(Typ);’’’)
con.commit()
print(‘Validation process completed – ’+
str(datetime.datetime.now()))

25. cur.execute(‘‘‘ALTER TABLE States ADD ShapeXY TEXT’’’)
cur.execute(‘‘‘UPDATE States SET ShapeXY = ‘POINT’||‘(’
||Long|| ‘ ’||Lat|| ‘)’ ’’’)
con.commit()
print(‘Spatial column created – ’+ str(datetime.datetime.now()))
output_path = r‘C:\Shrug2017\Scratch.gdb’
out_name = ‘us_states_terr’
geometry_type = ‘Point’

26. wkt = ‘‘‘GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',
SPHEROID['WGS_1984', , ]],
PRIMEM['Greenwich',0.0],
UNIT['Degree', ]],
VERTCS['WGS_1984',DATUM['D_WGS_1984',
PARAMETER['Vertical_Shift',0.0],
PARAMETER['Direction',1.0],UNIT['Meter',1.0]];
; ;
10000; E-09;
0.001;0.001;IsHighPrecision’’’
sr = arcpy.SpatialReference()

27. sr.loadFromString(wkt)
result = arcpy.CreateFeatureclass_management(output_path,
out_name,geometry_type, ‘’,
‘DISABLED’,‘DISABLED’,sr)
fcOut = result.getOutput(0)
arcpy.AddField_management(fcOut,‘State_Terr’,‘TEXT’,‘’,‘’,75)
arcpy.AddField_management(fcOut,‘Capital’,‘TEXT’,‘’,‘’,75)
arcpy.AddField_management(fcOut,‘Latitude’,‘DOUBLE’, 8,‘’, ‘’,
‘’, ‘NULLABLE’)

28. arcpy.AddField_management(fcOut,‘Longitude’,‘DOUBLE’,
8,‘’,‘’,‘’, ‘NULLABLE’)
arcpy.AddField_management(fcOut,‘Conus’,‘TEXT’,‘’,‘’, 3)
arcpy.AddField_management(fcOut,'Type','TEXT',‘’,‘’, 10)
print( ‘Shell featureclass created – ’+ str(datetime.datetime.now()) )
schema = [
‘Latitude’,‘Longitude’,‘Conus’,‘Type’ ]
curESRI = arcpy.da.InsertCursor(fcOut,schema)

29. # Query to extract data from SQLITE DB table
rows = cur.execute(‘‘‘SELECT ShapeXY,StateNme,Capital,
Lat,Long,Conus,Typ FROM States’’’)
for row in rows:
curESRI.insertRow(row)
# close insert cursor
del curESRI
# close SQLite database connection
con.close()
print(‘SQLite table insertion completed –’+
str(datetime.datetime.now()))

30. The point data that is displayed in ArcGIS Pro was created in 0
The point data that is displayed in ArcGIS Pro was created in of second.
The table below depicts the processing times for three different configurations
Arc
Map
Arc Pro
SQLite
1.47 s
0.39 s
0.004 s

31. Prior Steps Before Starting Demo2
Putting It All Together…
Code Focus
Prior Steps Before Starting Demo2
Create a MS SQL Server Express database using the SQL scripts provided.
Modified the previous script to add data to a MS SQL Server Express database table.
Upload ‘states_update1.csv’ to the temp table in MSSQL Express
PyODBC
Python Resources
Used

32. import pyodbc
import os
import datetime
print(‘Starting Update Process- ’ + str(datetime.datetime.now()))
# Sql Server connection
cnmsql = pyodbc.connect(‘DRIVER={SQL Server};
Server=Server;
Database=ShrugDemo;
UID=ScrptKng;PWD=python!’)
# Create cursor
cur = cnmsql.cursor()

33. # Issue update commands for states
counts = cur.execute(‘‘‘MERGE INTO dbo.States_Territories
USING dbo.States_Delta
ON dbo.States_Territories.State_Terr =
dbo.States_Delta.State_Terr
WHEN MATCHED THEN
UPDATE
SET Capital = dbo.States_Delta.Capital,
Latitude = dbo.States_Delta.Latitude,
Longitude = dbo.States_Delta.Longitude,
Conus = dbo.States_Delta.Conus
Type = dbo.States_Delta.Type

34. WHEN NOT MATCHED THEN
INSERT(State_Terr,Capital,Latitude,
Longitude,Conus,Type)
VALUES(dbo.States_Delta.State_Terr,
dbo.States_Delta.Capital,
dbo.States_Delta.Latitude,
dbo.States_Delta.Longitude,
dbo.States_Delta.Conus,
dbo.States_Delta.Type);’’’).rowcount
print(‘Number of rows added or updated: ’+ str(counts)+ ‘-’ + str(datetime.datetime.now()))

35. # Commits transactions to database
cur.commit()
# Closing cursor and database connection
cur.close()
cnmsql.close()

37. Thank you for your attention!
Daniel Thomas
TLCGIS.org
Goto for this presentation, scripts, code snippets, and more!