Programming and Automation

Agenda Reason for Data automation ModelBuilder Python Windows Scheduler

Strength of GIS is analysis Automation builds on that strength Allows the user to control the process but the system runs the process. Scripts, models and programs

Reasons for automation Easier Faster More Accurate Repeatable

In the past ... ESRI AML – Arc Macro Language SML – Simple Macro Language Avenue – ArcView 3.2 – C, C++ - Slowly fading Visual Basic – Microsoft says bye bye Perl Awk

Currently ESRI Modelbuilder Python (Arcpy) ArcObjects Java Silverlight FME

ESRI Simplified process means you don't need a developer to automate tasks ESRI has 3 way of automating tasks Model Builder (Any level) Python Script (Any Level) ArcObjects (ArcServer required)

ArcToolbox

ArcToolbox Tools that automate a single task Buffer Union Projections Easy to use and can be accessed in multiple locations (ArcMap, ArcCatalog, ArcToolbox) Restricted by license

ArcToolbox Contains ESRI tools/scripts 3rd party tools/scripts Personal tools/scripts

Model Builder

Modelbuilder Integrated with ArcToolbox Chaining tools together No programming needed Can run a single iterator Workflows can be very simple to very complex

Models are created and saved within a custom toolbox. The toolbox can contain your favourite tools or a set of tools that are used within you model, or just your model

Models are created by dragging and dropping tools into the Model Builder Can contain Multiple processes. Parameters can be added that require user input Output workspaces can be controlled

Uses Geoprocessing Data Management Cartography/Mapping Buffer Clip Unions Data Management Add xy Calculate fields topology Cartography/Mapping Representations Data driven pages

Geoprocessing simple Probably wouldn’t build a model for this

Geoprocessing

Geoprocessing using a simple iterator (loop)

3 processes in 1 model

Administrative

Models are not quite fully automated Require user input/Parameters Someone has to press the button to run them Remember to add meta data Models can be exported to Python

Python

Python The preferred ArcGIS scripting language Others can be used Executes a sequence of steps Can use conditional logic Can include numerous iteration (loops) Can be generated from Model builder or built independently.

What is Python High level language used to automate tasks through programs called scripts A good beginning language Logic is fairly simple to follow Works on a variety of systems (Windows, Linux, Unix) A version of python is installed when ArcGIS is installed

Basic Python X = 5 Y = 3 Print X + Y 8 First = “John” Last = “Masich” Print “First Last” JohnMasich

Lists >>>suits = ['Spades', 'Clubs', 'Diamonds', 'Hearts'] >>>values = ['Ace', 2, 3, 4, 5, 6, 7, 8, 9, 10, 'Jack', 'Queen', 'King'] >>>print suits[0] Spades >>>print values[12] King

Loops >>> x = 2 >>> multipliers = [1,2,3,4] >>> for num in multipliers: print x * num 2 4 6 8

Decision structures >>> x = 3 >>> if x > 2: ... print "Greater than two" ... Greater than two

The Python Window Accessed through ArcGIS Commands are typed into the window >>> import arcpy >>> arcpy.Buffer_analysis(“Nests”, “Buffered_Nests”, “100 meters”, “”,””,”ALL”) Script can be loaded from an external source

Python in ArcMap arcpy – lets python know you are working with ArcGIS tools and features. Without it the commands won’t work Buffer_analysis – refers to the tool that you are running. Data after that reflects the information that needs to be entered to run the tool Toolbox – Analysis Tools - Buffer

# This script runs the Buffer tool # This script runs the Buffer tool. The user supplies the input # and output paths, and the buffer distance. import arcpy arcpy.env.overwriteOutput = True try: # Get the input parameters for the Buffer tool inPath = arcpy.GetParameterAsText(0) outPath = arcpy.GetParameterAsText(1) bufferDistance = arcpy.GetParameterAsText(2) # Run the Buffer tool arcpy.Buffer_analysis(inPath, outPath, bufferDistance) # Report a success message arcpy.AddMessage("All done!") except: # Report an error messages arcpy.AddError("Could not complete the buffer") # Report any error messages that the Buffer tool might have generated arcpy.AddMessage(arcpy.GetMessages())

Block merge # --------------------------------------------------------------------------- # Block merge.py # Created on: 2013-02-19 16:37:05.00000 # (generated by ArcGIS/ModelBuilder) # Description: # Import arcpy module import arcpy # Set Geoprocessing environments arcpy.env.scratchWorkspace = "C:\\Users\\KFM\\Documents\\ArcGIS\\Default.gdb" arcpy.env.workspace = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb" # Local variables: CFP = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\a15384_cfp_blocks.shp" CFP_Reserves = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\ExternalReserves_A15384.shp" TKD = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\tkd_consultation_12-18-2012\\tkd_consultation_blobs_12_18_2012.shp" Mac_Fibre = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\macfibre\\A87345_Development.shp" Conifex = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Conifex\\Conifex_Mackenzie_Blocks_for_KFM_130124.shp" FTA = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\FTA\\FTN_C_B_PL_polygon.shp" mac_tsa = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Base\\mac_tsa" licencee_blocks = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\licencee_blocks" RESULTS = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\results_blocks" Original_FDU_clip = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\FSP\\Original_FDU_clip" CFP_2 = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\a15384_cfp_blocks.shp" Output_Feature_Class = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\a15384_cfp_blocks.shp" a15384_cfp_blocks_shp = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\a15384_cfp_blocks.shp" CFP_BLOCKS = "C:\\Users\\KFM\\Documents\\ArcGIS\\Default.gdb\\CFP_BLOCKS" Output_Feature_Class__3_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\tkd_consultation_12-18-2012\\tkd_consultation_blobs_12_18_2012.shp" Output_Feature_Class__2_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\macfibre\\A87345_Development.shp" FTA_Blocks_shp = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\FTA\\FTA_Blocks.shp" FTA_Blocks_shp__2_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\FTA\\FTA_Blocks.shp" Output_Feature_Class__7_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\licencee_blocks" Output_Feature_Class__4_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Conifex\\Conifex_Mackenzie_Blocks_for_KFM_130124.shp" results_blocks__2_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\results_blocks" Licensee_Blocks_Copy = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\Licensee_Blocks_Copy" licencee_blocks__2_ = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\licencee_blocks" licencee_blocks_clip = "F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\licencee_blocks_clip" # Process: Copy arcpy.Copy_management(licencee_blocks, Licensee_Blocks_Copy, "FeatureClass") # Process: Clip arcpy.Clip_analysis(FTA, mac_tsa, FTA_Blocks_shp, "") # Process: Add Field (4) arcpy.AddField_management(FTA_Blocks_shp, "COMPANY", "TEXT", "", "", "15", "", "NON_NULLABLE", "NON_REQUIRED", "") # Process: Calculate Field (4) arcpy.CalculateField_management(FTA_Blocks_shp__2_, "COMPANY", "\"FTA\"", "VB", "") # Process: Add Field (2) arcpy.AddField_management(Mac_Fibre, "COMPANY", "TEXT", "", "", "15", "", "NON_NULLABLE", "NON_REQUIRED", "") # Process: Calculate Field (3) arcpy.CalculateField_management(Output_Feature_Class__2_, "COMPANY", "\"MFL\"", "VB", "") # Process: Add Field (3) arcpy.AddField_management(TKD, "COMPANY", "TEXT", "", "", "15", "", "NON_NULLABLE", "NON_REQUIRED", "") # Process: Calculate Field (2) arcpy.CalculateField_management(Output_Feature_Class__3_, "COMPANY", "\"TKD\"", "VB", "") # Process: Append arcpy.Append_management("F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\ExternalReserves_A15384.shp", CFP, "NO_TEST", "BLOCK_ID \"BLOCK_ID\" true false false 20 Text 0 0 ,First,#,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Canfor\\ExternalReserves_A15384.shp,BLOCK_ID,-1,-1;LICENCE \"LICENCE\" true false false 15 Text 0 0 ,First,#;DIVISION_N \"DIVISION_N\" true false false 254 Text 0 0 ,First,#;AREA_HA \"AREA_HA\" true false false 19 Double 8 18 ,First,#", "") # Process: Add Field arcpy.AddField_management(CFP_2, "COMPANY", "TEXT", "", "", "15", "", "NON_NULLABLE", "NON_REQUIRED", "") # Process: Calculate Field arcpy.CalculateField_management(Output_Feature_Class, "COMPANY", "\"CFP\"", "VB", "") # Process: Dissolve arcpy.Dissolve_management(a15384_cfp_blocks_shp, CFP_BLOCKS, "COMPANY", "", "MULTI_PART", "DISSOLVE_LINES") # Process: Add Field (6) arcpy.AddField_management(RESULTS, "COMPANY", "TEXT", "", "", "15", "", "NULLABLE", "NON_REQUIRED", "") # Process: Calculate Field (6) arcpy.CalculateField_management(results_blocks__2_, "COMPANY", "\"RSLT\"", "VB", "") # Process: Add Field (5) arcpy.AddField_management(Conifex, "COMPANY", "TEXT", "", "", "15", "", "NON_NULLABLE", "NON_REQUIRED", "") # Process: Calculate Field (5) arcpy.CalculateField_management(Output_Feature_Class__4_, "COMPANY", "\"CFF\"", "VB", "") # Process: Delete Features arcpy.DeleteFeatures_management(licencee_blocks) # Process: Append (2) arcpy.Append_management("F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\FTA\\FTA_Blocks.shp;F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\macfibre\\A87345_Development.shp;F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\tkd_consultation_12-18-2012\\tkd_consultation_blobs_12_18_2012.shp;C:\\Users\\KFM\\Documents\\ArcGIS\\Default.gdb\\CFP_BLOCKS;F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\results_blocks;F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Conifex\\Conifex_Mackenzie_Blocks_for_KFM_130124.shp", Output_Feature_Class__7_, "NO_TEST", "SHAPE_Length \"SHAPE_Length\" false true true 8 Double 0 0 ,First,#,C:\\Users\\KFM\\Documents\\ArcGIS\\Default.gdb\\CFP_BLOCKS,Shape_Length,-1,-1,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\results_blocks,SHAPE_Length,-1,-1;SHAPE_Area \"SHAPE_Area\" false true true 8 Double 0 0 ,First,#,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\tkd_consultation_12-18-2012\\tkd_consultation_blobs_12_18_2012.shp,Shape_Area,-1,-1,C:\\Users\\KFM\\Documents\\ArcGIS\\Default.gdb\\CFP_BLOCKS,Shape_Area,-1,-1,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\results_blocks,SHAPE_Area,-1,-1;COMPANY \"COMPANY\" true true false 10 Text 0 0 ,First,#,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\macfibre\\A87345_Development.shp,COMPANY,-1,-1,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\tkd_consultation_12-18-2012\\tkd_consultation_blobs_12_18_2012.shp,COMPANY,-1,-1,C:\\Users\\KFM\\Documents\\ArcGIS\\Default.gdb\\CFP_BLOCKS,COMPANY,-1,-1,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\ThreeFeathersRecce.gdb\\Licensee\\results_blocks,COMPANY,-1,-1,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\Conifex\\Conifex_Mackenzie_Blocks_for_KFM_130124.shp,COMPANY,-1,-1,F:\\GIS\\Projects\\2013\\2012_50_ThreeFeathers\\LicenseeData\\FTA\\FTA_Blocks.shp,COMPANY,-1,-1;AREA_HA \"AREA_HA\" true true false 8 Double 0 0 ,First,#", "") # Process: Clip (2) arcpy.Clip_analysis(licencee_blocks__2_, Original_FDU_clip, licencee_blocks_clip, "")

PyWin or IDLE Outside of the ArcMap environment you can work with python in the IDLE program or use an open source such as PYWIN. Friendlier programming environment. Create, edit and execute tools

Running Scripts through Python scripts can be run through a basic program such as Windows task scheduler. Models Can’t be run this way This allows the user to run memory intensive or longer scripts out side of normal work hours

Management Perspective Cost effective Controlled output Less errors Data is where it should be. Create more confidence in the results Doesn’t necessarily mean the are correct! Process is easily transferred between users