ArcGIS Geoprocessing Advanced Scripting With Python Corey Tucker Nathan Warmerdam

Workshop Outline Review of scripting languagesReview of scripting languages –What is supported? What is recommended? Why? Data Access with cursorsData Access with cursors –Review types and options –Working with feature geometry Creating and using objects for parameter valuesCreating and using objects for parameter values How to create a script toolHow to create a script tool Making scripts more efficientMaking scripts more efficient –Use of layers and table views –Calling models Effective Error HandlingEffective Error Handling SummarySummary

Scripting Languages Many to choose from for geoprocessing tasksMany to choose from for geoprocessing tasks –Python –VBScript –JScript –Perl –Others Any language that supports COM IDispatch may be usedAny language that supports COM IDispatch may be used Python is the most commonly used by our users and is what is most commonly documentedPython is the most commonly used by our users and is what is most commonly documented

Scripting Languages Python Why is Python recommended by ESRIWhy is Python recommended by ESRI –It is free! –Established user community –Simple when it needs to be –Easy to maintain –Scalable –Cross platform (windows, UNIX, Linux) –Many IDE’s to choose from –Supports object-oriented and modular designs

Scripting Languages Python What version is supported?What version is supported? –ArcGIS installs version 2.1, but all versions from 2.1 up to are supported Why 2.1?Why 2.1? –Due to installation problems of PythonWin for later versions of Python, ESRI had to ship version 2.1. How can I upgrade my version of PythonHow can I upgrade my version of Python –Download install for –Download PythonWin for –Read Knowledge Base document #26872 from support.esri.com

ArcGIS 9.2 Bulletin A native Python module for ArcGIS will be includedA native Python module for ArcGIS will be included –Based on version –Eliminates need for PythonWin –Allows better error messaging –Cross platform support Can be used to run scripts on Unix and Linux that have a 9.2 ArcGIS Engine installationCan be used to run scripts on Unix and Linux that have a 9.2 ArcGIS Engine installation –9.0 and 9.1 scripts will just work. Older Dispatch methodology will still be supported.

Scripting Documentation and Samples Writing Geoprocessing Scripts with ArcGISWriting Geoprocessing Scripts with ArcGIS –PDF in Documentation Library –Explanation of methods and samples ArcGIS Online HelpArcGIS Online Help –Same content as PDF –Specific help and samples for all of the geoprocessor’s methods and properties Geoprocessing Programming GuideGeoprocessing Programming Guide –Logical layout of scripting objects with their methods and properties –Quick Reference

Section 1: Cursors

Geoprocessing workflows commonly require record by record access of field valuesGeoprocessing workflows commonly require record by record access of field values –Including the geometry field This is done using an object that points to a record in a table or feature class called a cursorThis is done using an object that points to a record in a table or feature class called a cursor ExamplesExamples –Using a field value in an logical expression if row.soiltype == “loam”: row.group = 2 row.group = 2 –Moving values from one table or feature class to another if row1.parcelid == row2.parcelid: row1.owner = row2.owner row1.owner = row2.owner –Accessing properties of a feature’s geometry print row.shape.FirstPoint Accessing Data with Cursors

There are three types of cursorsThere are three types of cursors –Search Cursor Read-only accessRead-only access –Update Cursor Read/Write/Delete access but no new recordsRead/Write/Delete access but no new records –Insert Cursor Read/Write access with capability of creating new recordsRead/Write access with capability of creating new records

Accessing Data with a Search Cursor A row object is returned from the search cursor objectA row object is returned from the search cursor object Fields are accessed as properties of the row objectFields are accessed as properties of the row object Use the row object’s GetValue and SetValue methods if your field name is a variableUse the row object’s GetValue and SetValue methods if your field name is a variable Destroy the row and cursor objects to remove read locks on the data sourceDestroy the row and cursor objects to remove read locks on the data source

Accessing Data with a Search Cursor from win32com.client import Dispatch gp = Dispatch("esriGeoprocessing.gpDispatch.1") rows = gp.SearchCursor("D:/St_Johns/data.mdb/roads") row = rows.Next() # Print concatenated values of road name and road type while row: print row.name + row.GetValue("type") print row.name + row.GetValue("type") row = rows.Next() row = rows.Next() # Delete the row and cursor objects so no locks remain del row del rows

Accessing Data with Cursors A where clause may be used to limit the records returned by the cursorA where clause may be used to limit the records returned by the cursor –Same as defining a definition query on a layer from win32com.client import Dispatch gp = Dispatch("esriGeoprocessing.gpDispatch.1") rows = gp.SearchCursor("D:/St_Johns/data.mdb/roads", “[neighborhood] = "Shea_Heights") “[neighborhood] = "Shea_Heights") row = rows.Next() # Print concatenated values of road name and road type while row: print row.name + row.GetValue("type") print row.name + row.GetValue("type") row = rows.Next() row = rows.Next() del row del rows

Reading Feature Geometry You must understand the hierarchy for geometry in order to use itYou must understand the hierarchy for geometry in order to use it –A feature class is made of features –A feature is made of parts –A part is made of points In Python termsIn Python terms –A single part feature looks like this [pnt, pnt, pnt] –A multipart polygon feature looks like this [[pnt, pnt, pnt],[pnt, pnt, pnt]] –A single part polygon feature with a hole (inner ring) looks like [ [pnt, pnt, pnt,,pnt, pnt, pnt]]

Reading Feature Geometry A null point is used as a separator between rings (holes) in a polygon partA null point is used as a separator between rings (holes) in a polygon part Use the PartCount property to get the number of parts for a featureUse the PartCount property to get the number of parts for a feature Use the GetPart method to retrieve the desired partUse the GetPart method to retrieve the desired part # Reading lines x = 0 while x < feat.PartCount: roadArray = feat.GetPart(x) roadArray = feat.GetPart(x) pnt = roadArray.Next() pnt = roadArray.Next() while pnt: while pnt: print pnt.x + "," + pnt.y print pnt.x + "," + pnt.y pnt = RoadArray.Next() pnt = RoadArray.Next() x = x + 1 x = x + 1

Reading Feature Geometry Features may be projected on-the-fly to another coordinate system using the Spatial Reference parameterFeatures may be projected on-the-fly to another coordinate system using the Spatial Reference parameter Create the Spatial Reference by describing another datasetCreate the Spatial Reference by describing another dataset # Describe features class with a GCS desc = gp.Describe("D:/data.mdb/geo_roads") # Create search cursor, using GCS spatial reference rows = GP.SearchCursor("D:/data.mdb/roads","", desc.SpatialReference) desc.SpatialReference) row = rows.Next()

Demo 1 – Reading Feature Geometry

Reading Feature Geometry - Tips Use field values instead of the properties of the geometry object if speed is really importantUse field values instead of the properties of the geometry object if speed is really important –Length –Area If you want the extent or centroid of a complex feature try creating simple features in a new feature classIf you want the extent or centroid of a complex feature try creating simple features in a new feature class –Features to Point –Feature Envelope to Polygon Using fields or simple geometry is faster that querying the geometry object of a complex featureUsing fields or simple geometry is faster that querying the geometry object of a complex feature

Writing Feature Geometry Insert cursors must be used to create new featuresInsert cursors must be used to create new features rows = GP.InsertCursor("D:/data.mdb/roads“) row = rows.NewRow() Use the Point and Array objects to create feature partsUse the Point and Array objects to create feature parts A part may be used to set a geometry fieldA part may be used to set a geometry field –A multipart feature is an array containing other arrays, where each array is a part

Writing Feature Geometry # Open an insert cursor for the feature class cur = GP.InsertCursor(fcname) # Create the array and point objects needed to create a feature lineArray = GP.CreateObject("Array") pnt = GP.CreateObject("Point") # Add two points to the array pnt.x = pnt.y = lineArray.Add(pnt) pnt.x = pnt.y = lineArray.Add(pnt) # Create a new row, or feature, in the feature class feat = cur.NewRow() # Set the geometry of the new feature to the array of points feat.shape = lineArray # Insert the feature cur.InsertRow(feat)

Writing Feature Geometry The geoprocessor checks the validity of the geometry before it is insertedThe geoprocessor checks the validity of the geometry before it is inserted –Problems such as invalid ring order or ring orientation are corrected automatically –Uses the same process found in the Check and Repair Geometry tools Writing features requires an exclusive schema lock on the feature classWriting features requires an exclusive schema lock on the feature class –Use the TestSchemaLock method to see if the data is being used by another process

ArcGIS 9.2 Bulletin Cursor performance will be FASTER!Cursor performance will be FASTER! –Much faster for Update operations New option to limit the returned fields on a rowNew option to limit the returned fields on a row New option to sort the rows on a field or set of fieldsNew option to sort the rows on a field or set of fields 9.0 and 9.1 scripts will just work9.0 and 9.1 scripts will just work Speed enhancements will be included in Service Pack 1 for ArcGIS 9.1Speed enhancements will be included in Service Pack 1 for ArcGIS 9.1

Section 2: Complex Parameters

Tools and Parameters Each geoprocessing tool has a collection of parameter valuesEach geoprocessing tool has a collection of parameter values Parameters define how the tool will workParameters define how the tool will work

Using Objects for Parameters Most tool parameters are easy to define using a string or a number, such as a data path or buffer distanceMost tool parameters are easy to define using a string or a number, such as a data path or buffer distance Some parameters are more complex and have complicated text string representations,Some parameters are more complex and have complicated text string representations, –spatial reference or field mapping Instead of using text strings, use objectsInstead of using text strings, use objects

Using Objects for Parameters Objects either created directly byObjects either created directly by –The geoprocessor’s CreateObject method –Another object, such as the descriptive object created by the Describe method –A tool, such as Create Spatial Reference

Using Objects for Parameters Creating a geodatabase feature datasetCreating a geodatabase feature dataset SR = Spatial Reference from win32com.client import Dispatch gp = Dispatch("esriGeoprocessing.GPDispatch.1") # Describe a dataset that has the desired SR desc = gp.Describe("D:/St_johns/city.mdb/city_boundary") # Create the FDS using the describe object's SR Object gp.CreateFeatureDataset("D:/St_johns/city.mdb“,"NewFDS", desc.SpatialReference) desc.SpatialReference)

Using Objects for Parameters Value TablesValue Tables –Used to define any tool parameter that accepts more than one value (e.g. Merge) –Multivalue parameters may be expressed as A string ( " value1;value2;value3 " )A string ( " value1;value2;value3 " ) As a Value TableAs a Value Table –Use LoadFromString method to load a multivalue strting that may be passed as a script argument –Use ExportFromString to create an output argument value Useful for script toolsUseful for script tools

Using Objects for Parameters Overlay tools, such as Union and Intersect use Value Tables with multiple columnsOverlay tools, such as Union and Intersect use Value Tables with multiple columns –Difficult to parse multivalue string when more than one column is used Value Tables with multiple columns are easy to navigateValue Tables with multiple columns are easy to navigate ("D:/data/new roads" 1;D:/data/soils 2)

Section 3: Script Tools

Creating Tools from Scripts Why?Why? –The script is generic and can be used with other data Script can use arguments from the userScript can use arguments from the user –You want to use a script in ModelBuilder Incorporate another system with a script wrapper or do branchingIncorporate another system with a script wrapper or do branching –You want to easily share your script Not everyone knows how to run a stand-alone scriptNot everyone knows how to run a stand-alone script Puts a familiar face on your workPuts a familiar face on your work

Creating Tools from Scripts Step 1: Create argument variablesStep 1: Create argument variables –Can use sys.argv[ ], but this is subject to windows limitations (1024 characters) –Should use GetParameterAsText, especially with multivalue parameters

Creating Tools from Scripts Step 2: MessagingStep 2: Messaging –Return informative messages during execution of the script –Return error messages when a problem arises

Creating Tools from Scripts Step 3: Add the script to a toolboxStep 3: Add the script to a toolbox –Give the tool a name, label and description –Set the tool source and use relative paths option if you plan on sharing the tool –Define the parameters that will correspond to your script

Creating Tools from Scripts Step 3: Defining ParametersStep 3: Defining Parameters –Parameters have several properties Name: What you see on the dialog and on the command line.Name: What you see on the dialog and on the command line. Type: Is it required, optional or derived?Type: Is it required, optional or derived? Direction: Is the data being used (input) or created (output)?Direction: Is the data being used (input) or created (output)? Multivalue: Do you want a list of values or just one?Multivalue: Do you want a list of values or just one? Default: Is there a default value?Default: Is there a default value? Environment: Does an environment provide a default value?Environment: Does an environment provide a default value? Domain: Do you want to provide a choice or limit input values?Domain: Do you want to provide a choice or limit input values? Dependency: Does this parameter depend on another?Dependency: Does this parameter depend on another?

Creating Tools from Scripts Step 3: Derived ParametersStep 3: Derived Parameters –All tools that will be used in ModelBuilder should have an output –If the script updates an input dataset, create a derived parameter and set its dependency to the input parameter Value

Creating Tools from Scripts Step 3: Parameter DependencyStep 3: Parameter Dependency –Some parameter types have built-in behavior when there is a parameter dependency Fields with an Input table or feature classFields with an Input table or feature class –Fields will be populated automatically in the dialog Derived parameter with an input parameterDerived parameter with an input parameter –The derived parameter value will automatically be set to the value of the input parameter it depends upon

Demo 2: Creating A Script Tool

Creating Tools from Scripts AML is a supported source for script toolsAML is a supported source for script tools –Need to change the default behavior of an AML to support this By default it must execute with arguments when openedBy default it must execute with arguments when opened Use the RegisterAMLAsExecutable registry file to make this changeUse the RegisterAMLAsExecutable registry file to make this change –File is in the ArcGIS\ArcToolbox\Scripts folder Any ArcInfo workstation module is supportedAny ArcInfo workstation module is supported –AML must start in the Arc module and then call other modules –Display windows and form menus are supported

Section 3: Tips for Efficiency

Efficiency Tips When working with selections, use layers and views instead of using the Select toolWhen working with selections, use layers and views instead of using the Select tool –Layers and views are in memory and avoid costs of writing to disk Use Make Feature Layer, Make Raster Layer or Make Table ViewUse Make Feature Layer, Make Raster Layer or Make Table View Use Select By Attribute to update the selection or provide a where clause when creating the layer or viewUse Select By Attribute to update the selection or provide a where clause when creating the layer or view

Efficiency Tips Call models from your scriptsCall models from your scripts –Instead of rewriting a model as a script, reference the model’s toolbox and call it as a tool –Simplifies code from win32com.client import Dispatch gp = Dispatch("esriGeoprocessing.gpDispatch.1") # Add custom toolbox gp.AddToolbox("D:/Data/St_Johns/MyTools.tbx")gp.MyRoadsModel("D:/Data/St_Johns/urban_roads",100, "C:/temp/results") "C:/temp/results")

Efficiency Tips Use Python Lists and Dictionaries for frequent access of attributesUse Python Lists and Dictionaries for frequent access of attributes –Instead of reading through a table multiple times with a cursor, read it once and load the values into memory –Use a list for an ordered collection of values –Use a dictionary if you have an unordered collection of values and want to access values using a key Great for working with coordinatesGreat for working with coordinates –Many examples in the source scripts of the Spatial Statistics tools

Section 4: Error Handling

Error Handling The Geoprocessor is just another COM Dispatch object to Python, or any other languageThe Geoprocessor is just another COM Dispatch object to Python, or any other language –Error messages are less than helpful Tool messages must be retrieved from the geoprocessorTool messages must be retrieved from the geoprocessor Scripts must add error handling routines to catch errors and return tool messagesScripts must add error handling routines to catch errors and return tool messages

Error Handling Non-tool methods on the Geoprocessor do not return error messagesNon-tool methods on the Geoprocessor do not return error messages –Cursors, Creating Lists, Describe, etc Use the Raise statement to trigger specific exceptionsUse the Raise statement to trigger specific exceptions –Especially useful when using tools and native methods together –Provides more meaningful context for messages

Error Handling try: # Get the input feature class and make sure it contains polygons # Get the input feature class and make sure it contains polygons input = sys.argv[1] input = sys.argv[1] dsc = gp.Describe(input) dsc = gp.Describe(input) if dsc.ShapeType != "polygon": raise "ShapeError" if dsc.ShapeType != "polygon": raise "ShapeError" # Get the new field name # Get the new field name fieldname = sys.argv[2] fieldname = sys.argv[2] # Make sure shape_length and shape_area fields exist # Make sure shape_length and shape_area fields exist if gp.ListFields(input,"Shape_area").Next() and \ if gp.ListFields(input,"Shape_area").Next() and \ gp.ListFields(input,"Shape_length").Next(): gp.ListFields(input,"Shape_length").Next(): # Add the new field and calculate the value # Add the new field and calculate the value gp.AddField(input, fieldname, "double") gp.AddField(input, fieldname, "double") gp.CalculateField(input,fieldname, gp.CalculateField(input,fieldname, "[Shape_area] / [Shape_length]") "[Shape_area] / [Shape_length]") else: else: raise "FieldError" raise "FieldError" except "ShapeError": print "Input does not contain polygons" print "Input does not contain polygons" except "FieldError": print "Input does not shape area and length fields" print "Input does not shape area and length fields" except: print gp.GetMessages(2) print gp.GetMessages(2)

Error Handling Use the sys module to access python syntax errorsUse the sys module to access python syntax errors –Useful when using exec or eval functions –Must import the sys module except: msgs = gp.GetMessages() msgs = gp.GetMessages() if msgs: if msgs: AddMsgAndPrint(msgs,2) AddMsgAndPrint(msgs,2) value = sys.exc_info()[1] value = sys.exc_info()[1] AddMsgAndPrint(value.msg + ": " + value.text,2) AddMsgAndPrint(value.msg + ": " + value.text,2)

Demo 3: Error Handling

For additional information about geoprocessing including service pack information, knowledge base articles, and recent information about issues go to:For additional information about geoprocessing including service pack information, knowledge base articles, and recent information about issues go to: Additional support

ArcGIS Geoprocessing: Introduction to ModelBuilder Wed 3:30 PM Room: 6E ArcGIS Geoprocessing: Advanced ModelBuilder Thu 8:30 AM Room: 6E ArcGIS Geoprocessing: An Introduction Wed 1:30 PM Room: 6E ArcGIS Geoprocessing: Developing Geoprocessing tools with ArcObjects Thu 10:30 AM Room: 6E ArcGIS Geoprocessing: Using Spatial Statistics in ArcGIS Thu 1:30 PM Room: 8 ArcGIS Extension: 3D Analyst Geoprocessing Tools Thu 3:30 PM Room: 5-B Additional technical sessions

Working with Layers and Table Views in Geoprocessing Wed 4:00 -4:30pm How to convert pGDB feature classes to.shp in a model Thu 11:30 – 12noon Island demo’s & Demo Theatre sessions How to use the Geoprocessing Environment Settings T1 - Tue 10:30 – 11:15am T1 - Wed 1:30 – 2:15pm Using Spatial Statistics for Crime Analysis T1 - Thu 9 -10am Geoprocessing Tool Documentation and Metadata T1 - Thu 11: noon Island Demo’s Demo theatre sessions

Thank you! Please fill out your survey form Questions?