Lecture 9: Geoprocessing with Python Dr. Taysir Hassan Abdel Hamid Associate Professor, Information Systems Dept., Faculty of Computers and Information Assiut University April 17, 2016 INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Outline More on Strings File I\O ArcPy essential Vocabulary ArcPy Modules Arcpy.mapping INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Strings Methods INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman File I\O Often read/write data from files Open / close files Read data from a file Writing data from a file INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman f.close() INF424: GIS. Dr. Taysir Hassan A. Soliman

GeoProcessing with Python 1. Every script using Arc functions must start with import arcpy import arcpy 2. Get user inputs with ‘GetParameterAsText’ 1 2 3 4 5 6 '''script.py ''' import arcpy   # get input feature class inputFeatureClass = arcpy.GetParameterAsText(0) This gets the first user entered parameter as text. INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

Essential ArcPy Vocabulary Arcpy Classes Arcpy Functions Arcpy Modules StandAlone Python Scripts Python Window Python Script tool Python Toolbox INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman What is Arcpy? ArcPy is a Python site package that provides a useful and productive way to perform geographic data analysis, data conversion, data management, and map automation with Python. ArcGIS applications and scripts written using ArcPy benefit from being able to access and work with the numerous Python modules developed by GIS professionals and programmers from many different disciplines.  INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman ArcPy provides access to geoprocessing tools as well as additional functions, classes, and modules that allow you to create simple or complex workflows.  Geoprocessing tools are functions available from arcpy—that is, they are accessed like any other Python function. However, to avoid confusion, a distinction is always made between tool and nontool functions (such as utility functions like ListFeatureClasses()). INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Tools are documented differently than functions. Every tool has its own tool reference page in the help system. Functions are documented in the ArcPy documentation. Tools return a Result object; functions do not. Tools produce messages, accessed through a variety of functions such as GetMessages(). Functions do not produce messages. INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Python and Arcpy Python is an open-source programming language that was conceived in the late 1980s by Guido van Rossum and introduced in 1991. Python is supported by a growing and varied user community and provides easy readability, clean syntax, dynamic typing, and an extensive collection of standard and third-party libraries. INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Arcpy ArcPy (often referred to as the ArcPy site package) provides Python access for all geoprocessing tools, including extensions, as well as a wide variety of useful functions and classes for working with and interrogating GIS data. Using Python and ArcPy, you can develop an infinite number of useful programs that operate on geographic data. INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman This next example shows how to execute the Buffer tool. When run in the Python window, the code will be transferred into the transcript section of the window along with the result. arcpy.Buffer_analysis("c:/data/Portland.gdb/streets", "c:/data/Portland.gdb/steets_buffer", "500 METERS") INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Python and ArcMap Python is used to automate geoprocessing tools such as the following which are in the ArcToolbox: Analysis, cartography, conversion, data management, editing, geocoding Python also allows more advanced processing such as looping through records in a database and reading and writing them It also allows manipulating layers in a map It allows creating and manipulating geometries (point, line, polygon) INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman arcpy arcpy was introduced to ArcGIS in its 10th version It is downloaded with your ArcGIS! You can find it under your ArcGIS folder, e.g. at: Program Files > ArcGIS >Desktop10.1 You just need an editor such as PythonWin, IDLE, ArcMap Python window, or even Calculator field to access arcpy If after typing import arcpy in front of the Python prompt you do not get an error, then you have the arcpy! >>> import arcpy # Got no error? Then you can slither! INF424: GIS. Dr. Taysir Hassan A. Soliman

Arcpy Programming Environments Python code for ArcGIS geoprocessing tasks can be written using many different development environments Text Editor Python Win IDLE Third Party Notepad, Notepad++, Wordpad Visual python, PyScripter INF424: GIS. Dr. Taysir Hassan A. Soliman

New Python Window in ArcGIS 10 INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman About Python Window New to ArcGIS Embedded Interactive Python Window in ArcGIS Great for: Testing small blocks of Python code Learning Python Building quick and easy workflows in Python Execution of tools INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Python Window INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Options at ESRI Python Scripts AcrObjects Model Builder INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Arcpy classes A class is analogous to an architectural blueprint. The blueprint provides the framework for how to create something. Classes can be used to create objects, often referred to as an instance. ArcPy classes, such as the SpatialReference and Extent classes, are often used as shortcuts to complete geoprocessing tool parameters INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Arcpy functions In ArcPy, all geoprocessing tools are provided as functions, but not all functions are geoprocessing tools. In addition to tools, ArcPy provides a number of functions to better support geoprocessing Python workflows. INF424: GIS. Dr. Taysir Hassan A. Soliman

Stand-alone Python script A stand-alone Python script is a .py file that can be executed from the operating system prompt, a Python Integrated Development Environment (IDE), or by double-clicking the .py file in Windows Explorer. INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Python script tool A Python script tool is a Python script that has been added to a geoprocessing toolbox. Once added as a script tool, the script tool becomes like any other geoprocessing tool—it can be opened and executed from the tool dialog box, used in the Python window and ModelBuilder, and called from other scripts and script tools. INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Example import arcpy roads = "c:/base/data.gdb/roads" output = "c:/base/data.gdb/roads_Buffer" # Run Buffer using the variables set above and pass the remaining # parameters in as strings arcpy.Buffer_analysis(roads, output, "distance", "FULL", "ROUND", "NONE") INF424: GIS. Dr. Taysir Hassan A. Soliman

ArcGIS Spatial Analyst Extension ArcGIS Network Analyst Extension ArcPy Modules Arcpy Modules Data Access arcpy.da Mapping Arcpy.mp Geostastistical Analysis Arcpy.ga ArcGIS Spatial Analyst Extension Arcpy.sa Time module Arcpy.time ArcGIS Network Analyst Extension Arcpy.na INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman arcpy Opens Modules arcpy can open many modules Module: is a self contained collection of functions and classes that does something These modules include: Data access module (arcpy.da) Mapping module (arcpy.mapping) Geostatistical Analyst module (arcpy.ga) ArcGIS Spatial Analyst extension module (arcpy.sa) ArcGIS Network Analyst extension module (arcpy.na) INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Acrpy.mapping INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman … import arcpy # imports ArcGIS geoprocessing functionality import arcpy.mapping # imports only the mapping module import os # imports Python’s core operating system import sys # variables/functions used or maintained by the interpreter # import env from arcpy and set the workspace environment from arcpy import env # ability to control ArcGIS environment env.workspace = “C:\data” from arcpy.management import * # Content imported into namespace. Can use content without prefix INF424: GIS. Dr. Taysir Hassan A. Soliman

Properties and Methods Manipulating ArcObjects requires knowing their properties Properties/Methods of two ArcObjects are shown below: Map Feature Class Properties layer count Name Spatial reference Map scale Extent Methods Add layer Clear Selection Select feature Properties Shape type Spatial reference Extent Methods Create feature Remove feature INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Statements A statement is a task that does not return a value, e.g., print, import, and if statements: import arcpy print “bye” # import functionality from ArcGIS Spatial Analyst import arcpy.sa from arcpy.sa import * arcpy.CheckOutExtension (“Spatial”) # Check the Spatial Analyst License import arcy.mapping as map If statement checks if a condition is true or false For loop is another statement, loops through a list for fc in fclist: print fc INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Environments We set the environment for tools to use them This includes setting the current workspace, output spatial reference, extent, raster analysis setting (cell size, mask) arcpy.env.workspace --- sets out the workspace arcpy.env.outputCoordinateSystem --- output coordinates arcpy.env.extent arcpy.env.cellSize arcpy.evn.mask INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman arcpy.env.extent # Set the extent environment using a keyword. arcpy.env.extent = "MAXOF" INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Example import arcpy # Check the current raster cell size and make sure it is a certain size # for standard output # arcpy.env.workspace = "c:/avalon/data" if arcpy.env.cellSize < 10: arcpy.env.cellSize = 10 elif arcpy.env.cellSize > 20: arcpy.env.cellSize = 20 arcpy.HillShade_3d("island_dem", "island_shade", 300) INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman arcpy.evn.mask Tools that honor the Mask environment will only consider those cells that fall within the analysis mask in the operation. Illustration INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Example # add buffer around the road feature class with given distances import arcpy arcpy.env.workspace = ”C:\data\City.gdb” #sets the workspace fc = “Roads” #variable feature class distanceList = [“100 meters”, “200 meters”, 400 meters”] # distances # loops through each distance in the distanceList # takes the first distance and puts it in variable dist, and repeats it 3 times for dist in distanceList: outName = fc+” _”+ dist arcpy.Buffer_analysis (fc, outName, dist) # outputs the feature class, its output name_distance # breaks out of the loop print “Buffering completed!” INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Another example Clipping a road feature in ArcMap Click on the Python Window icon in ArcMap Clear the window from old code, if necessary Type the following code: # clip Roads feature class from the Mineral_Spring city polygon, # output the clipped feature class into RoadsClip arcpy.clip_analysis (“Roads”, “Mineral_Springs”, “RoadsClip”) # Run it # You can see in ArcMap the new added RoadsClip feature class # which was created and automatically added to the table of content INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Tool messages When we execute a tool, there might be three types of messages: Informative messages Warning messages Error messages try: # start try block arcpy.Buffer (“C:/ws/roads.shp”, “C:/outws/roads10.shp”, 10) # print the tool messages except arcpy.ExecuteError: print arcpy.GetMessages (2) # any other error Except Exception as e: print e.message INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Functions Functions perform useful tasks, such as: Accessing geoprocessing tool messages (GetMessages) Listing data for batch processing, e.g.: ListFeatureClasses, ListFields, plus nine other list functions Retrieving a dataset’s properties (Describe) import arcpy # Set the workspace for ListFeatureClasses function arcpy.env.workspace = “c:/test” # For each feature class, create a scratch name and clip for fc in arcpy.ListFeatureClasses (): outName = arcpy.CreateScratchName (“clipped_” + fc, “”, “featureclass”, arcpy.env.workspace) arcpy.Clip_analysis(fc, “boundary”, outName) INF424: GIS. Dr. Taysir Hassan A. Soliman

Dealing with Functions/Methods Assigning a value to a property: # object.property = value for example: env.workspace = “C:/Temp” Return the value of a property: # object.property for example: print “The workspace is “ + env.workspace Use a method: # object.method (arg1, arg2, …) e.g., put a buffer for a road: arcpy.Buffer_analysis (“c:/input/roads.tif’, “c:/output.gdb/buffer_output, 100) INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Geometry objects We can create, delete, move, and reshape features # Create a geometry object and put it in the variable g g = arcpy.Geometry () # Run the Copy Features tool. set the output to the geometry object # Return a list of geometry objects (lines, streets) geometryList = arcpy/CopyFeatures_management (“c:/data/streets.shp”, g) # Loop through each geometry, totaling the lengths of the streets # for geometry in geometryList: length += geometry.length # Note: x +=1 means x=x+1 print “Total length: %f” % length INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Mapping module arcpy.mapping Is used to open and manipulate existing map documents (.mxd) and layer files (.lyr) It is used to query and alter the contents Find a layer with data source X and replace with Y Updata a layer’s symbology across many MSDs Generate a report listing document information Data sources, broken layers, spatial reference, etc. Can print, export, or save the modified document Allows adding, removing, and rotating data frames, and adding and removing layers We can manipulate properties of map documents and layers See next slide for an example INF424: GIS. Dr. Taysir Hassan A. Soliman

Manipulate map documents # Set data frame properties and export to a TIFF import arcpy # Find the Project.mxd map document mxd = arcpy.mapping.MapDocument (”c:\Project\Project.mxd”) # make a list of all the data frames in it; iterate through the frames for df in arcpy.mapping.ListDataFrames (mxd): df.rotation = 0 # rotate the map df.scale = 24000 # set the scale outFile = ”c:\Project\Output\\” + df.name + “.tif” arcpy.mapping.ExportToTIFF (mxd, outFile, df) del mxd INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Example 2 # Modify map document properties, save changes to a layer file, and save changes to the map document import arcpy mxd = arcpy.mapping.MapDocument (“input.mxd”) df = arcpy.mapping.ListDataFrames (mxd) df.scale = 24000 df.rotation = 2.7 for lyr in arcpy.mapping.ListLayers(mxd): if lyr.name == “Landuse”: lyr.visible = True lyr.showLabels = True lyr.saveACopy(“output.lyr”) mxd.save() del mxd INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman Manipulating Layers We can change properties of a layer, such as its: name, source data, visibility (make it on or off), transparency, label, definition query, display order, etc. import arcpy # open a file on our hard drive lyrFile = arcpy.mapping.Layer (”C:\Project\Date\Streets.lyr”) # we are going to change its name, from streets to highways for lyr in arcpy.mapping.ListLayers(lyrFile): if lyr.name.lower() == “highways”: # turn its label on lyr.showLabels = True lyr.saveACopy (r”C:\Project\Data\StreetsWithLabels.lyr”) # now the changed layer is saved as different layer del lyrFile INF424: GIS. Dr. Taysir Hassan A. Soliman

Adding a Python script as a tool We can add a script as a tool to a toolbox These are called script tools They become a new tool with all the properties of a tool, e.g., It will return messages, access to all environment settings, and automatically add the output to our map (to the table of contents in ArcMap) Can easily be shared, e.g., email it to other users they can double click it to run it) Tools automatically create dialog boxes (created for us by ArcGIS) We can even add the tool into the toolbar and menus INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman References http://pro.arcgis.com/en/pro-app/tool-reference/analysis/clip.htm http://pro.arcgis.com/en/pro-app/arcpy/geoprocessing_and_python/using-functions-in-python.htm https://www.e-education.psu.edu/geog485/node/162 http://resources.arcgis.com/en/help/main/10.2/index.html#//002z00000008000000 INF424: GIS. Dr. Taysir Hassan A. Soliman

INF424: GIS. Dr. Taysir Hassan A. Soliman End of Lecture INF424: GIS. Dr. Taysir Hassan A. Soliman