Extending ArcGIS using programming

Slides:



Advertisements
Similar presentations
WITHOUT LANGUAGE [ DEVELOPING GEO-PROCESSING MODELS USING ARCGIS MODELBUILDER 10 ] PROGRAMMING R. RYAN STEVENS / GIS RESEARCH ANALYST / THE POLIS CENTER.
Advertisements

Extending ArcGIS using programming David Tarboton GIS in Water Resources 22 October 2013.
Environmental GIS Nicholas A. Procopio, Ph.D, GISP Some slides from Lyna Wiggins (Rutgers University)
Why python? Automate processes Batch programming Faster Open source Easy recognition of errors Good for data management What is python? Scripting programming.
Standard watershed and stream delineation recipe - Vector stream (ex. NHD data) fusion into DEM raster (burning in) - Sink removal - Flow direction - Flow.
Terrain Analysis Using Digital Elevation Models (TauDEM)
Key Spatial Analysis Concepts from Exercise 3 Contours and Hillshade to visualize topography.
Concept Course on Spatial Dr. A.K.M. Saiful Islam Application of GIS in Watershed Analysis Dr. A.K.M. Saiful Islam Institute of Water and Flood.
ModelBuilder In ArcGIS 9.x By Tim Weigel GEOG 407/607 April 3 rd, 2006.
Geography 465 Overview Geoprocessing in ArcGIS. MODELING Geoprocessing as modeling.
ModelBuilder at ArcGIS 9.2 Lyna Wiggins Rutgers University May 2008.
Remote Sensing and GIS in Water Dr. A.K.M. Saiful Islam Hands on training on surface hydrologic analysis using GIS Dr. A.K.M. Saiful Islam.
Kurt Menke, GISP GRASS GIS Geographic Resources Analysis Support System.
Rebecca Boger Earth and Environmental Sciences Brooklyn College.
Welcome to Mapping Tom Sellsted – City of Yakima, Washington Vladimir Strinski – Hitech Systems.
Working with cursors in Python GISDE Python Workshop Qiao Li.
Digital Elevation Model Based Watershed and Stream Network Delineation
Introduction to Spatial Analysis and Spatial Modeling
Python: An Introduction
Distance. Euclidean Distance Minimum distance from a source (Value NoData) Input grid must have at least one source cell with the rest of the grid.
Digital Elevation Model Based Watershed and Stream Network Delineation Understanding How to use Reading
ArcGIS Network Analyst: Automating Workflows with Geoprocessing
Creating Watersheds and Stream Networks
Expanding the Functionality of ArcGIS Through Tool Building
Esri UC 2014 | Technical Workshop | Creating Watersheds, Stream Networks and Hydrologically Conditioned DEMS Steve Kopp Dean Djokic.
Digital Elevation Model Based Watershed and Stream Network Delineation
Introduction of Geoprocessing Lecture 9. Geoprocessing  Geoprocessing is any GIS operation used to manipulate data. A typical geoprocessing operation.
Terrain Based Flood Inundation mapping
David Tarboton Digital Elevation Models, Flood Inundation Mapping and River Hydraulic Properties David Tarboton
USU, RENCI, BYU, UNC, UVA, CUAHSI, Tufts, Texas, Purdue, Caktus
Extending ArcGIS using programming
Python: Building Geoprocessing Tools David Wynne, Ghislain Prince.
Extending ArcGIS via programming
Terrain Analysis Using Digital Elevation Models (TauDEM)
Hydrologic Terrain Processing Using Parallel Computing
Hydrologic Terrain Processing Using Parallel Computing
Automating parflow preprocessing steps for DEMs
The Mach System Sri Ramkrishna.
Flow field representations for a grid DEM
Extending ArcGIS using programming
Extending ArcGIS using programming
PYTHON: AN INTRODUCTION
Environmental GIS Nicholas A. Procopio, Ph.D, GISP
ArcGIS Topology Shapefiles, Coverages, Geodatabases
Terrain Analysis for Water Quality Modeling
Lecture 22: Using ArcToolbox Tools in Python
Types of geodatabases Introduction to GIS - Student notes
Watershed Analysis.
Automation of Input data preparation of TOPNET model using Python
Digital Elevation Model Based Watershed and Stream Network Delineation
Bug in Flow Direction in Ex3 data
Digital Elevation Model Based Watershed and Stream Network Delineation
Key Spatial Analysis Concepts from Exercise 3
Workflows with ENVI and Esri Agriculture workflows for ICARDA
Review- vector analyses
GIS FOR HYDROLOGIC DATA DEVELOPMENT FOR DESIGN OF HIGHWAY DRAINAGE FACILITIES by Francisco Olivera and David Maidment Center for Research in Water Resources.
Terrain Analysis Using Digital Elevation Models (TauDEM)
Geography & Technology
GENERAL VIEW OF KRATOS MULTIPHYSICS
Automating and Validating Edits
Channels, Watersheds, Flow Related Terrain Information
Terrain Analysis Using Digital Elevation Models
Web AppBuilder for ArcGIS
Leveraging ArcGIS Online Elevation and Hydrology Services
Network Analysis using Python
PYTHON: BUILDING GEOPROCESSING TOOLS
Network Analyst – Automating Workflows with Geoprocessing
Extending ArcGIS using programming
ArcGIS Pro: An Introduction Overview
Esri Production Mapping: An Introduction
Presentation transcript:

Extending ArcGIS using programming David Tarboton

Why Programming Automation of repetitive tasks (workflows) Implementation of functionality not available (programming new behavior)

Three Views of GIS Geodatabase view: Structured data sets that represent geographic information in terms of a generic GIS data model. Geovisualization view: A GIS is a set of intelligent maps and other views that shows features and feature relationships on the earth's surface. "Windows into the database" to support queries, analysis, and editing of the information. Geoprocessing view: Information transformation tools that derive new geographic data sets from existing data sets. adapted from www.esri.com

ArcGIS Pro Geoprocessing Help http://pro.arcgis.com/en/pro-app/help/analysis/geoprocessing/basics/what-is-geoprocessing-.htm

ArcGIS programming options Model builder Python scripting environment ArcObjects library (for system language like C++, .Net) AML Open standard data formats that anyone can use in programs (e.g. shapefiles, geoTIFF, netCDF)

Python Environments Python Window built into ArcGIS Pro http://pro.arcgis.com/en/pro-app/arcpy/get-started/python-window.htm Idle: Simple editor that is part of Python PyCharm: Powerful professional development environment, https://www.jetbrains.com/pycharm/

Each Geoprocessing tool has it’s own Python Command

Demo Python code from Geoprocessing History Use of Python Window Python code from online help Sequencing code into a script Editing and running with Idle

Example – Watershed delineation using Python (Steps from Exercise 4) Set Inputs DEM Outlet Threshold Set workspace Fill Flow Direction Flow Acumulation Snap Outlet Watershed Stream Raster Stream Link Catchment Vector Conversion 1 2 3 4 5 6 7 8 9 10 11

TauDEM http://hydrology.usu.edu/taudem/ 11/8/2017 Stream and watershed delineation Multiple flow direction flow field Calculation of flow based derivative surfaces MPI Parallel Implementation for speed up and large problems Open source platform independent C++ command line executables for each function Deployed as an ArcGIS Toolbox with python scripts that drive command line executables http://hydrology.usu.edu/taudem/

TauDEM Parallel Approach MPI, distributed memory paradigm Row oriented slices Each process includes one buffer row on either side Each process does not change buffer row Improved runtime efficiency Capability to run larger problems

Some Algorithm Details Pit Removal: Planchon Fill Algorithm 1st Pass 2nd Pass Initialization Planchon, O., and F. Darboux (2001), A fast, simple and versatile algorithm to fill the depressions of digital elevation models, Catena(46), 159-176.

Parallel Scheme Communicate Z denotes the original elevation. Initialize( Z,F) Do for all grid cells i if Z(i) > n F(i) ← Z(i) Else F(i) ← n i on stack for next pass endfor Send( topRow, rank-1 ) Send( bottomRow, rank+1 ) Recv( rowBelow, rank+1 ) Recv( rowAbove, rank-1 ) Until F is not modified Z denotes the original elevation. F denotes the pit filled elevation. n denotes lowest neighboring elevation i denotes the cell being evaluated Iterate only over stack of changeable cells

Pseudocode for Recursive Flow Accumulation Global P, w, A,  FlowAccumulation(i) for all k neighbors of i if Pki>0 FlowAccumulation(k) next k return Pki

Generalization to Flow Algebra Replace Pki Pki Pki i by general function

Parallelization of Flow Algebra 1. Dependency grid 2. Flow algebra function Executed by every process with grid flow field P, grid dependencies D initialized to 0 and an empty queue Q. FindDependencies(P,Q,D) for all i for all k neighbors of i if Pki>0 D(i)=D(i)+1 if D(i)=0 add i to Q next Executed by every process with D and Q initialized from FindDependencies. FlowAlgebra(P,Q,D,,) while Q isn’t empty get i from Q i = FA(i, Pki, k, k) for each downslope neighbor n of i if Pin>0 D(n)=D(n)-1 if D(n)=0 add n to Q next n end while swap process buffers and repeat

Parallelization of Contributing Area/Flow Algebra 1. Dependency grid A=1 A=1.5 A=3 D=2 D=1 B=-2 Queue’s empty so exchange border info. B=-1 A=1 A=1.5 A=3 A=5.5 A=2.5 A=6 A=3.5 and so on until completion A=1 A=1.5 A=3 D=0 D=1 resulting in new D=0 cells on queue A=1 D=1 D=0 A=1.5 D=2 B=-1 Decrease cross partition dependency Executed by every process with grid flow field P, grid dependencies D initialized to 0 and an empty queue Q. FindDependencies(P,Q,D) for all i for all k neighbors of i if Pki>0 D(i)=D(i)+1 if D(i)=0 add i to Q next A=1 D=0 D=1 D=2 A=1 D=0 A=3 A=1.5 D=2 D=1 B=-2 A=1 D=0 D=1 D=2 A=1 D=0 D=2 D=1 D=0 D=1 D=3 D=2 2. Flow algebra function Executed by every process with D and Q initialized from FindDependencies. FlowAlgebra(P,Q,D,,) while Q isn’t empty get i from Q i = FA(i, Pki, k, k) for each downslope neighbor n of i if Pin>0 D(n)=D(n)-1 if D(n)=0 add n to Q next n end while swap process buffers and repeat

Programming C++ Command Line Executables that use MPI Use GDAL/OGR library to read and write datasets in open standard formats for exchange with other programs ArcGIS Python Script Tools Python validation code to provide file name defaults Shared as ArcGIS Toolbox Any computer Requires ArcGIS

Q based block of code to evaluate any “flow algebra expression” while(!que.empty()) { //Takes next node with no contributing neighbors temp = que.front(); que.pop(); i = temp.x; j = temp.y; // FLOW ALGEBRA EXPRESSION EVALUATION if(flowData->isInPartition(i,j)){ float areares=0.; // initialize the result for(k=1; k<=8; k++) { // For each neighbor in = i+d1[k]; jn = j+d2[k]; flowData->getData(in,jn, angle); p = prop(angle, (k+4)%8); if(p>0.){ if(areadinf->isNodata(in,jn))con=true; else{ areares=areares+p*areadinf->getData(in,jn,tempFloat); } // Local inputs areares=areares+dx; if(con && contcheck==1) areadinf->setToNodata(i,j); else areadinf->setData(i,j,areares); // END FLOW ALGEBRA EXPRESSION EVALUATION C++

Maintaining to do Q and partition sharing while(!finished) { //Loop within partition while(!que.empty()) { .... // FLOW ALGEBRA EXPRESSION EVALUATION } // Decrement neighbor dependence of downslope cell flowData->getData(i, j, angle); for(k=1; k<=8; k++) { p = prop(angle, k); if(p>0.0) { in = i+d1[k]; jn = j+d2[k]; //Decrement the number of contributing neighbors in neighbor neighbor->addToData(in,jn,(short)-1); //Check if neighbor needs to be added to que if(flowData->isInPartition(in,jn) && neighbor->getData(in, jn, tempShort) == 0 ){ temp.x=in; temp.y=jn; que.push(temp); //Pass information across partitions areadinf->share(); neighbor->addBorders(); C++

Python Script to Call Command Line mpiexec –n 8 pitremove –z Logan.tif –fel Loganfel.tif

PitRemove Python

Validation code to add default file names Python

Summary GIS and Water Resources analysis capabilities are readily extensible with programming to do something new to repeat something needed frequently Model builder provides visual programming and helps learn ArcGIS python commands Python – cross platform, powerful and easy to use is a good programming language to start with (when your time is valuable) Compiled language programming for developers (C++) to achieve optimal efficiency (when the computers time is valuable)