Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 17-Feb-2011

Presentation outline What is MGET? The problems with building Python-based geoprocessing tools on top of ArcGIS MGET’s new GIS-agnostic architecture How to use MGET outside of ArcGIS

What is MGET? 200+ geoprocessing tools for marine ecology Free, open source software Written in Python, R, MATLAB, C#, and C++ Minimum requirements: Windows XP, Python 2.4 ArcGIS 9.1 or later currently needed for many tools ArcGIS and Windows are only non-free requirements ~1200 installations in 62 countries in last 18 months Also useful for terrestrial problems!

MGET plugs into ArcGIS The MGET toolbox appears in the ArcToolbox window

Geoprocessing with MGET Drill into the toolbox to find the tools Double-click tools to execute directly, or drag to geoprocessing models to create a workflow

Mapping species biodiversity

Identifying SST fronts ~120 km AVHRR Daytime SST 03-Jan °C 25.8 °C Mexico Front Cayula and Cornillion (1992) edge detection algorithm Frequency Temperature Optimal break 27.0 °C Strong cohesion  front present Step 1: Histogram analysis Step 2: Spatial cohesion test Weak cohesion  no front Bimodal Example output Mexico ArcGIS model

Analyzing larval connectivity Coral reef ID and % cover maps Ocean currents data Tool downloads data for the region and dates you specify Larval density time series rasters Edge list feature class representing dispersal network Original research by Eric A. Treml

Many MGET tools use ArcGIS Read/write vector datasets using geoprocessor’s cursors Create and geoprocess vector and raster datasets using ArcGIS’s geoprocessing tools (e.g. Spatial Analyst Tools)

Problems with ArcGIS Until recently, no Python interface for reading/writing cells of rasters ArcGIS 10 finally provided one: Prior to ArcGIS 10, you had to use other libraries, such as the Geospatial Data Abstraction Library (GDAL) Or write ASCII rasters and then run ASCII To Raster MGET 0.7 did this It is very, very slow!! import arcpy, numpy myArray = arcpy.RasterToNumPyArray('C:/data/inRaster') newRaster = arcpy.NumPyArrayToRaster(myArray + 1) newRaster.save('C:/data/outRaster')

Problems with ArcGIS Limited support for gridded scientific data formats: HDF: Only supports 2D grids Must extract to raster on disk, then read Georeferencing not supported NetCDF: File must have complete CF-compliant metadata Must access 3D and 4D data as 2D slices, calling Make NetCDF Raster Layer each time OPeNDAP: Not supported

Problems with ArcGIS ArcGIS is not designed to support 3D or 4D grids, not even with ArcObjects ArcObjects raster APIs only support 2 dimensions

Problems with ArcGIS Slow turnaround from ESRI on bugs; no access to devs For example: In 9.2, cursors could not set fields to NULL: ESRI did not fix this until 4 years later, in SP1 >>> import arcgisscripting >>> gp = arcgisscripting.create() >>> cur = gp.UpdateCursor('C:\\TestGDB.mdb\\TestTable') >>> row = cur.Next() >>> row.GetValue('MyField') >>> row.SetValue('MyField', None) Traceback (most recent call last): File " ", line 1, in ? ValueError: invalid input value

Problems with ArcGIS Conservation practitioners in developing countries— an important MGET user segment—cannot afford Arc We get regular requests to support free GIS platforms GRASS GIS

MGET needs a new architecture GRASS …we need to replace this. To support this fully…

What about using GDAL instead? GDAL is an open source library for reading and writing many formats of raster data The related OGR library reads/writes vector data

Problems with GDAL and OGR Limitations for several important formats, including: Cannot create ArcInfo Binary Grids Cannot create Personal GDB and File GDB feature classes Cannot read ArcGIS in-memory layers Like ArcGIS, GDAL data model is 2D Better, though: 3d and 4d supported as sets of 2D bands Very limited raster geoprocessing Project, contour, vector to raster, raster to vector No vector geoprocessing (can only convert formats) With code, you can implement basic operations on OGR layers (e.g. clip, buffer, intersect, sym. difference, etc.)

MGET’s new architecture None implemented yet

Example: Downloading MODIS L3 Chl-a images from NASA OceanColor Group NASA publishes global chlorophyll-a concentration images from several satellites Images in 2D HDF files downloadable over HTTP MGET tool automates download and conversion NASA HTTP Server

From ArcGIS Required parameters

From Python outside ArcGIS This example still requires that ArcGIS Desktop or ArcGIS Engine be installed >>> from GeoEco.Logging import Logger # Initialize MGET logging (optional) >>> Logger.Initialize() >>> >>> from GeoEco.DataProducts.NASA.OceanColor import OceanColorLevel3SMITimeSeries >>> OceanColorLevel3SMITimeSeries.CreateArcGISRasters( 'Aqua', 'Annual', '4km', 'CHL_chlor_a', 'C:\\Temp\\OceanColor') :34:48,049 INFO Querying time slices of aqua annual 4km chl_chlor_a from NASA OceanColor :34:48,099 INFO Query complete: 0:00:00 elapsed, 9 datasets found, 0:00: per dataset :34:48,109 INFO Importing 9 datasets into ArcGIS Folder C:\Temp\OceanColor with mode "replace" :36:17,338 INFO Import in progress: 0:01:29 elapsed, 2 datasets imported, 0:00: per dataset, 7 remaining, estimated completion time: 4:41:29 PM :41:16,598 INFO Import complete: 0:06:28 elapsed, 9 datasets imported, 0:00: per dataset. >>>

From Python without ArcGIS The catch: the CreateGDALRasters function has not been implemented yet But it is only 15 minutes of work and 10 lines of code We will implement this and similar functions en masse across MGET sometime this spring >>> OceanColorLevel3SMITimeSeries. CreateArcGISRasters (...) >>> OceanColorLevel3SMITimeSeries. CreateGDALRasters (...) Replace this with this

From other languages via COM R: MATLAB: Also VBScript, JScript, Perl, etc. C++ and C# can use early binding to GeoEco typelib This is all enabled by the pywin32 Python package > library(rcom) > oc <- comCreateObject("GeoEco.OceanColorLevel3SMITimeSeries") > comInvoke(oc, "CreateArcGISRasters", "Aqua", "Annual", "4km", "CHL_chlor_a", "C:\\Temp\\OceanColor“) NULL >> oc = actxserver('GeoEco.OceanColorLevel3SMITimeSeries'); >> oc.CreateArcGISRasters('Aqua', 'Annual', '4km', 'CHL_chlor_a', 'C:\\Temp\\OceanColor')

Summary Building geoproc tools with Python is quick and easy If you build directly on top of the ArcGIS geoprocessor, watch out for limitations and bugs Particularly if you work with large scientific datasets Open source technologies provide a viable alternative Very good support for many GIS formats, but not all Direct support from developers; quick bug fixes; free No comprehensive replacement for Arc’s many toolboxes No extant framework for building geoprocessing tools for multiple GIS platforms We are building one as part of MGET

Acknowledgements A special thanks to the many developers of the open source software that MGET is built upon, including: Guido van Rossum and his many collaborators; Mark Hammond; Travis Oliphant and his collaborators; Walter Moreira and Gregory Warnes; Peter Hollemans; David Ullman, Jean-Francois Cayula, and Peter Cornillon; Stephanie Henson; Tobias Sing, Oliver Sander, Niko Beerenwinkel, and Thomas Lengauer; Frank Warmerdam and his collaborators, Howard Butler; Timothy H. Keitt, Roger Bivand, Edzer Pebesma, and Barry Rowlingson; Gerald Evenden; Jeff Whitaker; Roberto De Almeida and his collaborators; Joe Gregorio; David Goodger and his collaborators; Daniel Veillard and his collaborators; Stefan Behnel, Martijn Faassen, and their collaborators; Paul McGuire and his collaborators; Phillip Eby, Bob Ippolito, and their collaborators; Jean-loup Gailly and Mark Adler; the developers of netCDF; the developers of HDF Thanks to our funders:

For more information Download MGET: me: Thanks for attending!