1. Open Source GIS: Explaining the Software Stack Presented by Aaron Racicot – GIS Programmer aaronr@ecotrust.org May 17th, 2006 A Citizen of Salmon Nation

2. Outline  Introduction – Personal and Ecotrust  Software Stack – Desktop, Web, DST  Raster Tools – GDAL, GRASS/QGIS  Vector Tools – OGR, PostGIS, GRASS/QGIS  Where is it all going?  Backups – Raster/Vector DST’s, Installation, and screen shots

3. Who am I… B.S. Computer Science Split Personality M.S. Environmental Science Open Source User/Developer GIS Programmer

4. Ecotrust - Salmon Nation

5. Conservation Economy

6. Why Open Source? “Free software” is a matter of liberty, not price. Free Software!

7. Freedom

8. Benefits/Limitations of Open Source Benefits  Software cost = $0  Source code is available and modifiable  User and development communities flourish  Development cycles are VERY fast Limitations  Total cost is NOT = $0  Many applications are not as polished as the proprietary counterparts  Compatibility with proprietary software can be an issue

9. OS GIS – Desktop Applications GRASS GIS

10. Software Stacks… Desktop Generic StackFOSS4G StackGrouping Application QGis, Grass, OSSIM, JUMP, UDig User Interface Application Dev. Environment Eclipse, QT, OpenGL High Level Utilities GeoTools, PostGIS Data Serving High Level Scripting Languages PHP, PERL, Python Low Level Utilities Shplib, GEOS, OGR/GDAL, PostGIS, R-Statistics, GMT Data Processing Low Level Languages C, C++, Java, Fortran System Software Operating System Linux, Darwin, Cygwin Hardware Drivers 32-64 bit processor drivers Desktop Software Stack

11. OS GIS - Server Tools GRASS GIS GDAL/OGR R-Statistics

12. Software Stacks... Server Web-based Software Stack Generic StackFOSS4G StackGrouping Client Side Browser Firefox, Safari, Netscape User Interface Client Side Scripting Java Script, Java Applets ~~~~~~~~~~~WEB~~~~~~~~~~~ Server Side Application Chameleon, Cartoweb3, Custom Data Serving High Level Utilities Mapserver Low Level Utilities Shplib, GEOS, OGR/GDAL, PostGIS, R-Statistics, GMT Data Processing High Level Scripting Languages PHP, PERL, Python Low Level Languages C, C++, Java, Fortran System Software Operating System/Drivers Linux, Darwin, Cygwin Hardware Drivers 32-64 bit processor drivers

13. Software Stack... DSTs (Ecotrust) DST Software Stack Generic StackFOSS4G StackGrouping Client Side Browser Firefox, Safari, Netscape User Interface Client Side Scripting Chameleon/Custom ~~~~~~~~~~~WEB~~~~~~~~~~~ Server Side Application Chameleon/Custom Data Serving High Level Utilities Mapserver, GRASS, PostGIS Low Level Utilities Shplib, GEOS, R-Statistics, GMT, OGR/GDAL Data Processing High Level Scripting Languages PHP, PERL Low Level Languages C, C++, Java, Fortran System Software Operating System/Drivers Linux (Fedora) Hardware Drivers 32 bit processor drivers

14. Raster Processing GRASS/GDAL - Data Crunching  Very powerful tool set (similar to ArcINFO/ArcGRID)  Poor GUI, but getting better  Very fast  Cross platform QGIS/JUMP/UDIG – Data Viewing  Both Raster and Vector support  Nice GUI  Cross platform  QGIS can integrate with GRASS

15. GRASS-GDAL

16. Vector Processing GRASS/OGR - Data Crunching  Very powerful tool set (similar to ArcINFO/ArcGRID)  Poor GUI, but getting better  Very fast  Cross platform QGIS/JUMP/UDIG – Data Viewing  Both Raster and Vector support  Nice GUI  Cross platform  QGIS can integrate with GRASS

17. GRASS-OGR

18. Where to go for more info OSGIS  Maptools - http://www.maptools.org  FreeGIS - http://freegis.org/  Open Source GIS - http://opensourcegis.org/ Standards  OGC - http://www.opengeospatial.org/ Desktop  GRASS - http://grass.itc.it/  QGIS - http://qgis.org/  UDIG - http://udig.refractions.net/confluence/display/UDIG/Home  JUMP – http://jump-project.org/  OpenEV - http://openev.sourceforge.net/ Server/Web  Mapserver - http://mapserver.gis.umn.edu/  GRASS - http://grass.itc.it/  PostGIS - http://postgis.refractions.net/ Tools  Remote Sensing - http://remotesensing.org/tiki-index.php  GDAL/OGR - http://gdal.maptools.org/index.html  PROJ.4 - http://proj.maptools.org/  R-Statistics - http://www.r-project.org/  GMT - http://gmt.soest.hawaii.edu/

19. The End Raster/Vector DST’s Installation Tool Screen Shots

20. Raster DST’s

21. Raster based DST Siuslaw Watershed Restoration Initiative Data Gathering Data Formatting Data Processing PHP Mapscript GRASS

22. Raster based DST $string = “host db_str usr_str”; $connection = pg_connect($string); if (!$connection){ pg_close($connection); echo "Error:Cannot connect to database "; } else { echo "Connected to the SCA database \n"; } $res_pg_exec = pg_exec($connection,$exec_str); PostgreSQL Type Interface for GRASS GIS

23. Raster based DST – GRASS Class class Grass_GIS { var $data_dir; // Where the GRASS data dir is var $rc_dir; // Where the GRASS rc dir is var $bin_dir; // Where the GRASS bin dir is var $location; var $mapset; function Grass_GIS($data_dir, $rc_dir, $bin_dir, $location, $mapset) { $this->data_dir = $data_dir; $this->rc_dir = $rc_dir; $this->bin_dir = $bin_dir; $this->location = $location; $this->mapset = $mapset; } function Grass_GIS_copy($grass_gis) { $this->data_dir = $grass_gis->data_dir; $this->rc_dir = $grass_gis->rc_dir; $this->bin_dir = $grass_gis->bin_dir; $this->location = $grass_gis->location; $this->mapset = $grass_gis->mapset; } function set_location($location) {$this->location = $location;} function set_mapset($mapset) {$this->mapset = $mapset;} function set_data_dir($data_dir) {$this->data_dir = $data_dir;} function set_rc_dir($rc_dir) {$this->rc_dir = $rc_dir;} function set_bin_dir($bin_dir) {$this->bin_dir = $bin_dir;} function run_command($cmd){ $std_output = ""; $grass_exec = ""; // Check that all the required info is available.. if ($this->data_dir != NULL && $this->rc_dir != NULL && $this->bin_dir != NULL && $this->location != NULL && $this->mapset != NULL){ // Here we will write out the script to run... $fp_data = fopen($this->data_dir."/grass_run.sh","w"); // write out text header fwrite($fp_data,"export GISBASE=".$this->bin_dir." \n"); fwrite($fp_data,"export GISDBASE=".$this->data_dir." \n"); fwrite($fp_data,"export GISRC=".$this->rc_dir."/.grassrc6 \n"); fwrite($fp_data,"export ETC=".$this->bin_dir."/etc \n"); fwrite($fp_data,"export PATH=".$this->bin_dir."/bin:".$this->bin_dir."/scripts:".$this->bin_dir. "/garden/bin:\$PATH:/usr/bin:/usr/local/bin/ \n"); fwrite($fp_data,"export LOCATION_NAME=".$this->location." \n"); fwrite($fp_data,"export MAPSET=".$this->mapset." \n"); fwrite($fp_data,"export LOCATION=".$this->data_dir."/".$this->location."/".$mapset." \n"); fwrite($fp_data,"\n"); fwrite($fp_data,$cmd."\n"); fflush($fp_data); fclose($fp_data); $fp_data = fopen($this->rc_dir."/.grassrc6","w"); fwrite($fp_data,"GISDBASE: ".$this->data_dir."\n"); fwrite($fp_data,"LOCATION_NAME: ".$this->location."\n"); fwrite($fp_data,"MAPSET: ".$this->mapset."\n"); fwrite($fp_data,"PAINTER: ppm\n"); fwrite($fp_data,"MAPLP: stuff.ppm\n"); fflush($fp_data); fclose($fp_data); $grass_exec = sprintf("chmod u+x ".$this->data_dir."/grass_run.sh"); exec($grass_exec,$std_output,$std_error); $grass_exec = sprintf($this->data_dir."/grass_run.sh"); exec($grass_exec,$std_output,$std_error); }

24. Raster based DST – PHP GRASS define("GRASSDATA_DIR", "/var/www/html/apps/siuslaw_target/data/GRASSDATA/"); $location_name = "OR_siuslaw“; $mapset_name = "land_targets“; $session_grass_dir = $session_tmp_dir."/GRASSDATA"; // Make a Dir to work in echo " Make a directory to work in inside this session "; $dir_exec = sprintf("mkdir ".$session_grass_dir); exec($dir_exec,$arr,$err1); $dir_exec = sprintf("mkdir ".$session_grass_dir."/rc"); exec($dir_exec,$arr,$err1); $template_path = GRASSDATA_DIR."siuslaw_epa_template"; echo " Copying over new GRASS GIS template "; $dir_exec = sprintf("cp -R ".$template_path." ".$session_grass_dir."/".$location_name); exec($dir_exec,$arr,$err1); // Make a MAPSET echo " Setting up the siuslaw specific area "; $dir_exec = sprintf("mkdir ".$ session_grass_dir."/".$location_name."/".$mapset_name); exec($dir_exec,$arr,$err1); // Copy over the projection info from the PERMANENT area echo " Setting projections "; $dir_exec = sprintf("cp ".$template_path."/PERMANENT/PROJ* ".$ session_grass_dir."/".$location_name."/".$mapset_name."/"); exec($dir_exec,$arr,$err1); $grass_session = new Grass_GIS($session_grass_dir, $session_grass_dir."/rc", "/usr/local/grass-6.0.0", $location_name, $mapset_name); $grass_session->run_command("g.list type=rast");

25. Raster based DST – AML GRASS $arr = $grass_session->run_command("g.region rast=owng_grass@PERMANENT"); echo " Starting to process the Siuslaw data \n"; //AML step //AH2 = (ah_idx + 1) * 10 echo " Creating ah2_idx_grass... anchor habitat priorities \n"; $arr = $grass_session->run_command("r.mapcalc ah2_idx_grass = \(\(ah_idx_grass@PERMANENT+1\)*10\)"); //AML step //own_idx = con(owng eq 31, 40,con(owng eq 83,30,con(owng eq 95,10,20))) echo " Creating own_idx_grass... ownership priorities \n"; $arr = $grass_session->run_command("r.mapcalc own_idx_grass = if\(owng_grass@PERMANENT==31,40,". "if\(owng_grass@PERMANENT==83,30,if\(owng_grass@PERMANENT==95,10,20\)\)\)"); //AML step //reclass tpa (use slice) and setnull trees younger than 15 years //veg_idx = setnull(age_g lt 15, slice(tpa_g,eqarea,40)) echo " Reclasification of tpa_g... to 0-40 \n"; $arr = $grass_session->run_command("r.rescale.eq input=tpa_g_grass@PERMANENT ". "output=tpa_rescale to=0,40"); echo " Creating the veg_idx... trees over ".$tpa_age." years of age \n"; $arr = $grass_session->run_command("r.mapcalc veg_idx_grass = ". "if\(age_g_grass@PERMANENT\>=".$tpa_age.",tpa_rescale,null\(\)\)"); //AML step //FING = setnull(rip_stab eq 0, (ah2 + veg_idx + own_idx +..\infra\rd_d_inv)) echo " Creating FING... the sum of ah2_idx_grass+own_idx_grass+veg_idx_grass+rd_d_inv_grass \n"; $arr = $grass_session->run_command("r.mapcalc fing_grass = ". "if\(rip_stab_grass@PERMANENT==0,null\(\),". "\(ah2_idx_grass+own_idx_grass+veg_idx_grass+rd_d_inv_grass@PERMANENT\)\)")\;

26. Raster Example – TREESystem Mapserver on the front end… GRASS on the back end… Web-based Query

27. Raster based DST - Mapfile LAYER NAME "Fishery Rev" STATUS DEFAULT DATA "./GRASSDATA/CAL_ocean2/FISHERY/cellhd/fishery_rast_rev_zone_8bit" TYPE RASTER PROJECTION "proj=aea" "datum=NAD27" "ellps=clrk66" "lat_1=34" "lat_2=40.5" "lat_0=0" "lon_0=-120" "x_0=0" "y_0=-4000000" "units=m" END

28. Vector DST’s

29. Vector based DST Mapserver PHP -> PHPMapscript -> PostGIS PostGIS

30. Vector based DST - Database // Now connect to the db $host_str = "host=localhost"; $db_str = "dbname=shelf_closure_db"; $usr_str = "user=web_user"; $string = $host_str." ".$db_str." ".$usr_str; $connection = pg_connect($string); if (!$connection){ pg_close($connection); echo "Error: Cannot connect to the database \n"; } else { echo "Connected to the SCA database \n"; } // Create a dynamic VIEW in PostGIS echo " * Create postgresql VIEW * \n"; $exec_str = sprintf("CREATE VIEW ". $previous_session_name."_".$previous_session_id." AS ". "SELECT bk.the_geom, sum(tr.total_lbs*tb.percent) as total_lbs ". "FROM tow%02d_3kblk_intersect as tb, trawl%02d". " as tr, cablk3km as bk, port as pt ". "where tb.blk_gid=bk.gid ".$species_sql." ".$port_sql." ".$gear_sql. " and tb.tow%02d_id=tr.tow_id2 ". "GROUP BY bk.the_geom HAVING count(tr.tow_id2)>2", $start_year_short,$start_year_short,$start_year_short); $result_pg_exec = pg_exec($connection, $exec_str); if (!$result_pg_exec) { printf(" %s \n", pg_errormessage());}

31. Vector based DST – Dump results // Execute the pgsql2shp to create the output echo " * Execute pgsql2shp create the shapefile in session directory * "; $dir_exec = sprintf("/usr/local/pgsql/bin/pgsql2shp -u web_user -f ". $session_tmp_dir."data/target_areas/test_output.shp shelf_closure_db ". "\"SELECT * from ".$previous_session_name."_".$previous_session_id."\""); exec($dir_exec,$arr,$err1); for ($ii=0;$ii<count($arr);$ii++) { echo $arr[$ii]." \n"; } // Drop the dynamic table echo " * DROP the postgresql VIEW * \n"; $result_pg_exec = pg_exec($connection, "DROP VIEW ". $previous_session_name."_".$previous_session_id); if (!$result_pg_exec) { printf(" %s \n", pg_errormessage()); }

32. Vector Example - OCEANSystem Rockfish – SFA - 2001Rockfish – SFA - 2003 Shelf Closure mid-2002

33. AJAX - Prototype function gotVariable(varName, varVal, showWait) { // URL to send the AJAX request to var url = 'http://boris.ecotrust.org/gulf_project/core/SCA_Model/contents.php'; // Push the variable onto the array of parms var newVars = new Array(varName,varVal); varStack.push(newVars); // Build the params for the URL request var pars = ""; for (var i=0; i<varStack.length;i++) { if (i == 0) {pars = pars + varStack[i][0] + '=' + varStack[i][1];} else {pars = pars + '&' + varStack[i][0] + '=' + varStack[i][1];} } // Debug text var boldText = document.createElement('b'); var actionText = document.createTextNode(varName + ' :: '); var action2Text = document.createTextNode(varVal); boldText.appendChild(actionText); var breakText = document.createElement('br'); document.getElementById('placeholder_current').appendChild(boldText); document.getElementById('placeholder_current').appendChild(action2Text); document.getElementById('placeholder_current').appendChild(breakText); // Make the AJAX request var myAjax; myAjax = new Ajax.Updater('placeholder_select', url, {method: 'get', parameters: pars, onLoading: showWaitImage, onComplete: removeWaitImage}); currentStep++; }

34. AJAX - Prototype $action_array = array("0" => "init_action", "Revenue by Species" => "process_rev_by_species", "Revenue by Gear Type" => "process_rev_by_gear", "Revenue Total" => "process_rev_by_total", "Num Vessel Landings" => "process_num_vessel_landings", "Distinct Vesseles" => "process_distinct_vessels", "Average Rev Per Vessel" => "process_average_rev_per_vessel", "Distinct Fishermen" => "process_distinct_fishermen", "Distinct Species" => "process_distinct_species"); if (isset($http_form_vars["Action"])) { $function_ret = $action_array[$http_form_vars["Action"]]($connection, $http_form_vars); } else { // deal with the initial request echo " \n"; $function_ret = $action_array[0]($connection, $http_form_vars); echo " \n"; }

35. AJAX - Prototype fwrite($fp_data, "palette(rainbow(".count($y_string)."))\n"); fwrite($fp_data, "bitmap(\"species_revenue_".$rand_number.".png\", type =\"png256\", height = 2, width = 4, res = 300)\n"); fwrite($fp_data,"plot(x,seq(min(".$y_s."),max(".$y_s.")*1.4,length=length(x)),type=\"n\",xl b=\"Year\",ylab=\"Revenue\")\n"); for ($i=1; $i<=count($y_string); $i++) { fwrite($fp_data,"lines(x,y_".$i.",col=".$i.")\n"); if ($i==1) { $fill_col = $i; } else { $fill_col = $fill_col.",".$i; } fwrite($fp_data,"legend(x=min(x),y=max(".$y_s.")*1.4,c(".$species_s."),fill=c(".$fill_col."), bg=\"white\", ncol=4)\n"); fwrite($fp_data,"title(\"Revenue plot for Species Groups at Port Group ".$port."\")\n"); fwrite($fp_data,"dev.off()\n"); fflush($fp_data); fclose($fp_data); $dir_exec = "/usr/local/bin/R CMD BATCH species_revenue_".$rand_number.".R"; exec($dir_exec,$arr,$err1); echo " \n";

36. GDAL/OGR – Shapefile validity function check_valid_shapefile($shape_file_name) { //echo "Checking that shapefile ".$shape_file_name." is valid... \n"; $valid = TRUE; $duplicate = FALSE; $importance_present = FALSE; $name_temp = array(); // Here we need to check for the validity of the shapefile $dir_exec = sprintf("/usr/local/bin/ogrinfo -al ".$shape_file_name); exec($dir_exec,$arr1,$err1); for ($ii=count($arr1)-1; $ii>=0; $ii--) { if ((substr_count($arr1[$ii],": Integer") > 0) || (substr_count($arr1[$ii],": Real") > 0) || (substr_count($arr1[$ii],": String") > 0)) { // Push them all onto an array $string_to_chop = strstr($arr1[$ii],":"); $key_word = str_replace($string_to_chop,"",$arr1[$ii]); array_push($name_temp,$key_word); //echo $key_word." \n"; }

37. GDAL/OGR – Cont. for ($ii=0; $ii < count($name_temp); $ii++) { for ($iii=$ii+1; $iii < count($name_temp); $iii++) { if ($name_temp[$iii] == $name_temp[$ii]) { // we have a duplicate $duplicate = TRUE; $duplicate_name = $name_temp[$ii]; } if ($name_temp[$ii] == "IMPORTANCE") { $importance_present = TRUE; } if ($duplicate == TRUE) { echo " ERROR: We found a shapefile with duplicate columns named ". $duplicate_name." \n"; $valid = FALSE; } if ($importance_present == FALSE) { echo " ERROR: We found a shapefile named ".$shape_file_name. " without an IMPORTANCE column \n"; $valid = FALSE; } return $valid; }

38. Installation

39. 10 Step Tool Install 1) First install Fedora Core4 with updates 2) Proj.4 tar -xzvf proj-4.4.9.tar.gz cd proj-4.4.9 cd nad/ cp../../proj-nad27-1.2.tar.gz. tar -xzvf proj-nad27-1.2.tar.gz cd.../configure make make install 3) GDAL Installed the ECW SDK (do a config, make and install) Installed the MrSid SDK (moved to /usr/local) wget http://gdal.org/dl/gdal-1.3.1.tar.gz tar -xzvf gdal-1.3.1.tar.gz cd gdal-1.3.1./configure --with-ecw --with mrsid=/usr/local/GeoExpressSDK/ make ogr-all make install which gdalinfo 4) GEOS wget http://geos.refractions.net/geos-2.2.1.tar.bz2 tar -xjvf geos-2.2.1.tar.bz2 cd geos-2.2.1./configure make make install

40. 10 Step Tool Install 5) POSTGRESQL tar -xzvf postgresql-8.1.2.tar.gz cd postgresql-8.1.2 NOTE the LDFLAGS here... this is to support GEOS for postGIS LDFLAGS=-lstdc++./configure --with-perl -- prefix=/usr/local/pgsql_8_1_2 gmake gmake install Add the shared library path to /etc/ld.so.conf : /usr/local/pgsql_8_1_2/lib Run /sbin/ldconfig Finally add the following to the profile file to make paths available: PATH=$PATH:/usr/local/pgsql/bin MANPATH=$MANPATH:/usr/local/pgsql/man export MANPATH Add the postgis user useradd postgis passwd [postgis@localhost pgsql]$ postmaster -D./data/ >./data/logfile.txt 2>&1 & 6) POSTGIS Just make sure that the Postgresql is configured with the LDFLAGS variable set and that the GEOS software is downloaded and installed from Refractions../configure --with-proj --with-geos --with-pgsql make make install Now we can create the database: createdb -O aaronr test_db createlang -U aaronr plpgsql test_db psql -f /usr/local/pgsql_8_1_2/share/postgresql/ contrib/lwpostgis.sql -d test_db

41. 10 Step Tool Install 7) PHP./configure --prefix=/usr/local/php4 --program-suffix=4 -- enable-force-cgi-redirect --with-config-file- path=/etc/httpd/ --with-gd=/usr/local/ --with-jpeg -- with-png --with-tiff --with-zlib --with-freetype-dir -- without-ttf --with-mysql --with-regex=system --enable- dbase --enable-dbx --enable-versioning --with- pgsql=/usr/local/pgsql_8_1_2/ make make install strip sapi/cgi/php cp sapi/cgi/php /var/www/cgi-bin/php4 cp php.ini-dist /etc/httpd/php.ini Made the following mod to /etc/httpd/php.ini: ; Directory in which the loadable extensions reside. ;extension_dir = "./" extension_dir = "/etc/httpd/php_mods" mkdir /etc/httpd/php_mods Added the following to /etc/httpd/conf/httpd.conf # # For PHP scripts as CGI-BIN AddType application/x-httpd-php-cgi.php4.phtml Action application/x-httpd-php-cgi /cgi-bin/php4 Now re-start the server: /etc/rc.d/init.d/httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] 8) MAPSERVER./configure --with-jpeg --with-gd --with-freetype -- with-zlib --with-png --with-pdf --without-tiff -- with-proj --with-threads --with-ogr --with-gdal --with-postgis --with-wfs --with-wmsclient -- with-wfsclient --enable-debug --with- php=/src/php/php-4.4.2 make cp legend mapserv scalebar /var/www/cgi-bin/ cp mapscript/php3/php_mapscript.so /etc/httpd/php_mods/ 9) GRASS CFLAGS="-g -Wall"./configure --with- gdal=/usr/local/bin/gdal-config --with- postgres-includes=/usr/local/pgsql/include/ -- with-postgres-libs=/usr/local/pgsql/lib/ make make install 10) R-Statistics tar -xzvf R-2.1.1.tar.gz cd R-2.1.1./configure make make install

42. Tool Screen Shots

43. Backup - GRASS

44. Backup – R-Statistics

45. Backup – PostGIS Geometry WKT Geometry

46. Backup - QGIS