Spatial Databases: Lecture 4 DT211-4 DT228-4 Semester Pat Browne

Outline  We will look in more detail on what happens when a spatial table is constructed in PostgreSQL/PostGIS.  We will describe how construct a table for a sub-set of the historical data set 1.  We take a closer look at a range of OGC queries that can be used in PostGIS 2.  A reminder of viewing and querying using OpenJump  We will look at map accuracy.

PostgreSQL-PostGIS  An object-relational DBMS with PostGIS spatial extensions.  Is completely Open Source  Compliant with OGS’s Simple Features for SQL  Has a spaghetti-like spatial data model  Spatial indexing  Supports OGC types and ‘native types’: point, line, box, path, polygon, and circle geometric types  Topology is under development.  Can perform overlay function  Simple features are based on 2D geometry with linear interpolation between vertices.

PostGIS levels of representation

PostGIS S/W components  PostGiS provides Open Database Connectivity ODBC connectivity. PostGIS includes extensions to the underlying PostgreSQL ODBC drivers which allow transparent access to GIS objects from PostGIS via the ODBC protocol. ODBC connectivity is part of the OGC standard.  PostGIS also provides Java Database Connectivity (JDBC), which is not part of the OGC standard.  GiST (Generalized Search Tree) provides high speed spatial indexing.  PROJ.4 is an open source library that provides coordinate reprojection to convert between geographic coordinate systems.  GEOS (Geometry Engine, Open Source) is a library used by PostGIS to perform all the operations in the OpenGIS Simple Features for SQL Specification. The GEOS library is used to provide geometry tests (ST_Touches(), ST_Contains(), ST_Intersects()) and operations (ST_Buffer(), ST_Union(),ST_Intersection() ST_Difference()) within PostGIS.

PostGIS and OGC standard 1. PostGIS implements and is compliant with the OGC’s Simple Features for SQL standard. 2. PostGIS supports all OGC types: Point, Line, Polygon, MultiPoint, MultiLine, MultiPolygon, GeometryCollection and operations on those types 3. PostGIS uses OGC well-known text format on the SQL command-line to represent GIS features.

Creating a table.  Recall from the labs that the basic steps to create a spatially enable table:  Create a table with the desired non-spatial attributes.  Add a spatial column with as PostGIS/OGC extension AddGeometryColumn  Insert the geometry with a SQL insert & select statements.

Creating a spatial table  On the following slides we will describe how to make a table containing a subset of the historical data set from National Monuments Service 1. We assume that the county 2 table exists and that Dublin is a single region in the county table.  Note a system generated identified ( gid ) is used as the primary key.  It is possible to include a geometry column at table creation time, but the system would not generate integrity constraints so we will stick with this method.

Creating a spatial table, step 1 CREATE TABLE "public"."dublin_historical“ (gid serial PRIMARY KEY, (gid serial PRIMARY KEY, "rmp_prop" int8, "map_symbol" int8, "entity_id" varchar(7), "co_id" int8, "smr_val0" numeric, "nat_grid_e" numeric, "class_desc" varchar(255), "nat_grid_n" numeric, "objectid" int8, "townlands" varchar(255), "scope_n1" varchar(255), "smrs" varchar(255));

Creating a spatial table, step 1  PostgreSQL/PostGIS will respond:  NOTICE: CREATE TABLE will create implicit sequence "dublin_historical_gid_seq" for serial column "dublin_historical.gid"  NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "dublin_historical_pkey" for table "dublin_historical"

Creating a spatial table, step 1  Examine the table (\d): Table "public.dublin_historical" Table "public.dublin_historical" Column | Type | Modifiers Column | Type | Modifiers gid | integer | not null default gid | integer | not null default nextval('dublin_historical_gid_seq'::regclass) nextval('dublin_historical_gid_seq'::regclass) Reset of data Indexes: "dublin_historical_pkey"PRIMARY KEY,btree (gid)

Creating a spatial table, step 2  Here we add the geometry column, PostGIS will automatically generate integrity constraints.  SELECT AddGeometryColumn('public','dublin_h istorical','the_geom','29900','POINT ',2);  We use an version of the Irish National Grid with SRID = The type of the geometry to be stored in the column must be included, in this case it is POINT.

Creating a spatial table, step 2  First system generated constraint ALTER TABLE dublin_historical DROP CONSTRAINT enforce_dims_the_geom; ALTER TABLE dublin_historical ADD CONSTRAINT enforce_dims_the_geom CHECK (ndims(the_geom) = 2); ADD CONSTRAINT enforce_dims_the_geom CHECK (ndims(the_geom) = 2);

Creating a spatial table, step 2  Second system generated constraint ALTER TABLE dublin_historical ADD CONSTRAINT enforce_geotype_the_geom CHECK (geometrytype(the_geom) = ‘POINT'::text OR the_geom IS NULL);

Creating a spatial table, step 2  Third system generated constraint ALTER TABLE dublin_historical ADD CONSTRAINT enforce_srid_the_geom CHECK (srid(the_geom) = 29900);

Creating a spatial table, step 2  The Primary Constraint was created in step1 CONSTRAINT dublin_historical _pkey PRIMARY KEY(gid);

Creating a spatial table, step 3  Next we insert the data from the all Ireland historical table into the newly created dublin_istorical table.  Only data is contained in Dublin is inserted into the new table. Check the OGC & PostGIS documentation on the contains predicate AKA ‘a spatial relationship function’.

Creating a spatial table, step 3 insert into "dublin_historical" ("rmp_prop","map_symbol", "entity_id", "co_id","smr_val0", "nat_grid_e“,"class_desc","nat_grid_n","objectid", "townlands","scope_n1", "smrs", the_geom) select "rmp_prop", "map_symbol", "entity_id", "co_id","smr_val0", "nat_grid_e", "class_desc", "nat_grid_n", "objectid", "townlands", "scope_n1", "smrs", h.the_geom from "county" c, historical h where contains(c.the_geom,h.the_geom) and name = 'Dublin'; name = 'Dublin';

PostGIS system tables  The next few slides describe the built PostGIS tables that provide the spatial functionality. We only outline the main features. For further details, please see the PostGIS 1.5 manual that was handed out in lectures, also available at; 

geometry_columns table Column | Type | Modifiers f_table_catalog | character varying(256) | not null f_table_catalog | character varying(256) | not null f_table_schema | character varying(256) | not null f_table_schema | character varying(256) | not null f_table_name | character varying(256) | not null f_table_name | character varying(256) | not null f_geometry_column | character varying(256) | not null f_geometry_column | character varying(256) | not null coord_dimension | integer | not null coord_dimension | integer | not null srid | integer | not null srid | integer | not null type | character varying(30) | not null type | character varying(30) | not nullIndexes: "geometry_columns_pk" PRIMARY KEY, btree (f_table_catalog, f_table_schema, f "geometry_columns_pk" PRIMARY KEY, btree (f_table_catalog, f_table_schema, f _table_name, f_geometry_column)

spatial_ref_sys table  Displaying a spherical earth on a flat surface requires a projection. This table uses a standard numbering, called the EPGS 1, to describe various projections. Using PostgreSQL’s expanded display we can examine the details for a particular projection representing the Irish National Grid: \x select * from spatial_ref_sys where srid=29900;

spatial_ref_sys table Column | Type | Modifiers Column | Type | Modifiers srid | integer | not null srid | integer | not null auth_name | character varying(256) | auth_name | character varying(256) | auth_srid | integer | auth_srid | integer | srtext | character varying(2048) | srtext | character varying(2048) | proj4text | character varying(2048) | proj4text | character varying(2048) |Indexes: "spatial_ref_sys_pkey" PRIMARY KEY, btree (srid) "spatial_ref_sys_pkey" PRIMARY KEY, btree (srid)

Valid Geometry? (Lakes table on CD) tpostgis=# select IsValid(the_geom) from lakes; tpostgis=# select IsValid(the_geom) from lakes; NOTICE: Self-intersection at or near point NOTICE: Self-intersection at or near point NOTICE: Self-intersection at or near point isvalid isvalid t t f f f t t t

Finding the centre point of a county  To find the centre point of each county:  select name, asText(Centroid(the_geom)) from county;

Finding a bounding box  To find the bounding box of each county:  select name, asText(envelope(the_geom)) from county ;  Or  select name, extent(the_geom)) from county group by name;  Must use ‘ group by ’ which can handle multiple polygons, also extent returns the geometry as text.

Finding the dimensions of a bounding box select name, extent(the_geom) from county where name= 'Dublin' group by name ;  Returns Dublin | BOX( , )  Using the above, we can measure the dimensions of the bounding box: select distance( geomFromText(‘Point….)’),geomFromText(‘Point….)’));

Point in Polygon  We can use a non-OGC function && to find objects.  SELECT name FROM original_county where GeomFromText('POINT( )', 29900) && the_geom;  Finds a given point in Dublin.  When constructing a query it is important to remember that only the bounding-box-based operators such as && can take advantage of the PostgreSQL’s GiST spatial index.

Find things near a point   select townlands from dublin_historical WHERE distance(the_geom, GeomFromText('POINT( )', 29900)) < 100;   When we submit such a query we should have some idea of the type of the result set (use PostGIS manual, OGGSFSQL)

Find things near a point   SELECT c.name, h.townlands FROM county AS c,dublin_historical AS h WHERE distance(h.the_geom, GeomFromText('POINT( )', 29900)) < 1000 and c. Name = 'Dublin County Borough';   Do we know the type of result set here?

Find things near a point SELECT c.name, h.townlands FROM county AS c, dublin_historical AS h st_dwithin WHERE st_dwithin(h.the_geom, PointFromText('POINT( )', 29900),1000) and c. Name = ' Dublin County Borough ';   Similar to previous query but using st_within.

Find things near a point SELECT gid,townlands FROM dublin_historical WHERE st_dwithin(the_geom, GeomFromText('POINT( )',29900),500) ORDER BY st_dwithin(the_geom, GeomFromText('POINT( )',29900),500) LIMIT 100;

Finding the largest county in Ireland   SELECT name, area(the_geom)/10000 AS hectares FROM county ORDER BY hectares DESC LIMIT 1;   Leaving out Northern Ireland   SELECT name, area(the_geom)/10000 AS hectares FROM county   where name != 'Northern Ireland'   ORDER BY hectares DESC LIMIT 1;

What is the length of roads fully contained within Dublin Borough? SELECT c.name, sum(length(r.the_geom))/1000 as roads_km FROM roads AS r, county AS c WHERE r.the_geom && r.the_geom AND contains(c.the_geom,r.the_geom) AND c.name = 'Dublin County Borough' GROUP BY c.name ORDER BY roads_km;

What historical objects are near Dublin roads? SELECT townlands FROM dublin_historical h, roads r, county c WHERE distance(h.the_geom,r.the_geom) < 1000 Dublin County Borough AND c.name = 'Dublin County Borough'; We say near is less than a kilometre from the road.

Overlays  We should distinguish the overlay operation and tests for overlap. The OGCSFSQL use two similar keyword. Table-on-table overlays are possible with the ST_Intersection() function  ST_Intersects(a,b) returns BOOLEAN  ST_Intersection(a,b) returns GEOMETRY   ST_Intersects(a,b)=TRUE | FALSE   ST_Intersection()=GEOMETRY

Efficiency of Search 1 It is expensive to process the exact geometry of an object. Therefore approximations such as bounding boxes (BB) or a convex s (CH) are used to help to examine candidate objects and decided whether a candidate fulfils the query or not. The are used as ‘geometric filter’

Efficiency of Search 1

Network Queries  In order to execute network queries, we need to augment the spatial information used in the OGC Simple Features for SQL standard. We will use pgRouting 1.

Network Queries 1 Shortest Path from Dublin to Waterford

Viewing available Network Queries using pgAdmin III

Network Queries -- The start node 381 is in Dublin -- The destination is node 660 in Waterford -- Must return columns 'id', 'source', 'target' and 'cost' SELECT * FROM shortest_path(' SELECT gid as id, source::integer, target::integer, length::double precision as cost FROM all_roads_e', 381, 660, false, false);

Network Queries The edges are stored in all_roads_e The source and target information was created in OpenJump using Tools | Analysis | Planar Graph. Details will be described in labs: gid | integer | not null default nextval('egclass) id | bigint source | bigint target | bigint the_geom | geometry length | double precision

OpenJump 1  OpenJump GIS can act as a client for PostgreSQL/PostGIS. SQL queries can be issued and view directly from OpenJump. If the data is required in the database for later use, then the user must explicitly create a table from the result set. In short PostgreSQL/PostGIS queries bring a layer into OpenJump, the do not update PostgreSQL/PostGIS.

Using OpenJump with PostgreSQL/PostGIS  To add a PostGIS table to OpenJump select  Layer | Run Datastore Query  The first time you do this you will have to establish a connection with PostgreSQL/PostGIS.  In the Run Datastore Query Window click on the icon on the top right (2 small disks)  This will open the Connection Manager  Click Add

Using OpenJump with PostgreSQL/PostGIS To make a new connection click here

Using OpenJump with PostgreSQL/PostGIS

 To run a PostGIS spatial query in OpenJump select  Layer | Run Datastore Query

Irish Road Work  We will use a simplified version of the Irish Road Network (IRN), consisting of national primary roads augmented with motorways and some dual carriage –ways.  Instructions on how to construct topology were re given in Labs.  We used pgRouting to compute the shortest path according to Dijkstra's algorithm.

How to construct a network  Start with a single road table that consists of the Irish primary roads.  Calculate the length for each road segment and store it in a newly created length column.  Generate the topology using PostgreSQL 1 or OpenJump.  We can now apply various network queries such as shortest path.  These steps were covered in a lab.

Sligo to Cork

Length of all roads sections fully contained in Dublin County Borough.  SELECT Sum(ST_Length(r.the_geom))/1000 AS kilometers FROM all_roads r, dublin_regions dr WHERE ST_Contains(dr.the_geom, r.the_geom) AND dr.name = ‘Dublin County Borough';

What two houses within 500 meters of the Chester Beatty Library have the most residents?  SELECT b1.residents  FROM buildings b1, buildings b2  WHERE  b2.name = 'CHESTER BEATTY LIBRARY' and  ST_DWithin(b2.the_geom,b1.the_geom, 500) ORDER BY residents DESC LIMIT 2;

Latitude of most northern and most southern buildings.  select st_y(st_transform(the_geom,4326)) as latitude from buildings order by latitude desc limit 1;  select st_y(st_transform(the_geom,4326)) as latitude from buildings order by latitude asc limit 1;  select st_y(the_geom) as easting from buildings order by easting desc limit 1;

Counties that have exactly 1 neighbour   SELECT c1.name   FROM all_counties c1, all_counties c2   WHERE touches(c1.the_geom, c2.the_geom) = 'TRUE'   GROUP BY c1.name   HAVING count(c2.name) =1;

Proportions SELECT saps_label,primary_degree10_4,(male1_1 + female1_1) as Pop, ((cast (primary_degree10_4 as float)) / (cast ((male1_1 + female1_1) as float)) * 100) as Grad_percent FROM dublin_eds WHERE primary_degree10_4 IS NOT NULL ORDER BY Grad_percent DESC LIMIT 1; DESC LIMIT 1; saps_label | primary_degree10_4 | pop | grad_percent saps_label | primary_degree10_4 | pop | grad_percent Mansion House A | 279 | 3802 | Mansion House A | 279 | 3802 |

Accuracy  You should calculate the area of county Meath and compare the result to the area stored in the all_counties table, you should have get two different figures for the area of Meath. Which area is correct? To answer this question we would need to know the accuracy of the area stored in the database and the accuracy the map that we used to calculate the area.

Accuracy  Obviously no spatial database or GIS can increase the positional accuracy of a spatial dataset. The accuracy does not change as the viewing scale changes.  So we need to know the scale of the original survey and the expected accuracy at that scale.

Accuracy  The accuracy of a map is dependent on the differences between the true position of features and their representative position in the map.  To find the true position requires highly accurate devices such as industrial grade Global Positioning Systems (GPS) and sophisticated mathematical software.

Accuracy  A common of way of defining positional accuracy for maps is to place limits on the root mean square error (RMSE) for individual position components (the X,Y and possibly Z i.e. height)  The RMSE is derived from the square root of the average of the squared discrepancies when compared to a higher level independent survey. The RMSE is normally defined in terms of ground scale errors (e.g. +- one metre).

A Rough Guide to Accuracy Scale Expected RMSE 1:1000 RMSE < 0.5 metres 1: < RMSE < 2 1:10,000 4 < RMSE < 5 4 < RMSE < 5 1:250, < RMSE < 120

Accuracy  RMSE can be viewed as one criteria for positional accuracy. It is measured with respect to some more precise truth (often GPS). RMSE is a measure of absolute accuracy, with respect to a more precise framework.  Together with an acceptable RMSE a map must be consistent within itself. That is the various components must fit together. This is a measure of relative accuracy.

Accuracy   If data has a nominal scale of say 1:250,000, it still may be more precise than the rough guidelines on slide 57 would indicate.   For example it could be derived from more accurate maps e.g. 1:10,560, 1:50,000, or 1:126,720.   Without meta-data it is difficult for humans or computers to know a map’s accuracy.

Accuracy  The following table shows the expected absolute and relative accuracy values for well defined points within each accuracy category. The relative values apply up to the stated maximum measured distances quoted in the table.

Accuracy From The British Ordnance Survey (OSGB)