1. Quick Lesson on Databases
Relational databases are key to managing complex data
You’ve been using relational databases with “Joins” and “Relates” in ArcGIS
GeoDatabases are relational databases
Structured Query Language (SQL) is the primary language for relational databases
You’ve been using SQL statements in ArcGIS to query data

2. Relational Databases Need to represent data with a complex structure
Plot
Species
Tree

3. Database Tables What you’ve seen in ArcGIS only more flexible
Tables are made up of “fields” (columns) and “records” (rows)
Queries are used to combine and subset tables into new tables
Each table should have a unique, integer, ID, referred to as a primary key
Greatly improves query performance

4. Field Data Types Numeric Dates Text Binary Large Objects (BLOB)
Float or integer
Auto numbered, use for primary keys
Dates
YYYY-MM-DD HH:MM:SS.SS
:23:12.34
Text
Specified width
“Variant” width
Binary Large Objects (BLOB)

5. What’s Wrong With This? Tree Query LAT LON MEASYEAR MEASMON MEASDAY
COMMON_NAME HT
1995 6 22
Douglas-fir 49 27 95 66
118 76
147
185
105 89
1996 23 90 96 99

6. Relational Databases Allow us to “relate” tables to:
Reduce the overall amount of data
Removes duplicates
Makes updates much easier
Improves search speeds

7. Entity-Relationship Diagram
ERD
Unified Markup Language (UML)
Relationship Types
One to one
One to many
Many to many
Entities
Plot
Relationships
Species
Tree

8. Plot ID Lat Lon Year Month Day 1 45.446392 -122.236107 1995 6 22 2
Species
Tree ID
Common Name 1
Douglas-fir 2
Ponderosa Pine ID
PlotID
SpeciesID
Height 1 49 2 27 3 95 4 66 5
118 … 12 90 13
Primary Key
Foreign Key

9. Database Normalization
Eliminate duplicate columns from the same table
Move fields that have “duplicate” row entries and move them to a related table
All field entries should be dependent on the primary key
There should be only one primary key in each table

10. Database Dictionary
Defines each of the tables and fields in a database
A database forms the basis for data management behind many GIS projects, web sites, and organizations
Proper documentation is key to long term success!
Database design (including ERDs)
Database Dictionary

11. Geospatial Databases Not required to store spatial data! Provide:
Field types for spatial data: point, polyline, polygon, etc.
Spatial operations: union, intersect, etc.
Spatial queries: return records that overlap with a polygon, etc.
Some provide spatial reference control

12. Relational Databases Enterprise-Level File-Level SQL Server PostgreSQL
MySQL
Oracle
Sybase
File-Level
Geodatabase
MS-Access

13. What we really want What we need from a database:
Distributed, concurrent access (concurrency)
Automatic Backup
Version control
Unlimited amounts of data
Quick data access
Inexpensive
Broad OS Support
File-level copying
GeoSpatial queries, operations, data types

14. What we have SQL Server PostgreSQL ESRI Geodatabase MS-Access
Concurrency
Yes No
Automatic backup
Versioning
Data Size
100s of millions
100,000?
Performance
Fast
Good
Poor
Cost
$600 per CPU
Free
~$10,000 w/ArcGIS
~$400 OS
Windows
Any
File-level copy
Spatial Queries
Spatial data types
Spatial operations

15. Structured Query Language (SQL)
Comes from the database industry
“INSERT”, “DELETE”, and “SELECT” rows in tables
Very rich syntax
Portions of “SELECT” grammar used heavily in ArcGIS:
Selecting attributes
Raster calculator
Geodatabases

16. Transaction SQL “SQL” is a subset of T-SQL
T-SQL allows full management of a database:
Create & drop:
Tables, fields/columns, relationships, indexes, views, etc.
Administrative functions
Varies some between databases

17. Using SQL
All Databases have “query editors” that allow us to write, save, edit, and use SQL queries
Use programming languages to “write” queries and “fetch” records from the database

18. SQL: SELECT SELECT Field1, Field2 FROM TableName
JOIN TableName2
WHERE Filter1 AND Filter 2
GROUP BY Field1,Field2
ORDER BY Field1 [DESC], Field2 [DESC]

19. Selecting Fields SELECT * SELECT Field1,Field2
Returns all fields as new table
SELECT Field1,Field2
SELECT Table1.Field1,Table2.Field1
Return specified fields
SELECT Table1.Field1 AS NewName
Avoids name collisions

20. Plot ID Lat Lon Year Month Day 1 45.446392 -122.236107 1995 6 22 2
Species
Tree ID
Common Name 1
Douglas-fir 2
Ponderosa Pine ID
PlotID
SpeciesID
Height 1 49 2 27 3 95 4 66 5
118 … 12 90 13

21. Example 1: All Fields SELECT * FROM Tree
Returns all the records and fields in tree ID
PlotID
SpeciesID
Height 1 49 2 27 3 95 4 66 5
118 … 12 90 13

22. Example 2: Specific Fields
SELECT PlotID, Height
FROM Tree
Returns all rows but only specified fields
PlotID
Height 1 49 27 95 66
118 … 2 90

23. Example 3: Specific Rows
SELECT PlotID, Height
FROM Tree
WHERE Height>50
Returns all rows but only specified fields
PlotID
Height 1 95 66
118 …

24. Selecting Tables FROM Table1
Returns contents of one table
FROM Table1 INNER JOIN Table2 ON Table2.ForeignKey=Table1.PrimaryKey
Returns records from Table2 that match primary keys in Table1
Does not return all rows in Table1

25. Example 4: Joining
SELECT PlotD,Lat,Lon,Height FROM Trees INNER JOIN Plots ON Trees.PlotID=Plots.ID
PlotID
Height
Lat
Lon 1 49 27 95 66 …

26. Example 4: Joining SELECT PlotD, Height, Lat, Lon, Common_Name
FROM Trees
INNER JOIN Plots
ON Trees.PlotID=Plots.ID
INNER JOIN Species
ON Trees.SpeciesID= Species.ID
PlotID
Height
Lat
Lon
Common_Name 1 49
Douglas-fir 27 95 66 …

27. Selecting Tables (con’t)
FROM Table1 OUTER JOIN Table2 ON Table2.ForeignKey=Table1.PrimaryKey
Returns all matches between Table1 and Table2 and any records in Table1 that don’t match records in Table2
Missing values are NULL

28. Filters or “WHERE” clauses
SELECT * FROM Table1 WHERE (Field1 Operator Value1) BooleanOperator (Field1 Operator Field2)

29. Filter Examples WHERE: Notice: ID = 1 Area < 10000 Area <= 10000
Name = “Crater Lake” (case dependent)
Name LIKE “Crater Lake” (ignores case, except in PostgreSQL!)
Notice:
String values have double quotes
Syntax for strings vary some between databases

30. SQL Comparisons Equals: = Greater than: > Less than: <
Greater than or equal: >=
Less than or equal: <=
Not equal: <>
Like: case independent (except in PostgreSQL), string comparison with wild cards (%)
In PosgreSQL use “upper(..)” or “lower(..)”

31. Boolean Operators A B
A AND B
A OR B
NOT A
NOT B T F

32. More Complex Filter Examples
WHERE:
Name LIKE “Hawaii” AND Area < 10000
Species LIKE “Ponderosa” AND DBH > 1

33. ORDER BY SELECT * FROM Table 1 ORDER BY LastName DESC, FirstName DESC
Careful with performance on large datasets and string fields

34. GROUP BY Aggregates data SELECT Species ,AVG(Height) FROM Trees
GROUP BY Species
Only aggregated fields can appear in SELECT list

35. SQL INSERT
INSERT INTO TableName (Field1,Field2) VALUES (Value1,”Value2”)
String values must be in quotes
Other values can also be in quotes
If the table has an “auto numbered” ID field, it will be added automatically
Otherwise, very difficult to set the ID field

36. SQL DELETE DELETE FROM TableName WHERE ID=Value DELETE FROM Plot
- Deletes one row
DELETE FROM Plot
WHERE PlotID=12
- Deletes all rows with PlotID=12
- Deletes everything in TableName!

37. Database Performance
Default Search
Indexed Search
Primary Key Search

38. Indexes Added to a table Adds overhead to INSERT and DELETEs
Typically for one field
Adds overhead to INSERT and DELETEs
Important for:
Large tables
Complex queries
Especially text searches!
Took query on SeaMap database from over 2 days to several minutes

39. Maintaining Performance
Always use integer, auto numbered primary keys
Avoid iterative or hierarchical queries
Sometimes code is faster:
Do simple query, load into RAM and sort
With REALLY big data, don’t use SQL
NoSQL, accessing data directly, without the use of a relational database package
There are “NoSQL” products in the works
Avoid text searches and sorts

40. Rasters and Databases Don’t put rasters into a database!
Makes it impossible to backup and restore the database
Put a file path to the rasters in the database