Intro to PostgreSQL Mike Callahan WFO Louisville September 8, 2009

Topics Relational database theory Relational database theory PostgreSQL Reference PostgreSQL Reference psql psql Snoopy Snoopy SQL syntax SQL syntax SELECT SELECT UPDATE UPDATE INSERT INTO INSERT INTO DELETE FROM DELETE FROM CREATE TEMP TABLE CREATE TEMP TABLE COPY COPY

Databases in AWIPS PostgreSQL Databases Sources Data D2D WHFS OtherApplications SQL psql pg_...

Relational Theory Data are stored in groups called 'tables' or 'relations' Data are stored in groups called 'tables' or 'relations' Tables consist of 'rows' or 'records' and 'columns' or 'fields' Tables consist of 'rows' or 'records' and 'columns' or 'fields' Rows are usually identified by a unique 'key' which may be a single column or a group of columns Rows are usually identified by a unique 'key' which may be a single column or a group of columns Columns can be linked to other tables with similar columns Columns can be linked to other tables with similar columns Good design of the database structure or 'schema' is critical. Good design of the database structure or 'schema' is critical. "All the data in a row should be dependant on the key, the whole key, and nothing but the key." "All the data in a row should be dependant on the key, the whole key, and nothing but the key."

Working with databases Most of the time, you will work with existing databases Most of the time, you will work with existing databases However, you still need to know the schema However, you still need to know the schema We will concentrate on four tasks We will concentrate on four tasks Extracting data Extracting data Changing data Changing data Inserting data Inserting data Deleting data Deleting data

Why PostgreSQL Switching database server from HP to Linux Switching database server from HP to Linux PostgreSQL is free, Informix for Linux is not PostgreSQL is free, Informix for Linux is not PostgreSQL is the most advanced open source relational database, MySQL did not have triggers, PostgreSQL did PostgreSQL is the most advanced open source relational database, MySQL did not have triggers, PostgreSQL did Version on AWIPS is 8.2, latest version is 8.4 Version on AWIPS is 8.2, latest version is 8.4 Documentation is excellent and is at Documentation is excellent and is at We will use version 8.2 for our examples We will use version 8.2 for our examples

PostgreSQL Documentation Preface Preface Gives history, conventions, etc. Gives history, conventions, etc. Tutorial Tutorial A quick introduction to SQL A quick introduction to SQL SQL Language SQL Language A description of the standard and the PostgreSQL additions A description of the standard and the PostgreSQL additions Reference: SQL Commands Reference: SQL Commands Details of each command Details of each command

psql … / interactive/app-psql.html The standard PostgreSQL application to work with databases The standard PostgreSQL application to work with databases It uses a superset of standard SQL It uses a superset of standard SQL It can be interactively like DBaccess, or in scripts like sqlcmd It can be interactively like DBaccess, or in scripts like sqlcmd

Common psql Command Line Options -A, --no-align: set output to non-aligned, no padding -A, --no-align: set output to non-aligned, no padding -c sql, --command sql: execute the sql command and then exit -c sql, --command sql: execute the sql command and then exit -d name, --dbname name: name of database, same as name as the first non-option argument -d name, --dbname name: name of database, same as name as the first non-option argument -f name, --file name: use name as the source of commands -f name, --file name: use name as the source of commands -o name, --output name: put the output in name -o name, --output name: put the output in name -q, --quiet: suppress welcome messages -q, --quiet: suppress welcome messages -t, --tuples-only: suppress print column names, result row counters, etc -t, --tuples-only: suppress print column names, result row counters, etc -?, --help: get command line help -?, --help: get command line help

Common psql Meta-commands \a: toggle output format (aligned/unaligned) \a: toggle output format (aligned/unaligned) \c name: connect to a database name \c name: connect to a database name \copy table: copy a table to/from a file \copy table: copy a table to/from a file \d : list all tables (display) \d : list all tables (display) \d table: show information about a table \d table: show information about a table \e: edit the query buffer \e: edit the query buffer \g: execute the query buffer (go) \g: execute the query buffer (go) \h command: display help on command \h command: display help on command \i name: read name into query buffer (input) \i name: read name into query buffer (input)

Common psql Meta-commands \o name: send output to name \o name: send output to name \p: display the query buffer \p: display the query buffer \q: quit the program \q: quit the program \r: resets (clears) the query buffer \r: resets (clears) the query buffer \t: toggle the output headers on/off \t: toggle the output headers on/off \w name: write the query buffer to name \w name: write the query buffer to name \! command: execute the Linux command \! command: execute the Linux command \?: help on meta-commands \?: help on meta-commands

Interactive psql Typing in a query loads the query buffer Typing in a query loads the query buffer The command will not run unless terminated with a semicolon (;) The command will not run unless terminated with a semicolon (;) Meta-commands all start with blackslash (\) Meta-commands all start with blackslash (\) Exit with \q Exit with \q

Snoopy Snoopy is an interactive, GUI front-end to psql Snoopy is an interactive, GUI front-end to psql It was designed for the hydro database, but will work with any AWIPS database It was designed for the hydro database, but will work with any AWIPS database Queries are in a scrolling window, data is in a separate scrolling window Queries are in a scrolling window, data is in a separate scrolling window Help is available about tables, columns, rows, SQL syntax, and hydro database data dictionary Help is available about tables, columns, rows, SQL syntax, and hydro database data dictionary Available on the AWIPS LAD Available on the AWIPS LAD

Snoopy Screen Shot

Snoopy Help Available only for the hydro database

SQL for PostgreSQL …/interactive/sql-commands.html SQL, pronounced “sequel”, is the standard way of working with relational databases SQL, pronounced “sequel”, is the standard way of working with relational databases The standard is well-supported but weak so every database manager adds commands to the standard The standard is well-supported but weak so every database manager adds commands to the standard In these examples, SQL keywords will be CAPITALIZED In these examples, SQL keywords will be CAPITALIZED However, case is NOT important except in strings However, case is NOT important except in strings

Basic Data Types in SQL Character, Varchar: A sequence of alphanumeric characters enclosed in single quotes; ‘a string 123’ Character, Varchar: A sequence of alphanumeric characters enclosed in single quotes; ‘a string 123’ To include a single quote in a string double it; 'Mike''s string' To include a single quote in a string double it; 'Mike''s string' Integers: A number without a decimal point; 125, -2 Integers: A number without a decimal point; 125, -2 Real: A number with a decimal point; , Real: A number with a decimal point; , Decimal, Numeric: Fixed decimal point, accurate but slow math Decimal, Numeric: Fixed decimal point, accurate but slow math Boolean: TRUE or FALSE Boolean: TRUE or FALSE in psql TRUE is displayed as t, FALSE is displayed as f in psql TRUE is displayed as t, FALSE is displayed as f

Date/Time Data Types in SQL Date: The word followed by a date enclosed in single quotes; DATE ‘ ’ Date: The word followed by a date enclosed in single quotes; DATE ‘ ’ Other formats are allowed but this is the standard. Other formats are allowed but this is the standard. Time: The word followed by a time enclosed in single quotes; TIME ‘02:34:00’ Time: The word followed by a time enclosed in single quotes; TIME ‘02:34:00’ Seconds and colons are optional. Seconds and colons are optional. Timestamp: Both; TIMESTAMP ’ :20:00’ Timestamp: Both; TIMESTAMP ’ :20:00’ Interval: A time interval; INTERVAL ‘2 days’, INTERVAL ‘12 hours’ Interval: A time interval; INTERVAL ‘2 days’, INTERVAL ‘12 hours’ Many times you in PostGreSQL you don't need to type the descriptor. For example: ' ', '02:34', etc. Many times you in PostGreSQL you don't need to type the descriptor. For example: ' ', '02:34', etc.

Special Values TIMESTAMP ‘NOW’ TIMESTAMP ‘NOW’ The current time, date, or timestamp The current time, date, or timestamp DATE ‘TODAY’ DATE ‘TODAY’ The current date or midnight timestamp The current date or midnight timestamp TIMESTAMP ‘TOMORROW’ TIMESTAMP ‘TOMORROW’ Tomorrow date or midnight timestamp Tomorrow date or midnight timestamp DATE ‘YESTERDAY’ DATE ‘YESTERDAY’ Yesterday date or midnight timestamp Yesterday date or midnight timestamp TIME ‘ALLBALLS’ TIME ‘ALLBALLS’ Midnight 00:00:00 UTC time Midnight 00:00:00 UTC time

Common SQL Commands SELECT SELECT Extract data from one or more tables Extract data from one or more tables UPDATE UPDATE Change data in a table quickly Change data in a table quickly INSERT INTO INSERT INTO Add new data to a table Add new data to a table DELETE FROM DELETE FROM Remove data from a table Remove data from a table CREATE TEMP TABLE CREATE TEMP TABLE Create a temporary table Create a temporary table COPY COPY Send the contents of a table to a file or vice versa Send the contents of a table to a file or vice versa

SELECT Most SQL work involves creating SELECT statements Most SQL work involves creating SELECT statements You cannot harm data using a SELECT Command You cannot harm data using a SELECT Command SELECT statements can be simple but can also be almost incomprehensible SELECT statements can be simple but can also be almost incomprehensible SELECT statements are broken up into clauses SELECT statements are broken up into clauses FROM, WHERE, GROUP BY, HAVING, UNION, INTERSECT, EXCEPT, ORDER BY, LIMIT, FOR UPDATE FROM, WHERE, GROUP BY, HAVING, UNION, INTERSECT, EXCEPT, ORDER BY, LIMIT, FOR UPDATE We will only look at a few simple queries We will only look at a few simple queries

Basic SELECT SELECT SELECT List of columns to extract List of columns to extract FROM FROM List of tables to work with List of tables to work with WHERE WHERE Condition to limit selection Condition to limit selection GROUP BY GROUP BY Grouping expression Grouping expression HAVING HAVING Condition to limit grouping Condition to limit grouping ORDER BY ORDER BY Sort order Sort order LIMIT LIMIT Number of rows to extract Number of rows to extract

SELECT Command Structure SELECT FROM WHERE GROUP BY ORDER BYLIMITHAVING List of columns List of tables ConditionExpression ConditionSort orderNumber

Building a SELECT Query You must know the schema (structure) of your database to know the tables and the columns you want to extract You must know the schema (structure) of your database to know the tables and the columns you want to extract Know the condition you want to filter the selection Know the condition you want to filter the selection Know the sort order you want to see the data, the order will be undefined if not specified Know the sort order you want to see the data, the order will be undefined if not specified

Example Tables and columns Tables and columns location: lid, name location: lid, name Table names need to be included only if column names are ambiguous Table names need to be included only if column names are ambiguous Condition Condition location.state = 'KS' location.state = 'KS' Order Order location.name location.name SELECT lid, name FROM location WHERE state = 'KS' ORDER BY name; Find all IDs and names in Kansas sorted by name

Joining Two Tables Tables and columns Tables and columns location: lid, name location: lid, name observer: lastname observer: lastname Condition Condition location.lid = observer.lid and location.state = 'KS' location.lid = observer.lid and location.state = 'KS' Sort order Sort order observer.lastname observer.lastname Retrieve the list of locations, ids, and observers in KS sorted by observer's last name SELECT lid, name, lastname FROM location, observer WHERE location.lid = observer.lid AND location.state = 'KS' ORDER BY lastname;

More on WHERE Clause Comparison operators =, != or <>, >, >=, <, <= Logical operators AND, OR, NOT String concatenation 'a' || 'string' BETWEEN operator WHERE fs BETWEEN 10 AND 50 IN operator WHERE state IN ('KY', 'IN', 'OH') IS NULL test WHERE fs IS NULL

Matching in Character Columns LIKE Used to set up search targets % - zero or more characters _ - a single character WHERE lid LIKE '___K1' or lid LIKE '___K2' ~ Used to set up POSIX regular expression searches WHERE lid ~ '…K[12]' Use escape notation for backslash expressions '…\w\d' becomes e'…\\w\\d'

Substrings Use to extract part of a character column based on position (count from 1) Use to extract part of a character column based on position (count from 1) SUBSTRING(column FROM begin FOR length) SUBSTRING(column FROM begin FOR length) Ex. show phone number of observers in the Kansas City KS Metro area code Ex. show phone number of observers in the Kansas City KS Metro area code SELECT name, hphone FROM observer WHERE SUBSTRING(hphone FROM 1 FOR 3) = '913' SELECT name, hphone FROM observer WHERE SUBSTRING(hphone FROM 1 FOR 3) = '913'

SELECT tricks Testing an expression SELECT SUBSTRING('ABCDE' FROM 1 FOR 3) Get all the fields in a table SELECT * FROM location Name shortcut SELECT l.lid, lastname FROM location l, observer o WHERE l.lid=o.lid AND lastname LIKE 'a%' Rename columns SELECT lid AS Location_ID FROM location... Eliminate duplicate rows SELECT DISTINCT county FROM location Order shortcut SELECT lid, name FROM location ORDER BY 2 Descending order...ORDER BY lid DESC

Date/Time/Timestamp Functions AGE(timestamp, [timestamp]) AGE(timestamp, [timestamp]) calculate interval between two timestamps, second argument defaults to NOW calculate interval between two timestamps, second argument defaults to NOW …WHERE AGE(obstime) < INTERVAL '6 hours' …WHERE AGE(obstime) < INTERVAL '6 hours' EXTRACT(unit from timestamp) EXTRACT(unit from timestamp) extract the unit from a timestamp extract the unit from a timestamp common units: century, day, decade, dow (day of week Sun=0, Mon=1, etc), doy (Julian date), hour, minute, month, quarter, second, timezone, week, year common units: century, day, decade, dow (day of week Sun=0, Mon=1, etc), doy (Julian date), hour, minute, month, quarter, second, timezone, week, year …WHERE EXTRACT(dow from datcrst) = 3 …WHERE EXTRACT(dow from datcrst) = 3

Group Functions COUNT() Instead of displaying the rows, just count them SELECT COUNT(*) FROM observer AVG() Calculate the average of a column MAX() Calculate the maximum in a column MIN() Calculate the minimum in a column SUM() Calculate the sum of a column

The GROUP BY Clause When using the previous functions, the GROUP BY clause tells what to group We want the highest crest at EVERY location, sorted by ID If we don't use GROUP BY we will get the highest crest of ALL IDs instead of the highest crest at EVERY ID. SELECT l.lid, MAX(stage) FROM location l, crest c WHERE l.lid = c.lid GROUP BY l.lid ORDER BY l.lid;

The UPDATE Command Used to quickly change the values in a column Used to quickly change the values in a column This command can save a lot of work but can also be dangerous This command can save a lot of work but can also be dangerous Has SET, FROM, and WHERE clauses Has SET, FROM, and WHERE clauses Here is a trick: Here is a trick: Create the WHERE clause but use SELECT * Create the WHERE clause but use SELECT * You will see the rows you are going to change You will see the rows you are going to change Build rest of query Build rest of query

The UPDATE Clauses SET SET The column and the value you want to assign it The column and the value you want to assign it FROM FROM A list of tables you used in the SET clause A list of tables you used in the SET clause WHERE WHERE The conditions the data must match The conditions the data must match

UPDATE Command Structure UPDATE FROM WHERE Table List of tables Condition SET Column and value

Using UPDATE Ex. 913 area code changes to 914 Ex. 913 area code changes to 914 Table and column Table and column observer: hphone observer: hphone Value Value ‘914’ || SUBSTRING(hphone FROM 4) ‘914’ || SUBSTRING(hphone FROM 4) Condition Condition SUBSTRING(hphone FOR 3) = '913' SUBSTRING(hphone FOR 3) = '913' UPDATE observer SET hphone = '914‘ || SUBSTRING(hphone FROM 4) WHERE SUBSTRING(hphone FOR 3) = '913' SELECT * FROM observer WHERE SUBSTRING(hphone FOR 3) = '913' Check WHERE Clause UPDATE rows

The INSERT INTO Command Insert data into a new row Insert data into a new row Has only VALUES clause Has only VALUES clause Can also use a SELECT query Can also use a SELECT query

The INESRT Clause VALUES VALUES A list of data that you want to insert A list of data that you want to insert If the order of the columns is not listed, it must match the order defined in the schema If the order of the columns is not listed, it must match the order defined in the schema For safety sake, always list the columns For safety sake, always list the columns SELECT SELECT A standard select query, but the schema of the result must match the target A standard select query, but the schema of the result must match the target

INSERT Command Structure INSERT INTO VALUES Table, column list List of values INSERT INTO SELECT Table, column list Select query -or-

Using INSERT Want to add a new row in datum Want to add a new row in datum datum schema is datum schema is lid - location id lid - location id ddate - datum date ddate - datum date elev - elevation elev - elevation Our values are ABEK1, , Our values are ABEK1, , INSERT INTO datum (lid, ddate, elev) VALUES ('ABEK1', ' ', );

Using INSERT (con't) Want to add data from a temporary table named junk into the height table Want to add data from a temporary table named junk into the height table junk and height have the same schema junk and height have the same schema INSERT INTO height SELECT * FROM junk;

The DELETE FROM Command Remove rows Remove rows Has only WHERE clause Has only WHERE clause There is no quick undo so be careful There is no quick undo so be careful If you accidentally leave off the WHERE clause, ALL the rows will be deleted! If you accidentally leave off the WHERE clause, ALL the rows will be deleted! Here is another trick: Here is another trick: Create the WHERE clause but use SELECT * Create the WHERE clause but use SELECT * You will see the rows you are going to delete You will see the rows you are going to delete Change SELECT * to DELETE Change SELECT * to DELETE

The DELETE Clause WHERE WHERE The condition the rows must match The condition the rows must match

DELETE Command Structure DELETE FROM WHERE TableCondition

Using DELETE Want to remove an observer to make a location inactive Want to remove an observer to make a location inactive The location ID is CANK1 The location ID is CANK1 DELETE FROM observer WHERE lid = 'CANK1' SELECT * FROM observer WHERE lid = 'CANK1' Check WHERE Clause DELETE row

The CREATE TEMP TABLE Command Creates a temporary table that can be used in future queries Creates a temporary table that can be used in future queries Table is automatically deleted once psql session is completed. Table is automatically deleted once psql session is completed. In Snoopy, table is deleted at end of last statement in query. In Snoopy, table is deleted at end of last statement in query. It has a simple structure It has a simple structure

CREATE TEMP TABLE Command Structure CREATE TEMP TABLEAS Table Select query

Using CREATE TEMP TABLE Want to copy gage readings for the last three days into a temp table, junk Want to copy gage readings for the last three days into a temp table, junk CREATE TEMP TABLE junk AS SELECT * FROM height WHERE AGE(obstime) < INTERVAL '3 days';

The COPY Command There is a quick and easy way to copy a table to a file There is a quick and easy way to copy a table to a file You could process a table without working in SQL You could process a table without working in SQL This is similar to the Informix UNLOAD command This is similar to the Informix UNLOAD command It has many options but we will only look at a few It has many options but we will only look at a few It combines well with the CREATE TEMP TABLE command It combines well with the CREATE TEMP TABLE command You must be logged on as postgres for this command to work You must be logged on as postgres for this command to work

COPY Command Structure COPY TO WITH FileOptions Table FROM

Example COPY Commands COPY observer TO 'observer.dat' WITH DELIMITER '|'; COPY observer TO 'observer.dat' WITH DELIMITER '|'; Same as Informix UNLOAD command except default delimiter is a TAB Same as Informix UNLOAD command except default delimiter is a TAB COPY crest TO 'crest.csv' WITH CSV; COPY crest TO 'crest.csv' WITH CSV; Good way of moving data into Excel Good way of moving data into Excel COPY observer FROM 'observer.dat' WITH DELIMITER '|'; COPY observer FROM 'observer.dat' WITH DELIMITER '|'; Moving data into a table Moving data into a table

Combining CREATE TEMP TABLE and COPY We want to copy today's stages into a file, stg_tdy.dat We want to copy today's stages into a file, stg_tdy.dat CREATE TEMP TABLE junk AS SELECT * FROM height WHERE AGE(obstime) < INTERVAL '1 day'; COPY junk TO stg_tdy.dat;

In Conclusion PostgreSQL may not be as popular as MySQL, it will do what we need it to do PostgreSQL may not be as popular as MySQL, it will do what we need it to do The PostgreSQL documentation is excellent and available on-line as HTML or off-line as PDF The PostgreSQL documentation is excellent and available on-line as HTML or off-line as PDF psql is good to use in scripts psql is good to use in scripts Snoopy is good to use interactively Snoopy is good to use interactively The best way to learn SQL is to play with it The best way to learn SQL is to play with it You cannot harm data by SELECT queries You cannot harm data by SELECT queries