I Learnded SQL And So Can You!

Background Started with iovation as a Client Manager in 2015. After seven months, moved to a Data Analyst role. Before iovation, I had experience with data analysis, but no experience retrieving data. Starting my Data Analyst role at iovation, I had… No prior knowledge of SQL No computer science background Used Excel spreadsheets for data analysis Never heard of Postgres My first task: Get familiar with iovation’s Postgres QA databases

Me after one day…

Me after one day…

Learning SQL Takes Focus ”You must feel the force around you. Here… between you and me... The tree... The rock... The SELECT... The FROM... JOINING ON... everything.” - Yoda, The Empire Stikes Back (1980) Luckily, the SQL syntax is pretty easy to understand 😄😄😄

How familiar are we with SQL? I can build and maintain large scale optimized postgres databases I can write fancy SQL queries that run efficiently I regularly use relation databases and write SQL I’ve used relational databases like Postgres and know enough SQL to get by I’ve written a few SQL queries I’ve heard of SQL

Basic Layout of a SQL Query SELECT – The columns you want to see in the result set FROM – The table you are pulling data from JOIN – Other tables you are pulling data from, specifying how they are related to each other WHERE – Criteria to filter your result set GROUP BY – Summarize/compute results into groups ORDER BY – Sort the data

Reading SQL vs Understanding SQL SELECT FROM JOIN WHERE GROUP BY ORDER BY FROM JOIN SELECT GROUP BY WHERE ORDER BY

CONCEPT – JOINs Basic Concept: JOINs combine rows from two (or more) tables, based on a related column. Each row from the two related column is evaluated to see if the values match. When they match, the rows are combined into a temporary table. Columns are then selected from the temporary table to be shown in the results. Core concept of a relational database Many types of JOINs to account for different ways of relating tables to one-another (can even JOIN a table to itself) The database’s Query Optimizer is in charge of figuring out the most efficient way to join these tables together SELECT p.patient_nbr, at.description, p.age FROM patients p JOIN admission_type at ON p.admission_type_id = at.id

CONCEPT – GROUP BY Clause SELECT p.age, COUNT(p.patient_nbr), FROM patients p JOIN admission_type at ON p.admission_type_id = at.id Basic Concept: Group rows together based on matched values, and compute (summarize, rollup,) values based on the groups. - Aggregation

Consistency … not data consistency (althought thats important too) I’m talking about consistent process and organization for your work!

Consistency also means organized… Whatever workflow process you have, be consistent and organized. Consistent requirements Keeps you and the requestor/reviewer on the same page (sometimes it’s hard to get consistent requirements) Consistent validation steps Reviewers can can understand why you did what you did Consistent workflow See Google Doc

Use Aliases No aliases 👎 Has aliases 👍👍👍 Aliases are your friend! Use them and be consistent when using them. See examples. While this top query works without aliases, its confusing. We can’t tell which fields in the SELECT come from which table. Aliases… Simplify your work: don’t have to write out full table names Allow you to rename fields to something more applicable Help with confusion (there happens to be a ‘description’ field in multiple tables...) If left out, often cause errors due to ambiguitiy Give reviewers a helping hand Has aliases 👍👍👍

Validation Extremely important. Often forgotten. Why? Accuracy - You want to be sure that the result-set is the expected one. Code Reviews - Reviewers might want to know what you did. Data results can be surprising - Be prepared to defend your work! Revisiting a query - If you wrote a query a month ago, you might not remember why you did that weird WHERE clause. Know how to validate your query Simple things I look at… Row-counts match: If the results looks odd, it probably isn’t correct. Are there oddities in the data itself?

Tips, Tricks and Gotchas Comments Think ahead about how much data you’re retrieving SELECT * FROM big_table is bad COUNT(*) - Great for validation and learning about a table SELECT 1; SELECT 2+2; SELECT function(); - Great for date math! COALESCE(); WHERE 1 = 1 WHERE TRUE BEGIN TRANSACTION Gotchas NULLs: are they truly NULL values, or are they empty strings? Joining on different data types

Resources stackoverflow.com W3schools.com discuss.codecademy.com Free Not Free, but can be worth it stackoverflow.com W3schools.com discuss.codecademy.com Free Data Sources NOAA climate datasets: http://ncdc.noaa.gov/data-access/quick-links American Economic Ass. (AEA): http://www.aeaweb.org/RFE/toc.php?show=complete Google Finance: https://www.google.com/finance EHDP Large Health Data Sets: http://www.ehdp.com/vitalnet/datasets.htm Airports and their locations: http://www.infochimps.com/datasets/airports-and- their-locations Machine Learning Data Set Repository: http://mldata.org/ Facebook Social Networks (since 2007): http://law.di.unimi.it/datasets.php Lynda.com Udemy.com Corsera.com