Week 5 – Nov 4, 2015 Data Analysis

Class today Last week & Homework review SQL: Purpose and functions Data Cleaning Pivot Tables, Power Pivots and Power Query (again) exercise Lab on Power Pivots Homework

HOMEWORK & LAB REVIEW

SQL: PURPOSE AND FUNCTIONS

SQL Introduction SQL stands for: Structured Query Language It is the most commonly used relational database language today. SQL works with a variety of different programming languages & vendors such as Visual Basic, Tableau, Excel, etc.

SQL is used for: Data Manipulation – Query 1+ tables – Insert/Delete/Modify tuples in tables Data Definition – Create/Alter/Delete tables and their attributes (won’t do this in this class) Data Administration All are expressed with a SQL statement or command

Tables Explained The schema of a table is the table name and its attributes: Product(PName, Price, Category, Manfacturer) A key is an attribute whose values are unique; we underline a key Product(PName, Price, Category, Manfacturer) The DAMA Dictionary of Data Management

Tables Explained A tuple = a record (all attributes of atomic type) A table = a set of tuples (like a list, but unordered)

Database Set Up Instructions to load Northwind DB and AdventureWorks is available on canvas DEMO

Basic Query Structure

1) SELECT SELECT 1

2) FROM SELECT * FROM Person.Address * asks for every column in a table

3) COUNT(*) SELECT COUNT(*) FROM Person.Address Returns the number of records in a table

4) WHERE SELECT COUNT(*) FROM Person.Address WHERE City=‘Bothell’ WHERE: helps us filter our dataset

3) COUNT 5) GROUP BY 11)DISTINCT SELECT COUNT(PostalCode), City FROM Person.Address GROUP BY City SELECT DISTINCT PostalCode FROM Person.Address

5) GROUP BY6) SUM() We can define an aggregation in SQL as well SELECT Size, AVG(ListPrice) FROM Production.Product WHERE Size IS NOT NULL GROUP BY Size

7) LIKE SELECT City, PostalCode FROM Person.Address WHERE City LIKE ‘U%’ GROUP BY City SELECT * FROM Person.Address WHERE City LIKE '[bsp]%'; %: A substitute for 0+ characters _ : A substitute for a single character [charlist], [!charlist]: sets/ ranges of characters to match, not match

8) TOP SELECT TOP 10 * FROM Person.Person SELECT * FROM Person.Person LIMIT 10 SELECT * FROM Person.Person WHERE ROWNUM <= 10 Microsoft SQL Server MySQL Oracle

9) ORDER BY SELECT TOP 10 * FROM Person.Person ORDER BY LastName (DESC) Microsoft SQL

SQL Joins visual-explanation-of-sql-joins/ visual-explanation-of-sql-joins/ Cool visual interpretation of Joins Tableau example of how joins can go wrong

JOIN ON SELECT * FROM Customers JOIN Orders ON Customers.CustomerID = Orders.CustomerID SELECT * FROM Customers c JOIN Orders o ON c.CustomerID = o.CustomerID

INNER, LEFT, SELECT * FROM Customers INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID SELECT * FROM Customers c LEFT JOIN Orders o ON c.CustomerID = o.CustomerID

OUTER JOINS

Microsoft Architecture for Self-service Business Intelligence

Microsoft Power BI for Office 365 Analyze Visualize Share Find Q&AQ&A Mobile Discover Search, access, and transform public and internal data sources with Power Query Share data views and workbooks refreshable from on-premises and cloud based data sources, with Power BI Sites Easy data modeling and lightning fast in-memory analytics with Power Pivot Bold new interactive data visualizations with Power View and Power Map Ask questions and get immediate answers with natural language query Mobile access through HTML5 and touch optimized apps for Windows 8, RT and iOS tablets Scalable | Manageable | Trusted Microsoft slide

EXCEL PIVOT TABLES

Learning objectives Understanding pivot tables Learn how to create pivot tables Use pivot tables to quickly answer questions about data

What is a Pivot Table A way to summarize, filter, and explore data interactively When we have lots of data, with lots of columns, there is often too much detail and we lose sight of the questions we want answered

Intro to PivotTables A pivot table is essentially a dynamic summary report Data can be in the worksheet or in an external file Allow you to reorganize and summarize data dynamically PivotTables unlock awesome interactivity with the data

How do we create a PivotTable Insert > Tables > Recommended PivotTables Insert > Tables > PivotTable – If you create a table for the range, then Excel will refresh the pivot table without the need to manually indicate the new data range

PIVOT TABLES Sort Count Total Change Aggregation Percent of Total Visualizations Filter Directly Query Databases, including OLAP

PIVOT TABLES PROs/CONS Pros: – Flexible – Works with any SQL report – Calculate large range of metrics very quickly and easily – Easily segmented – Summarizes a large amount of data quickly Cons: – Can be manually time consuming – Excel can take time to churn through really large data sets – Sometimes flying blind – Limited style options – Text not allowed in the data values area

POWER QUERY

Why Power Query? Discovery and connectivity to a wide range of data sources, including publicly available data of all sizes & shapes. Highly interactive and intuitive experience for rapidly and iteratively building queries over any data source, any size. Consistency of experience, and parity of query capabilities over all data sources. Joins across different data sources; ability to create custom views over data that can then be shared with others in your organization Microsoft slide

POWER PIVOT

POWER PIVOTS

POWER PIVOT ADVANTAGES

POWER PIVOT LAB

HOMEWORK

Homework Two parts practice: Part A: More SQL practice Part B: Excel PivotTable and Power Pivot practice Both due by Thurs, 11/12 at 5pm Required reading Reading: Keeping up with the Quants Ch. 5, Excel Ch. TBD

Next week SQL Quiz More Excel (VLOOKUPS) Data collection, cleaning and modeling Stats Review Introduction To Tableau Possibly a Guest Speaker