1. Greg R Beaumont October 25, 2014
A New Frontier of Government & Open Data: What it Means to You & Microsoft BI Tools
Greg R Beaumont
October 25, 2014

2. About Me
I have worked on Microsoft BI stack projects since 2007
Using Excel since 1998 (Undergraduate Thesis)
Consulting for GNet Group since 2009:
Education:
St. Mary’s University of Minnesota – B.A. in Biology/Pre-Med
Carlson School of Management (University of Minnesota) – MBA
Self-Directed Learning – Business Intelligence
@GRBeaumont
linkedin.com/in/gregbeaumont

3. Agenda Open Data history, status, and examples
Microsoft BI tools and Open Data
Demos of Open Data with MSBI tools
OpenFDA API (Excel, Power Query)
Seattle Real Time Fire 911 Calls API (Excel, Power Query, PowerPivot, Power View, Power Map)
Chicago Crime & Weather mashup (Excel, Power Pivot, Power View, Azure Machine Learning)
The Future of Open Data
Recommended Links
Q & A

4. What is Open Data?
“Open Data and content can be freely used, modified, and shared by anyone for any purpose”
Opendefinition.org

5. The History & Evolution of Open Data
Integrated Open Data
Stored Data
Open Data
Scientific Data
Government Data
Organizational Data
Governments
Organizations
Corporations
BI Tools
Linked Open Data
Future Opportunities?

6. Open Data Goes Global

7. Open Data Site Examples
United States Government – Data.gov
UK Government – Data.gov.uk
Australian Government – Data.gov.au
City of Chicago – Data.cityofchicago.org
Other Cities on Socrata platform – Cities-Map/ivbs-benj
World Bank Data -
United Nations Data -
cBioPortal for Cancer Genomics (Memorial Sloan Kettering Cancer Center) -
Many, many more…

8. Minneapolis Open Data
The City of Minneapolis passed an Open Data policy in July of 2014: ents/webcontent/wcms1p pdf Open Twin Cities is a group of citizens involved with Open Data in Minnesota, and you can learn more about their group and meetings via Meetup.com:

9. Open Data & MSBI Tools Chicago Crime Open Data
Chicago Crime Open Data Report
NOAA Weather Open Data
NOAA Weather Open Data Report

10. Open Data & MSBI Tools Chicago Crime Open Data
Chicago Crime Open Data Report
Chicago Crime & NOAA Weather Open Data Mashup Report
NOAA Weather Open Data
NOAA Weather Open Data Report

11. Level of complexity, effort, usage difficulty, know-how required
MSBI Tool Spectrum
Power View
Reporting Services
Excel & Excel Services
PerformancePoint
PowerPivot
Least
Business User
Level of complexity, effort, usage difficulty, know-how required
BI Dev
Most

12. Team BI Personal BI Organizational BI PowerPivot for SharePoint
PowerPivot for Excel
Organizational BI
Analysis Services (OLAP) + SharePoint

13. Excel Family of Tools Excel Power Query Power Map Power Pivot
Power View

14. Team BI Personal BI Organizational BI PowerPivot for SharePoint
PowerPivot for Excel
Organizational BI
Analysis Services (OLAP) + SharePoint

15. Power Query ETL for Personal BI
Connect to Numerous File Types and Connections
CSV, Text, XLS, XLSX, etc.
SQL, Azure, Access, Oracle, IBM, MySQL, etc.
OData
APIs
HDFS, Azure HDInsight (Big Data)
Transform, Filter, Unpivot, Concatenate Data
Data Imported into an Excel PivotTable

16. Power Query Demo for an API
Data in the demo is from an OpenFDA Adverse Drug Event API:
The url used for the API in the demo is:
]&count=receivedate
The url specifies a count of events, by Receive Date, between Jan 1, and Jan 1, 2016
Other parameters for the API can be explored at:

17. Power Query Demo for an API
Go To Demo (Excel, Power Query)

18. Demo for a Real Time API
Data in the demo is from the Seattle Real Time Fire 911 Calls API (Socrata SODA API):
xkqj
The url used for the API in the demo is:
xkqj.csv?$order=datetime%20DESC&$limit=300&$where=datetime%20%3E% 20' ‘
The url specifies the most recent 300 events on rows with Address, Latitude, Longitude, Type, Date, Time, and Incident Number
Other parameters for the API can be explored at:

19. Go To Demo (Excel, Power Query, Power View, Power Map)
Demo for a Real Time API
Go To Demo (Excel, Power Query, Power View, Power Map)

20. Thoughts on APIs
APIs are fantastic for organizations providing the data
Limit the amount of data per query
Efficient
Track API usage
APIs are challenging for BI professionals and Data Scientists
“Give me all of the data!”
Multiple queries required to get all of the data for cubes, data mining, etc.

21. Demo for a Data Mashup
Open Data Set #1: City of Chicago Crimes 2001 – Present:
Open Data Set #2: Weather Data for Chicago
From NOAA: OR
From CDC:
Open Data Set #3: Census Data - Selected socioeconomic indicators in Chicago, 2008 – 2012
socioeconomic-indicators-in-C/kn9c-c2s2
Dimension Data Sets:
Date Dimension – Homemade in Excel
Time Dimension – Homemade in Excel

22. A Few Idiosyncrasies in the Data
I only used Weather data for Chicago Midway Airport. A formal solution should have all of the weather stations mapped to community areas.
Community Area Socioeconomic Indicator Census Data is for The Crime Data is for
Population data for Community Areas is two columns for 2000 & 2010
All attempts were made in the reports to account for the slightly offset years of some of the data
For Q & A…How do we determine whether data with idiosyncrasies should be compared?

23. Chicago Crime & Weather Mashup
Find, define, and analyze the data
Pull Data into Excel using Power Pivot and/or Power Query
Architect the conformed model in Power Pivot
Explore the data in Excel
Create dashboards and reports using Excel, Power View, Power Map
Look for trends and relationships in the data by slicing and dicing BI reports and dashboards
Evaluate the data for additional analysis using distribution charts, control charts, and pareto charts in Excel
Create a Machine Learning Model in Azure ML. Train and test the Model.
Browse the results of the Azure ML model in Excel and review the accuracy for predicting future Chicago Crime trends based upon a weather forecast using an Azure Machine Learning (ML) predictive analytics model

24. Before We Start the Demo…
Please note that while this demo has some entertaining implications, it is only a demo and conclusions should not be drawn from it:
I do not know the accuracy of the Open Data sets for Chicago Crime or Weather
I created this demo in a few days, and there has been no peer review
Correlation Does Not Equal Causation
If weather and crime rates correlate, it does not mean that one causes the other
Weather might not cause crime rates to change, just as crime rates may not cause the weather to change. However, it is possible that they correlate and move together.
Example:
If Beer Sales are up at Minneapolis restaurants whenever Bratwurst Sales are up at Target Field in October, there is a correlation.
The cause might be something totally separate, such as the Twins making the playoffs.

25. Demo for Chicago Crime & Weather Mashup
Go To Demo (Excel, Power Query, Power View, Azure Machine Learning)

26. Chicago Crime and Weather Mashup Architecture (Quick Mashup for Demo)
Community Areas (with Socioeconomic Indicator Data)
Weather
Chicago Crime
Time
Date

27. Community Areas (with Socioeconomic Indicator Data)
Chicago Crime and Weather Mashup Architecture (for a Production Architecture)
Community Areas (with Socioeconomic Indicator Data)
Weather
Date
Time
Chicago Crime

28. Video: Police Predict Crimes as in 'Minority Report'

29. A Proposed Future Architecture for Open Data
TODAY
THE FUTURE?
Data.gov
Operational Data Store
Conformed Data Warehouse or Architecture
Data.gov.uk
Scientific Data
Minneapolis Open Data
APIs
Flat files
OData Feeds
Etc.
Store History
Data in One Place
Physical Database
Nonprofit Opportunity?
Open Data Wiki?
Conformed Keys
Conformed Dimensions
Standard Access Methods
Logic Only or Physical?
Direct Query Technology?
Integrated Reports
Self Service Reporting
Reports by Non-IT folk
Usable Data for the Masses

30. Where to start? Open Data Wiki-style approach
ODS (Operational Data Store), standardized keys and agreed upon, standardized common dimensions (Date, Time, Zip Codes, Country-Region-City)
Not every data set will conform, and many will have architectural differences to resolve (granularity, snowflake vs star schema, many-to-many, etc)
Integration into an Organization’s Data Warehouse
Org Dim Date Table
Org Dim Table
Open Data Dim Table
Open Data Fact Table #1
Org Fact Table #1
Org Fact Table #2
Org Dim Zip Code 5 Table
Org Dim Table

31. Recommended Data Sets for Organizational Use
US Bureau of Labor Statistics Open Data (Inflation, CPI, Unemployment, etc.):
Weather Data:
From NOAA: OR
From CDC:
USDA Food Price Outlook:
USDA Feed and Grain Data:
custom-query.aspx
Data.Medicare.Gov
Gas Price Data
formulations-4338e
I could go on for dozens of slides!

32. Some Use Case Examples
Weather – Almost every type of organization can be impacted
Economic Data – Unemployment by geography, CPI, GDP, etc are relevant to most for-profit companies
Crime Data – Could be used for Government, Retail, Security, Fraud Detection
Commodities Data – Food and Feed manufacturing, consumer spending habit projections, agricultural services
Gas Prices – What organization doesn’t rely on logistics?
Medicare Data – Claims and reimbursement by category, hospital, provider

33. Q & A
@GRBeaumont
linkedin.com/in/gregbeaumont