1. Twitter Movie Sentiment Using Python, SQL Server, Azure SQL DB, Azure ML, & Power BI
Bradley Ball

2. Speaker Introduction: Bradley Ball
Over 15 Years IT Experience, Former Microsoft MVP
Previous experience DBA, for the U.S. Army, The Executive Office of the President, Sr. SQL DBA Staff Specialist at Publix, Data Platform Practice Manager for Pragmatic Works
Currently Sr. Consultant for Microsoft Premier Support for Developers
Pro SQL Server 2012 Practices Author
Chapter 14 PAGE & ROW COMPRESSION!
Managing Author on Pro Admin 2014 Guide

3. Agenda Movie Industry Sentiment Architecture
How to Collect the Data with Python
Storing the Data
Scoring the Data
Visualize the Data

4. The Movie Industry The movie industry:
Generated 38 billion in revenues in 2016
Employs more than 404,000 people per year

5. What is Sentiment
Sentiment – “A view or attitude toward a situation or event; an opinion”
Some movies are good

6. What is Sentiment
Sentiment – “A view or attitude toward a situation or event; an opinion”
Some movies are…. not

7. Architecture
Collect Twitter Data
Store Data
Score Data
Visualize Data

8. BeautifulSoup /Selenium/Scrapy
Architecture
Score Sentiment
BeautifulSoup /Selenium/Scrapy
Export Table
Visualize Data
JSON to Pandas to CSV
Import CSV

9. BeautifulSoup -IMDB
BeautifulSoup
Scrapes rendered html

10. BeautifulSoup -Twitter Attempt 1
Doesn’t work
Tutorials lie
Twitter uses AJAX, AJAX does not render to html

11. Selenium -Twitter Attempt 2
Each element is its own single dimensional array
Arrays did not line up
Tutorials kinda lie, it works but not how you need it to work

12. Scrapy - Twitter Attempt 3
Victory!
Scrapes AJAX
Scrolls Pages
Uses Twitter’s Advanced Search URL
Doesn’t user Twitter API’s
TweetScraper – Github by Jon Bakerfish

13. JSON to Pandas Files saved in JSON Read in JSON files
Insert to Pandas data frame
Save to a CSV file

14. SQL Server Transform table with T-SQL
Import Flat File created from Pandas into:
SQL Server Transform table with T-SQL

15. Azure SQL DB Export the data Create Scored table
Use this source for Azure ML

16. Azure ML
Import Data
Transform it with R – remove punctuation and special characters
Calculate Sentiment
Export it to Scored table in SQL

17. Azure SQL DB
Transform the data to a time series

18. Power BI
Look for Trends

19. Demo

20. What Did We Just Cover Movie Industry Sentiment Architecture
How to Collect the Data with Python
Storing the Data
Scoring the Data
Visualize the Data

21. References
Scraping Infinite Scrolling Pages -
TweetScraper by Jon Bakerfish -
Web Scraping with Python and BeautifulSoup -