1. Presentation by: ABHISHEK KAMAT ABHISHEK MADHUSUDHAN SUYAMEENDRA WADKI
Using text mining on State of the union addresses to gain political insights
Presentation by:
ABHISHEK KAMAT
ABHISHEK MADHUSUDHAN
SUYAMEENDRA WADKI

2. Introduction
Mining the data to find interesting patterns, useful insights, customer data and their relationship - data mining
Text mining - aims at finding useful insights from the dataset comprised of text data.
Examples:
Sentiment analysis.
Google : Search engine
Facebook, Instagram : Hashtags
Text mining on State of the union addresses to gain political insights
Project findings (trends and issues) on interactive dashboards.

3. Big Problem
Text mining involves writing programs to analyze the text data to retrieve something useful from the data
Approaches:
Bag of Words: Uses the entire collection of words that constitute the text to determine the sentiment
TFIDIF: Calculates the word frequency that is relative to the total word count of the document. (Except stop words)

4. Small Problem
Using text mining algorithms to extract the political insights from the presidential addresses of “State of the Union” speeches of every president since 1790.
Project these insights and trends in interactive dashboards.
Find a correlation between the most frequent words that appear in the presidential state of the union addresses and the trends in the issues facing our country.
Emphasis on a particular word in a speech, implies some important trend or issue in that year.

5. Proposed Solution No ready dataset that we can use.
Python scraper using library “Beautiful Soup”(scraping the state of the union website)
Clean the data
Hadoop’s map reduce platform
Determines the word frequency of each word per year.
Divides the entire data into key value pairs
We use this information to deduce the trend of topics in that year’s presidential state of the union address.

6. Proposed Solution - Dashboards
Store the mined data in a database and then project it on to various dashboards.
Planning on D3.js or Chart JS
Few plans to implement dashboards:
Changes in trends between two presidents who served consecutively.
Change of trends in a single president’s entire term.
Determine major trends over a period of time

7. Data analysis and experimental work plan to evaluate the proposed solution
No dedicated training and test set.
Measuring effectiveness - comparing the results of our model with the major events in the history
Ex:
9/11 attack of 2001
Mr. Donald J Trump. Speeches - See the trends related to borders, security, wall, Mexicans, Muslims etc
These aspects show how well the dashboards reflect these results.

8. Related Work Dimensions and features.
A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization.
Dimensions and features.
Beyond TFIDF Weighting for Text Categorization in the Vector Space Model.
How to weigh the word ?
An Improved Feature Space for Sentiment Analysis.
congressional bill - approvals.

9. Related Work (Contd…) Stemming and its effects on TFIDF Ranking.
Why stemming ? Word Isolation.
Refinement of TF-IDF Schemes for Web Pages using their Hyperlinked Neighboring Pages.
Better classification.
An improved TF-IDF approach for text classification
Confidence, language independent.

10. Conclusion
How would text mining algorithms in extracting the political insights help ?
Who would use them Journalists, politicians.