1. Market Intelligence Analysis
E6895 Advanced Big Data Analytics Task Milestone 1:
Market Intelligence Analysis
<Jia Ji, Tianrui Peng>
February 2nd, 2017

2. Task Introduction
Question: If you never invested in stocks before, how do you decide which stocks to invest in?
Motivation:
designed for small investor
provide information about stock investment
provide data for future stock prediction research or project
Goal:
By extracting sentiment from social media data, we aimed to provide statistical information that can help users decide which stocks to invest in. Users can check our application for daily updates about sentiment information related to stock prediction.
Our also build a stockTwits database that can be used for projects related to stock prediction such as building machine learning algorithms.

3. State-of-art Based on News 2. Based on internet data sources
Evaluating a company’s performance to help with accessing the validity of a stock
2. Based on internet data sources
Google Trends.
Written by Tobias Preis, published in Scientific Reports. Their research shows that “increases in search volume for financially related terms tends to predict large losses in stock market.
Sentiment analysis of Twitter posts [1]
Sentiments can impact sales and financial gains.
This paper focuses on one-day-ahead prediction
of stock based on sentiment analysis of Twitter.
Based on economic model (futures trading model, hedging model)

4. Dataset StockTwits data:
Post messages similar to tweets about stock or exchange record
Show useful information for stock prediction and general trending
We will design and build the database for this task.

5. Result Visualization (Web)
Technologies
Mongo DB
Python
NLTK
Flask
Linode
PROJECT
StockTwits
Result Visualization (Web)
Machine Learning (Stock Prediction)
Collect Data (Python Script)
TASK
Data Analysis (Statistical Methods)
Public API for query data (Python)
Mongo DB
Data Query Interface (Python)

6. Analysis Analysis results: Top N most popular stocks
Top N most bullish stocks
Top N most bearish stocks
We also provide a useful database for any
future research to analyze StockTwits’s data:
Potential usages:
compare user prediction with
real stock price. (accuracy of each user)
Followers
Number of likes
People can use these information for building various machine learning applications to predict stock market.

7. Design database structure and collect information
Task Milestones Plan
Design database structure and collect information
Text analysis
Query data through StockTwits API
Data analysis based on the information in the database
Calculate popularity for each stock base on various information
Sentiment analysis based on our data
Calculate sentiment score for each stock
Create standard API
Provide daily updates to user via our API
Provide StockTwits database with analytical result for future research

8. Reference
Si, Jianfeng, Arjun Mukherjee, Bing Liu, Qing Li, Huayi Li, and Xiaotie Deng. "Exploiting Topic based Twitter Sentiment for Stock Prediction." ACL (2) 2013 (2013):
Schumaker, Robert P., and Hsinchun Chen. "Textual analysis of stock market prediction using breaking financial news: The AZFin text system." ACM Transactions on Information Systems (TOIS) 27, no. 2 (2009): 12.
Yoo, Paul D., Maria H. Kim, and Tony Jan. "Machine learning techniques and use of event information for stock market prediction: A survey and evaluation." In Computational Intelligence for Modelling, Control and Automation, 2005 and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, International Conference on, vol. 2, pp IEEE,
Stock Market Prediction Using Twitter Sentiment Analysis. pdf) 15 (2012). StockMarketPredictionUsingTwitterSentimentAnalysis.pdf
Preis, Tobias, Helen Susannah Moat, and H. Eugene Stanley. "Quantifying trading behavior in financial markets using Google Trends." (2013).