MID-SEM REVIEW

Contents Abstract Introduction Scope Objectives Existing system & Proposed system Algorithms Design Modules with functionalities Implementation Action Plan

Abstract Social Networking sites provide tremendous impetus for Big Data in mining people’s opinion. Public API’s catered by sites such as Twitter provides us with useful data for perusing writer’s attitude with reference to a particular topic, product etc. To discern people’s opinion, tweets are tagged into positive, negative or neutral indicators. This project provides an effective mechanism to perform opinion mining by designing a end to end pipeline with the help of Apache Flume, Apache HDFS, Apache Oozie and Apache Hive. To make this process near real time we study the workaround of ignoring Flume tmp files and removing default wait condition from Oozie job configuration. The underlying architecture employed here is not restricted only to opinion mining but also has a gamut of applications. This paper explores few of the use cases that can be developed into actual working models.

Introduction Twitter data sentiment analysis can be an excellent source of information and can provide insights that can: Determine marketing strategy Improve campaign success Improve product messaging Improve customer service Generate leads

Scope The recent emerging area of interest is sentiment analysis of social issues. Now a day most of the research scholars have been working on Twitter and YouTube comments data set. To perform sentiment analysis the most and common source of data set are web pages, social web site like face book, twitter, YouTube etc. There is a vast scope for research scholars to increase the accuracy level up to some extent by using well designed sentence structure. But, Sarcastic comments are the ones which are very difficult to identify.

Objectives To implement an algorithm for real time classification of twitter data into positive, negative or neutral sets. To extract the meaning of an input text or tweet using natural language processing. To determine the attitude of the mass into various objective sets towards the subject of interest. To improve the accuracy of the analysis using our algorithm.

EXISTING SYSTEM There are many non big data softwares, in the non big data softwares we cannot store large amount of data and we cannot retrieve the data immediately. DISADVANTAGES OF EXISTING SYSTEM No immediate data retrieval Less security Efficiency is less

PROPOSED SYSTEM We proposed System to extract opinion based on sentiment analysis for dynamic tweet data which is stored in HDFS file system ,by using this HDFS file system data transmission is so fast and by default fault tolerance will be given by Big data.   ADVANTAGES OF PROPOSED SYSTEM Actionable intelligence can be achieved Security increases Speed of transmission Huge amount of Data Retrieval can be done

Algorithms Topic Adaptive Sentiment Classification Auto Inclusion of Sensitive Word Keyword Identification

Topic Adaptive Sentiment Classification In social media, a Twitter user may have different opinions on different topics. Thus, topic adaptation is needed for sentiment classification of tweets on emerging and unpredictable topics. The algorithm focuses on cross-domain sentiment analysis on tweets, and we propose a semisupervised topic-adaptive sentiment classification model (TASC). It transfers an initial common sentiment classifier to a specific one on an emerging topic.

TASC has three key components. The semi-supervised multiclass SVM model is formalized. We set feature vector in the model into two parts: fixed common feature values and topic-adaptive feature variables To tackle the content sparsity of tweets, more features are extracted, and split into two views: text and non-text features. The algorithm iteratively minimizing the margins of two independent objectives separately on text and non-text features to learn coefficient matrices.

Auto Inclusion of Sensitive Word This algorithm would simply look up a word in a dictionary, and if not present there, it was probably misspelled. Unfortunately, not all misspelled words result in an unknown word. Misspelled words resulting in existing words are called context-sensitive spelling errors, since a context is required to detect an error. The proposed method mitigates the effect of sparse data by preprocessing the corpus and extracting extra information on PoS tags.

Keyword Identification Keyword extraction from text data is a common tool used by search engines and indexes alike to quickly categorize and locate specific data based on explicitly or implicitly supplied keywords. Various methods of locating and defining keywords have been used, both individually and in concert. Despite their differences, most methods have the same purpose and attempt to do the same thing: using some heuristic (such as distance between words, frequency of word use, or predetermined word relationships), locate and define a set of words that accurately convey themes or describe information contained in the text.

DESIGN

Modules with Functionality Extracting Tweets from twitter based on the query keyword In this module based on the twitter access key and consumer key, we are going to extract tweets based on the query keyword. Storing Tweets to Hadoop In this module extracted tweets are stored into data nodes of hadoop . Based on the Hash tag Retrieve data In this module whatever the data stored in the hadoop we are going to retrieve back in the format of json data, based on the hash tag we will get tweets and finally store it into database.

Modules with Functionality Preprocessing of Tweets Remove Unnecessary words Remove Hyperlinks Remove Special characters Get filtered data   Sentiment Process In this module we have keep first initial positive, negative and neutral words ,based on this initial expansion we are going to give positive,negative and neutral count for the words in the tweet and finally we will detect the sentiment of the tweet

Implementation

Action Plan Storing of tweets into HDFS Preprocessing of tweets Classification of tweets Final Result