Big Data Sources – Web, Social media and Text Analytics Piet Daas, Olav ten Bosch, Ali Hürriyetoglu, Dick Windmeijer THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

ESTP Big Data training course nr. 3 Overview Hands on (learning by doing) Learn how to: Collect ‘data’ – from Web pages and Social media Process ‘data’ Analyse ‘data’ Learn how to extract information from textual data - Text mining, text analytics, Natural Language Processing …

Overview Day 1 Introduction Social media and official statistics Exercise: Create ‘keys’ for Twitter API access Exercise: Connect to Twitter API Exercise: Get user, profile and tweets (in your own language) 3

Overview (2) Day 2 Day 3 Web scraping explained Exercise: Use web robots Web scraping tips and tricks Exercise: Learn how to collect data from websites Feedback Day 3 Text mining and topic identification of tweets Exercise: Analyse tweets: identify topics Sentiment analysis Exercise: Analyse tweets: sentiment & more Natural Language Processing Demonstration Exercise: Extra time for more advanced analysis 4

Overview (3) Day 4 Text mining of web pages Exercise: Analyse document: content Exercise: Analyse web sites: content & topics Overview of the course & dealing with private data Exercise: Time to redo exercises/extra work Feedback Wrapping up, removing data 5

Why analyse text? Texts are a source of information not commonly used in official statistics Potential applications are, automatically: Classify answers to open questions Code description of jobs/educations/products Identify activity code of companies from web site text Detailed product identification from descriptions on web sites Classify cause of death from medical reports Sentiment analysis of messages …

Why analyse text? (2) It is therefore important to: Learn how to extract information from textual data This training course will focus on this topic Goal is to learn the basics by a hands-on approach Is a starting-point for more advanced studies Key steps are: collection, processing and analysis Obtain insights in methods and approaches that can be applied to extract information from texts

Examples of interesting books Manning (1999). Foundations of Statistical Natural Language Processing. MIT Press. Feldman and Sanger (2007) The Text Mining Handbook, Cambridge Univ. Press. Kao, Poteet (2007) Natural Language Processing and Text Mining, Springer Manning, Raghavan and Schütze (2008) Introduction to Information Retrieval, Cambridge Univ. Press. Weiss, Indurkhya, Zhang (2010) Fundamentals of Predictive Text Mining, Springer Aggarwal, Zhai (2012) Mining Text Data, Springer  Miner, Elder, Fast, Hill, Nisbet, Delen (2012) Practical Text Mining and Statistical Analysis for Non-structured Text Data Applications, Elsevier

Practical tips Use our laptops Dual boot Windows / Linux Need to collect your own data! Connect to WiFi (CBS-Public) Web robots: via browser plugin (Windows) Twitter data: either in R or in Python (Linux) Python Notebooks will be distributed

R-packages for text analytics tm: Text Mining Package A framework for text mining applications within R NLP: Natural Language Processing Infrastructure Basic classes and methods for Natural Language Processing SnowballC: Snowball ‘stemmers’ … An R interface to the C libstemmer library … Currently supported languages are Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish. stringr: Wrappers for Common String Operations A consistent, simple and easy to use set of wrappers for string operations. wordcloud: Word Clouds For pretty word clouds RColorBrewer: ColorBrewer Palettes Provides color schemes for maps (and other graphics) twitteR: R Based Twitter Client Provides an interface to the Twitter web API More info: https://cran.r-project.org/package=<name_package>

Text analytics libraries for Python NLTK: Natural Language toolkit Collection of NLP tools TextBlob Built on top of NLTK, especially useful for beginners spaCy Fast NLP implementation Gensim For topic modeling and similarity detection Pattern Web mining module for Python and more Pyparsing For parsing text

Essential step for Twitter studies

Create keys for Twitter API access Make sure you have a Twitter account If not, go to https://twitter.com/signup Login and visit https://apps.twitter.com/app/new Fill in a name, description, web site and agree Copy all keys and tokens (all four), paste them in a text file and save this!! (don’t share them) You will need them during this course!!

14