Corpus Linguistics I ENG 617 Rania Al-Sabbagh Department of English Faculty of Al-Alsun (Languages) Ain Shams University rsabbagh@alsun.asu.edu.eg Week 10

Installation Prerequisites We need to download and install these two before we start: Visual C++ 2015 Build Tools Editra: Python editor Notepad++ Week 10

Corpus Compilation It is always a good idea to look for a ready made corpus either from sources such as the LDC and ERLA or from individual researchers. However, sometimes you have to compile your own corpus. As you compile the corpus, you need to make sure that it follows the criteria of a well-designed corpus. Do you remember what those criteria are? In corpus and computational linguistics, corpus compilation is referred to as corpus harvesting as well. Week 10

Resources for Corpus Harvesting: Print Books Depending on your study, you may compile your corpus from print books, online written resources, or audiovisual resources. For print books, one can check the following for a text machine-readable version of the books Project Gutenberg Oxford Internet Archive If such a version does not exist, one may need to work on a scanned version of the book and use an Optical Character Reader (OCR) software program. OCR programs convert scanned images into text files. They are never 100% accurate but they save much typing time. There are many free online OCRs, though. Week 10

Resources for Corpus Harvesting: Web as Corpus When we compile data from online resources, we are using the “Web as Corpus”. This is a term coined a few years ago and there is an entire series of workshops that carry the same name as well as a SIG. Software programs used to compile corpora from the Web are referred to as scrappers, spiders, or crawlers. Can you guess why? Today, we will learn how to scrap texts from news website, Twitter, and Facebook using Python. Week 10

Installing Python Python is a high-level programming language widely used in corpus and computational linguistics. It comes with many libraries or modules that can help us harvest Web-based texts. To start, we need to download Python from here. Double click the executable file to start the installation. Notice where the folder in which Python will be installed. To check the Python is correctly installed type > python Week 10

Harvesting Newspaper Websites 1 Python has a very nice library/module to harvest articles from Arabic and English newspaper websites. It’s newspaper. To install newspaper, direct your cmd to the folder where Python is installed and then to “Scripts” To download and install newspaper, it is as simple as typing pip install newspaper3k To make sure that the module is correctly installed, run Python shell and type import newspaper Week 10

Harvesting Newspaper Websites 2 Now, we will run the following code as: python getNewsArticles.py > getNewsArticles_Output.txt The code takes as input a list of URLs with one URL per line like this one. It returns the articles titles and texts. Let’s get a closer look at this very basic code to understand it. Week 10

Harvesting Newspaper Websites 3 Now, the question is how to get the URLs of the articles? For that purpose, we will need another code. Run the code! Do you remember how? Do you remember how to direct the output to a file? Now examine the output, what are the newly acquired URLs? Week 10

Harvesting Tweets Twitter does not allow to collect tweets unless you are using its API (Application Programming Interface). API enables Twitter to regulate the scraping process so that it does not lead to too much traffic and no private profiles get violated. Before you scrap with Twitter, you need: Download and install Python 2.7 Install Tweepy library/module Get Twitter API Download the following code and run it. Week 10

Harvesting Facebook Similar to Twitter, Facebook has its API. You can scrap public pages and groups. Before scrapping anything, you need to get an API key from here. We will be using Python 27 and the following two codes: To scrap groups To scrap pages You also need to install urllib2 Python library. For groups, you will need to get the group ID. Week 10

Code Credits Credit for Twitter Scrapper Credit for Facebook Scrapper Week 10