590 Web Scraping – Test 2 Review Topics Removing superfluous garbage Pruning common words Inconsistent data Readings: Text – chapters 7, revisited April 4, 2017

Lecture Titles

Test 2 Sample Questions – Lec23 What do the statements do ? input = re.sub('\n+', " ", input).lower() Write a statement to: To remove footnote references like [23] from text xxx

Test 2 Sample Questions – Lec22 What do the statements do? Explain in detail x = bsObj.find("div", {"id":“y"}).findAll("a", href=re.compile("^(/wiki/)((?!:).)*$")) driver.implicitly_wait(5) Write a statement to: To remove footnote references like [23] from text Explain how to connect to /a/b/c/phantomjs Give a parse function that just logs the site using a scrapy function

Test 2 Sample Questions – Lec21 What do the statements do? (each statement considered individually, not collectively) grammar = r"NP: {<[CDJNP].*>+}“ cp = nltk.RegexpParser(grammar) cp.evaluate(test_sents) Write a statement to: What is meant by IOB accuracy? Precision, Recall, F xxx

Test 2 Sample Questions – Lec20 What are tag patterns matched against? Give three things that match <DT>?<JJ>*<NN> illustrating as much of breadth in the matches as possible. I What is? (what features are used) the unigramChunker the bigramChunker

Test 2 Sample Questions – Lec19 What do the statements do? input Write a statement to: To Is NLTK more like scrapy, Selenium or BeautifulSoup and why? What is meant by the acronym POS? Using the function ngrams(list, n) and FreqDist write a section of code that will print the most frequent trigram in a string named “text”

Test 2 Sample Questions – Lec18 Google page rank is based on keywords and _____. What is meant by inverse document frequency? And what is the logic behind this metric? x

Test 2 Sample Questions – Lec17 What do the statements do? xp = lambda x: response.xpath(x).extract_first() Write xpaths to select : . Can you use BeautifulSoup with all of the other libraries that we have talked about? Which library that we have discussed would be least likely to work with scrapy? What are the major components of the Scrapy architecture? Is it synchronous or asynchrounous? (answer async) What routines from spiders interact with the engine?

Test 2 Sample Questions – Lec16 In scrapy shell? What is the main object return from a request? Why is a scraping project “fragile,” in that it can work one day and then the same code might not work the next day? What should you do to lessen this fragility? Answer (write to be as flexible and tolerant of webpage format changes as possible) What is the crawl template for genspider? What does the parse routine on slide 18 Lecture 16 do?

Test 2 Sample Questions – Lec15 What is a Named Entity? What are all-discussed/some of the categories of Named Entities? <slide 6> What does IOB stand for? What shape features can be used to help identify NE? What single POS is best for indicating a NE? In slide 13 showing a general classifier working on a window of words before anf after the current word. What features are used as input to the classifier? Given a data set of tagged-IOB data explain how to train and test a classifier? Given a classifier and the results: give formulae in terms of tp, fp, tn, fn for Recall Precision F

Test 2 Sample Questions – Lec14 What is scrapy? A library, a framework or both? What do the following statements do? q = response.css("div.quote")[0] t = q.css("div.tags a.tag::text").extract() t = q.css("span.text::text").extract_first() Draw the project folder structure generated by “scrapy startproject x” What are the two main functions in a spider? What command could be used to generate a skeleton for a spider named “spidy?” Start_urls is used be scrapy to generate what function?

Test 2 Sample Questions – Lec13 How is Selenium different from a standard python library such as Beautiful soup? Explain what driver_init does? The answer it initializes the driver, while correct is insufficient in details. In the statement “link = driver.wait.until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, "Sign in to"))) What is the EC? Explain how to find a button named “SubmitButton” and click it. What does the code on slide 13 of Lec 13 do?

Test 2 Sample Questions – Lec12 No sample questions from this lecture.

Test 2 Sample Questions – Lec11 What is UTF-8? An encoding of characters in 8-bytes An encoding of characters in 8-bits None of the above What is the relation of ASCII and UTF-8? How can you convert from UTF-8 to bytes streams that can be written with write? <ch 6 2-getUtf8Text.py>