1. INF 141: Information Retrieval
Discussion Session
Week 3 – Winter 2010
TA: Sara Javanmardi

2. Open Source Web Crawlers

3. Heritrix
Extensible, Web-Scale, Distributed
Internet Archive’s Crawler

4. Internet Archive
dedicated to building and maintaining a free and openly accessible online digital library, including an archive of the Web.

5. Nutch Apache’s Open Source Search Engine Distributed
Tested with 100M Pages

6. WebSphinx 1998-2002 Single Machine Lots of Problems (Memory leaks, …)
Reported to be very slow

7. Crawler4j Single Machine Should Easily Scale to 20M Pages Very Fast
Crawled and Processed the whole English Wikipedia in 10 hours.

8. Architecture Should be Extremely Fast,
otherwise it would be a bottleneck

9. Docid Server Key-value pairs are stored in a B+-tree data structure.
Berkeley DB as the storage engine

10. Berkeley DB
Unlike traditional database systems like MySQL and others, Berkeley DB comes in form of a jar file which is linked to the Java program and runs in the process space of the crawlers.
No need for inter-process communication and waiting for context switch between processes.
You can think of it as a large HashMap:
Key
Value

11. Adding a URL to Frontier
public static synchronized int getDocID(String URL) { if there is any key-value pair for key = URL return value else docID= lastdocid+1 put (URL, docID) in storage return -docID }
We add the URL to the frontier
put (docID,URL) in URL - Queue

12. Things to Know:
Crawler4j only handles duplicate detection in the level of URLs, not in the level of the content.
Frontier can be implemented as Priority Queue .

13. Why Priority Queue? Politeness: do not hit a web server too frequently
Freshness: crawl some pages more often than others
E.g., pages (such as News sites) whose content changes often

14. Assigning Priority
Prioritizer assigns to URL an integer priority between 1 and K
Heuristics for assigning priority
Refresh rate sampled from previous crawls
Application-specific (e.g., “crawl news sites more often”)

15. Assignment 3: Crawling digg.com

16. Assignment 3 Programming Part
You can do the assignment 3 in groups of 1, 2 or 3.
Tiffany Siu
James Milewski , Matt Fritz
Kevin Boomhouwer
Azia Foster
James Rose , Sean Tsusaki , Jeff Gaskill
Tzu Yang Huang
Fiel Guhit ,Sarah Lee
Rob Duncan, Ben Kahn, Dan Morgan
Qi Zhu (Chess), Zhuomin Wu
Lucy Luxiao, Melanie Sun, Norik Davtian
Alex Kaiser, Sam Kaufman
Nery Chapeton
Jason Gahagan
Melanie Cheung, Anthony Liu
Chad Curtis , Derek Lee, Rakesh Rajput
Andrew J. Santa Maria
Zack Pelz

17. Crawling One Digg Category …
Initial Seeds

18. digg.com/robots.txt
User-agent: * Disallow: /aboutpost Disallow: /addfriends Disallow: /addim Disallow: /addlink Disallow: /ajax Disallow: /api Disallow: /captcha Disallow: /css/remote-skins/ Disallow: /deleteuserim Disallow: /deleteuserlink Disallow: /diginfull ... … User-agent: Referrer Karma/2.0 Disallow: /

20. Things To Do Download the jar file and import it.
Download the dependency libraries and import them.
Download Crawler4j-example-simple.zip and complete the source code to crawl digg.com

27. Extra Credit Question 1) Extract story id from digg page:
<div  class="news-body"id=" ">
2) Send to API:

28. Questions?