2. Open Inside: The Open Source Tools that Power Archive-It Archive-It Partners 2009 Gordon Mohr, Internet Archive November 4, 2009

3. Archive-It Unifies Many Tools Archive-It: managing, designing, monitoring, scheduling, reporting Integrated Tools: collecting, storing, displaying, searching

4. Open Source & Standards from IA 3 open source software projects –Heritrix collecting –Wayback displaying –NutchWAX searching 1 co-developed ISO standard –WARC File Format storing

5. Open Source from Elsewhere Linux Apache/Tomcat MySQL Lucene-Nutch-Hadoop

6. Why Open Source? Open Source Initiative says: “ Open source is a development method for software that harnesses the power of distributed peer review and transparency of process. The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in. ” More than access to source code: Right to change, reuse, extend Wins: –Harmonize formats, practices –Avoid duplication of effort –Reduce costs

7. Projects Genesis: 2003 Internet Archive wanted more control over its own software & collections Discussions with national libraries USA, Canada, UK, France, Iceland, Sweden, Norway, Finland, Denmark, Italy, Australia Desire to share tools, formats, experiences avoid duplicated effort, closed & inflexible tools Formed: International Internet Preservation Consortium (IIPC) http://www.netpreserve.org

8. Heritrix

9. What is Heritrix? Open-source Extensible Web-scale Archival-quality Web crawling software http://crawler.archive.org

10. Heritrix Motivations Deeper, specialized, in-house crawling Open source –Encourage collaboration on features and best practices –Avoid duplication of work, incompatibilities Archival-quality –Perfect copies –Keep up with changing web –Meet evolving needs of Internet Archive and International Internet Preservation Consortium

11. Heritrix Overview Heritrix means heiress Java, modular Project website: http://crawler.archive.org –News, downloads, documentation, issue-tracking –Sourceforge: open source hosting site Source-code control (SVN) Official downloads “ Lesser ” GPL or Apache license – easy reuse Outside contributions welcome

12. Milestones 1.0 release in March 2004 Major releases since: –1.2 new scope options (2004) –1.4 improved memory use (2005) –1.6 remote control (2005) –1.8 scaling (2006) –1.10 protocols, formats, fixes (2006) –1.12 “ smart ” duplicate reduction (2007) –2.0 “ smart ” prioritization (2008) –1.14 WARC, performance (2008-2009)

13. Archive-It Uses Heritrix 1.14.3+ AKA “ 1.15.4 ” WARC/1.0 Many minor fixes Same as all contract/national crawls Available as developer build Will become 1.14.4

14. Heritrix – future Next major release: Heritrix 3.0 –Crawl configuration by ‘ Spring ’ –Scriptable configuration –Web-service remote control Other upcoming priorities –“ Smart ” continuous/automatic revisits (3.2) (from change detection to prediction) –Rich media improvements –Spam/trap/mirror suppression –Automate ever-larger crawls

15. Heritrix – more info Project website –http://crawler.archive.orghttp://crawler.archive.org Source code –Sourceforge ‘ SVN ’ Discussion –http://tech.groups.yahoo.com/group/archive- crawler/http://tech.groups.yahoo.com/group/archive- crawler/ Issues/Bugs –http://webarchive.jira.com/browse/HERhttp://webarchive.jira.com/browse/HER Key IA staff: –Steve Sisney, Gordon Mohr

16. Wayback

17. What is Wayback? Open Source Java Modular Scalable Customizable Web Archive Access Tool http://archive-access.sourceforge.net/projects/wayback

18. Wayback – the beginning Inception in 2005 –Aim: URL-based browsing ‘ as if ’ at previous dates –Contrasts with classic: Open source, diverse installs Java vs. Perl/C Refactored: –Many extension points –Basis for new features & experiments First release: “ 0.2.0 ” December 2005 Now at 1.4.2 (July 2009)

19. Wayback Features Starting with an URL: –See list of captures by date –See extension URLs (same site) –View a capture Once browsing ( “ replay ” ): –Browse web ‘ as it was ’ –Best-match clickthroughs

20. Wayback: Modular Components Query User Interface –Calendar, Search Engine, XML Replay User Interface –Archival URL, Timeline, Proxy Resource Index –CDX, BDB, Remote, Nutch, Aggregated Resource Store –Local ARC, HTTP 1.1 Remote ARC

21. Archive-It Uses Wayback 1.4.2+ UI customized Adds server-side rewriting-mode Available from project source-control Next major release: 1.6.0

22. Wayback – more info Website – http://archive-access.sourceforge.net/projects/wayback/http://archive-access.sourceforge.net/projects/wayback/ Source code – Sourceforge ‘ SVN ’ Discussion –https://lists.sourceforge.net/lists/listinfo/archive-access- discusshttps://lists.sourceforge.net/lists/listinfo/archive-access- discuss Issues/Bugs – https://webarchive.jira.com/browse/ACChttps://webarchive.jira.com/browse/ACC Key IA staff: – Brad Tofel

23. NutchWAX

24. What is NutchWAX? Open Source Java Full-Text Indexing End-User Querying for Web Archives Built on Lucene/Nutch/Hadoop http://archive-access.sourceforge.net/projects/nutch

25. NutchWAX Background Lucene –Open-source Java full-text indexing –Popular, mature Nutch –Extensions to Lucene –For web content, access, scale Hadoop –Spun off from Nutch –Inspired by Google ’ s Map-Reduce

26. NutchWAX Inception in 2005 Nutch Web Archive eXtensions –Utilities for using (W)ARCs as Nutch input –Configuration for date dimension –Handle repeated URLs First release – “ 0.2.1 ” – July 2005 –Now at 0.12.8 (September 2009)

27. Archive-It Uses NutchWAX 0.12.8 Latest official release Recent changes driven by Archive-It –Caching support –Index maintenance processes (merging) –‘ Reboost ’ for reranking

28. NutchWAX – more info Website – http://archive-access.sourceforge.net/projects/nutchwax/http://archive-access.sourceforge.net/projects/nutchwax/ Source code – Sourceforge ‘ SVN ’ Discussion –https://lists.sourceforge.net/lists/listinfo/archive-access- discusshttps://lists.sourceforge.net/lists/listinfo/archive-access- discuss Issues/Bugs – https://webarchive.jira.com/browse/WAXhttps://webarchive.jira.com/browse/WAX Key IA staff: – Aaron Binns

29. WARC

30. What is WARC? IIPC ISO Standard Flexible Simple Format for Web Archive Files http://tinyurl.com/2eusle (drafts)

31. WARC Overview WARC = Web ARChive file format Next generation of ARC, called for by IIPC –ARC format created by the Internet Archive –Over 1PB of ARCs gathered since 1996

32. WARC Goals Store arbitrary metadata (e.g., subject classifier, discovered language, encoding) Data compression and record integrity Store all control information from the harvesting protocol (e.g., request headers) Store the results of data migrations Store a duplicate detection event Distinguishable from the legacy ARC Globally unique record identifiers Deterministic handling of long records (e.g., truncation, segmentation).

33. ARC vs. WARC Both are a simple sequence of content blocks, each introduced by a small text header ARCs only 1-line header + protocol response WARCs add: –multi-line header with extensible fields –New record types: Request, Response, Resource Metadata, Revisit, Conversion, Warcinfo, Continuation

34. What does the future hold?

35. Expand and improve toolset –Driven by user requests, contributions, sponsors –Unify access tools –Verify and improve internationalization

36. What does the future hold? Keep up with the web –New formats, protocols, design techniques –Content challenges: Deep content Spam Interactive applications / AJAX / Javascript

37. Thank You Gordon Mohr Internet Archive Web Group gojomo@archive.org

38. Thank You Gordon Mohr Internet Archive Web Group gojomo@archive.org