1 Archive-It Training University of Maryland July 12, 2007

2 Archive-It Mission Help memory institutions preserve the Web Provide web based archiving and storage capabilities No technical infrastructure required User-friendly application

3 Archive-It Application Open Source Components Heritrix: web crawler Arc File: archival record format (ISO work item) Wayback Machine: access tool for viewing archived websites (Arc files) Nutchwax: bundling of Nutch (an open source search engine) used to make archived sites full text searchable All developed by Internet Archive:

4 Web Archiving Definitions Host: a single or set of networked machines, designated by its Internet hostname (ex, archive.org) Scope: rules for where a crawler can go Sub-domains: divisions of a larger site named to the left of the host name (ex. crawler.archive.org)

5 Web Archiving Definitions Seed: starting point URL for the crawler. The crawler will follow linked pages from your seed url and archive them if they are in scope. Document: any file with a distinct URL (image, pdf, html, etc).

6 General Crawling Limitations Some web content cannot be archived: Javascript: can be difficult to capture and even more difficult to display Streaming Media Password protected sites Form driven content: if you have to interact with the site to get content, it cannot be captured. Robots.txt: The crawler respects all robots.txt files (go to yourseed.com/robots.txt to see if our crawler is blocked)

7 Archive-It Crawling Scope Heritrix will follow links within your seed site to capture pages Links are in scope if they the seed is included in the root of their URL All embedded content on seed pages is captured Sub-domains are NOT automatically crawled Can specify path (i.e. limit crawler to single directory* of host) - ex: *Always end seed directories with a ‘/’

8 Seed and Scope Examples Example seed link: is in scopewww.archive.org/about.html link: is NOT in scopewww.yahoo.com embedded pdf: is in scopewww.rlg.org/studies/metadata.pdf Embedded image: is in scopewww.rlg.org/logo.jpg link: crawler.archive.org NOT in scope Example Seed Link NOT in scopewww.archive.org/webarchive.html

9 Changing Crawl Scope Expand crawl scope to automatically include sub-domains using Scope Rules on the ‘edit’ seed page Use ‘crawl settings’ to constrain your crawl by limiting overall # of documents archived, block or limit specific hosts by document number or regular expression.

10 Access Archived pages are accessible in the Wayback Machine 1 hour after crawl is complete (sooner for larger crawls) Text Searchable 7 days after crawl is complete Public can see your archives through text search on Archive-It templates web pages (hosted on archive-it.org), or partner made portals.

11

12 Creating Collections

13 Creating Collections Your collection needs: A name chosen by your institution A unique collection identifier: this is an abbreviated version of your collection name Seeds: these are the starting point URLs where the crawler will begin its captures Crawl frequency: how often your collection will be crawled (you can change this at the seed level once the collection is created) Metadata: adding metadata is optional for your collection except for the collection description which will appear on public Archive-It site

14 Crawl Frequency Options Daily crawls last 24 hours, all other crawls last 72. Seed URLs within the same collection can be set to different frequencies. The Test frequency allows you to crawl seeds without gathering any data so the crawl will not count against your total budget. In a test crawl all regular reports are generated. Test crawls only run for 72 hours and will crawl up to 1 million documents. Test crawls must be started manually (from Crawls menu).

15 Managing Collections

16 Editing Seeds

17 Enabled: Scheduled for crawling (limited to 3) Disabled: publicly accessible, not scheduled for crawling (unlimited) Dormant: publicly accessible, not scheduled for crawling (unlimited)

18 Crawl Settings Advanced crawl controls: crawl and host constraints All controls found under crawl settings link

19 Crawl Constraints Limit the number of documents captured per crawl instance (by frequency) Captured URL totals could be up to 30 documents over limit, due to URLs in crawler queue at the time limit is reached

20

21 Host Constraints Block or limit specified hosts from being crawled Blocks/limits apply to all named sub- domains of a host Using Regular Expressions here is OPTIONAL

22

23 Monitoring Crawls

24 Monitoring Crawls

25 Manually Starting a Crawl Select the crawl frequency you want to start Using this feature will change your future crawl schedule Should always be used to start test crawls Crawl should start within 5 minutes of start

26 Reports

27 Reports are available by crawl instance

28 Archive-It provides 4 downloadable, post-crawl reports Top 20 Hosts: lists the top 20 hosts archived Seed Status: reports whether seed was crawled, show if the seed redirected to a different URL and if robots.txt file blocked the crawler Seed Source: shows how many documents and which hosts were archived per seed MIME type: lists all the different types of files archived

29 Reports can be opened in Excel Above is a portion of the seed source report

30 Offsite Hosts in Reports Embedded content on a website can have a different originating host than the main site address – can contain content from in the form of a logo or any other embedded element on an pagewww.archive.org –When seed is crawled, rlg.org will show up in the host reports even though it was not a seedwww.archive.org

31 Search

32 Search results include hits from any seed metadata entered.

33

34 Wayback Machine Displays page as it was on the date of capture The date of capture is displayed in the archival URL, breaks down as yyyymmddhhmmss it.org/270/ / was captured on August 1, 2006 at 21:16:37 GMT

35 Archive-It Help Online help wiki (link within application) Partner Specialist for support (including technical) List serv: Report all technical bugs, issues and questions to