WebArchive – Archive of the Czech Web Mgr. Jan HUTAŘ

Why we started with WebArchiv?  amount of documents published on the Internet is growing dramatically – average lifespan is 40 days --> if the documents are not archived a part of the national cultural heritage would disappear forever  need to save and keep accessible the documents on the CZ web  about 90% documents on the web exist only in electronic form  trend around the world (Australia, Sweden, Internet Archive … etc.)  NK ČR is suppose to do it – it is deposit library  main mission of the NK is to collect, catalog, permanently preserve documents published in the territory and make them available to the general public

The beginning  launched in 2000 – till 2002 – grant project R&D „Registration, preservation and access of national electronic resources in the Internet“ by Ministry of Culture  cooperation with Moravian Library Brno and Institute of Computer Science at the Masaryk University Brno  they are our „IT department“ ;-)  only grants money  we are still going on!

Main Aims  to implement best solution in the field of archiving of the national web, i.e. bohemical online-born documents  prepare tools, methods and conditions for collecting, archiving and preserving web resources  to provide long-term access to them  large-scale automated harvesting of the entire national web and selective archiving are being carried out, including thematic „event-based“ collections  to solve current legal issues (the legal deposit legislation, CA) Legal Deposit Act doesn‘t cover online-born documents and according to the Copyright Act, it is not possible to make archived data available to public.  set selection criteria for selective approach / harvest  to establish conditions for cooperation between libraries and publishers of electronic documents

Workflows Prague:  Resource selection  Cataloguing for the National Bibliography (MARC21)  Providing Dublin Core metadata for interested publishers  Making archive access agreements with publishers Brno:  Running WebArchiv hardware  Software localization, maintenance and development  Pre-harvesting resource analysis  Harvesting, indexing, access Results so far:  4 harvesting rounds of.cz domain (2001, 2002, 2004, 2006)  5 event-oriented harvests  several times per year – harvests of sites under agreements  5.4 TB archive with 136 million files

Selection Criteria  The amount of documents on the Internet is quite big – for selective approach we need to find the ones with „research value“ For acquisition (harvesting) 2 approaches: 1. selective approach - only selected documents are harvested and archived – according to selec. criteria 2. complete harvest – of the entire national domain for example.cz. We need only to set harvester…  approaches are different in different countries  trend is to do both (Australia, Denmark)

Criteria –selective approach  to set selection criteria was very difficult Web Cultural Heritage EU Culture 2000 program)  we coordinated "Web Cultural Heritage„ project (in the frame of EU Culture 2000 program)  Content  Resource type  Original form  Access  Format  Domain  National aspect

Criteria –selective approach 1. Contents Web resources of art or research value, news stories and feature articles and resources as outputs of government and other offices. Promotion material of an individual or a corporation is omitted. 2. Resource Type Serials, monographs, conference proceedings, research and other reports, academic works etc. 3. Original form Only resources originally published in the web – it means they have no traditional/printed copy 4. Access Only freely accessed resources are collected

Criteria –selective approach 5. Format Resources available in formats that are interpreted by common web browsers without necessity of installing plug-ins are collected. 6. Domain Resources accessible at servers under the top level domain.cz and at servers under the other domains … 7. National aspect Resources according to „authors nationality“, „national language“, „country or nation as a subject“

What we have done…  continuous testing of:  SW tools  applications for harvesting, archiving, indexing and accessing of the web pages  only open source SW  effort / push to change legislation  international cooperation (activities in R&D within IIPC – even before we become a member)IIPC  we have opened part of our archive for public (since autumn 2005)archive  the whole archive archive should be open right now (only local access)

Harvest of the.cz domain  2001 first try of the whole domain harvest of the.cz domain, 1 PC + tape robot, cz2001 includes over 3 mil. of unique URLs (107 GB) – not completed  2002 harvest interrupted - lack of space on data storage and floods. cz2002 includes 315,5 GB, from URLs harvested over 10 mil. docs  in 2003 no harvest  2004 March- October, from URLs harvested 32,5 million files = 1,2 TB  september nd harvest of.cz by Heritrix. Stoped – no data storage space. Limits: max docs/server, max. file size 100 MB  all harvest executed by the NEDLIB harvester, deep links HERITRIX  from 2004 new harvester HERITRIX

Registrované domény v.cz

Harvests of.cz domain in quick view yearTotal downloaded documents Size – non comprimated [GB] Days of running Amount of second level domains % from all reg. domains 20013,015, ,32238% ,249, ,02269% ,141,5751, ,37875% 20059,336, ,7952% ,378,0193, ,88074%

Present state of the project  4-6 times/year is   4-6 times/year is harvested collection of selected resources (agreement with NK), about 350 servers. increase is around 10GB of data for each harvest  it is still rising  harvest of „small“ amounts of data is successful   analysis of the domain.cz was done  servers „suspicious“ from unrelevancy were rejected (mail, mysql apod.) as well as duplicates – number of URLs decreased from 540 to 378 thousands BUT BUT …   from 2004 we were not able to keep running the harvest of the whole.cz domain. – problem of Heritrix with memory using  new release solved this issue

Present state of the project  main standards are used (MARC21, DC, ISSN and URN)  selected docs are catalogued in an ALEPH library system which supports Z39.50 and OAI-PMH protocols  selected resources (with agreements) at least 4 times a year WebArchiv  at present we have in WebArchiv saved cca 5,5 TB of data (uncompressed) ≈ 158 milions of documents   in the end of 2007  all data will be moved on the new data repository  in 2008 archive of the project should become a part of prepared project of „National Digital Library“ at National Library (together with Kramerius and Manuscriptorium)

Software changes  2004 development and support of NEDLIB harvester was canceled – we replaced it by Heritrix  consecutive change over to SW developed by IIPC (International Internet Preservation Consortium) ARC  archival file format nedlib replaced by ARC format (used by Heritrix)  warc format in near future – then wayback

Harvester Heritrix – advantages system modularity, extensibility, continual development (v.1.8), very good and fast support from Internet Archive developers open source codes and modularity allow cooperation of third party on its development – good for us ;-)  2 parts – framework and add on modules  Framework – basic control over harvests, user interface, process managemenst, harvest settings  modules – used for specific harvest implementation, set up each harvest step by step

Harvester Heritrix - problems  not possible to leave the whole process of harvesting without the control of experts  trap detection  extraction of links from websites (Java)  memory problems (whole domain harvest) - solved  incremental harvest and changes detection

SW for access  from IIPC (IA) NutchWAX Nutch  fulltext document indexing - NutchWAX, extension/superstructure over search engine Nutch  WERA  WERA (successor of NWA tools) – user interface for accessing documents on the web – it can deal with Czech diacritics (accents etc. – display it, search by it, sort)user interface for accessing documents on the web  ARCWayback  ARCWayback make index over whole archive, it allows access into archive by URL and time  Wayback  Wayback for only restricted on-site access from within the library is possible to all files in the archive

Nutch a NutchWAX Nutch search engine, by IA  open source search engine, by IA  comes from Apache Lucene architecture Nutch is able to:  download and work up millions of sites in a month, manage and control their index and search in this index 1000times/second NutchWAX  superstructure over SE Nutch made for indexing of documents archived by Heritrix  set of indexing and query plug-in, which add some needed metadata to index

WERA - WEb aRchive Access  cooperation between IIPC, Internet Archive and NWA  use some parts from NWA  very easy navigation, nice user interface (time line with documents version in time)  search hits in URL form are displayed very digestedly, each hit has link to the timeline to get differ. version of the same URL  possibility to search by URL address (like Wayback M.)  archived docs and WERA are linked by NutchWAX index

How does it work actually?  harvest of docs – by the Heritrix crawler, docs are saved to data storage in ARC format  to make archived docs accessible we have to make index + interface, which display seach hits 1. making of the fulltext index over the collection of selected resources v- for searching by the words- NutchWAX 2. making of global index to provide access of the whole archive - ARCWayback  displaying of docs from archive - WERA and Wayback

WebArchiv – Infrastructure A1 new crawl; A2 end crawl -> index; A3 update fulltext; A4 update host list

WERA - ukázka

Our future  main aim  main aim – finish 2006 harvest, >> keep in processing the whole.cz domain harvest every year  go on with selective collection and increase the amount of resources in it  provide legal access to the whole archive – localy- according to the new CA (searching by URL and by the time of harvest  implemantation of incremental harvest (changes identification in repeatedly harvested docs)  Harvesting of bohemical resourcs outside the.cz domain - some language recognition tool  Adaptive incremental harvesting

Our future  Identification of duplicate (or rather very similar) documents  Incremental indexing - adding of new docs into already made index, not to make new one everytime  Fulltext indexing of the whole archive  Selective harvesting on demand  Permanent linking into the archive  Access limitations set by the new copyright law  OAI-PMH implementation on top of the registration database  Building METS structures on top of the archive  integration of the archive into the proposed NDL 2007/08

People Librarians, project management:  National Library: 3.5 FTE IT management  Moravian Library – 1 part- time IT  Masaryk University – 6 part- time

Useful links – in english;-)  WebArchiv homepage  Petr Žabička Digital Cultural Heritage and the Cooperation of National Memory Institutes Archiving the Czech Web: Issues and Challenges  this presentation  Petr ŽABIČKA: WebArchiv, Czech Web Archive