Harvesting digital newspapers at the Bibliothèque nationale de France Géraldine Camile Bibliothèque nationale de France Tallinn, 2015-01-30 20 mn de présentation Définitions: Metrics: we talk about concepts and definitions Statistics: we talk about numbers and results

Summary Context and objectives of the “subscription-based press project” Harvesting news websites with robots Results and lessons learnt The future of the project – and its alternatives I’ll present you the approach adopted at BnF to collect online newspaper thanks to web harvesting technologies This project is called the subscription-based press project So first I’ll present the context and objectives of the projects Then there will be a more technical description of the workflow I’ll finish by the results, the lessons learnt, and the next steps of the project 2

Context and objectives of the “subscription-based press project”

Collecting digital news at the BnF Harvesting of news websites since 2010 Use of crawlers 100 news websites harvested every day Only freely accessible content Using robots to collect digital equivalents of newspapers “Subscription-based” press project Obtain passwords from publishers and crawl protected content Focus on the PDF versions to ensure collection continuity As microfilming budgets for local editions of regional newspapers are decreasing What is the context? At the BnF, we harvest news websites since 2010 We use harvesting robots, called crawlers, to collect 100 news websites every day Up to last year, it was only on the freely accessible part of the website However, many parts of the website, often the most interesting one, are only accessible upon payment. As the law on internet legal deposit allows the BnF to ask for password, We decide to ask press publishers for passwords to collect the protected content: This is the idea behind the “subscription-based” press project And we also decided to focus on the PDF versions of local edition of regional newspapers. In fact, their paper version is not collected by BnF anymore, as they were microfilmed. And microfilming budget were decreasing, so we needed a replacement solution. 4

The subscription-based press project Various actors within the Library Law, Economy and Politics department Legal deposit department: printed periodicals service Legal deposit department: digital legal deposit service IT department Different skills and approaches for printed and digital periodicals Calendar A one-year experiment Started end 2012; assessment end 2013 Now in production mode This project grouped together various actors within the library … This combination was a way to associate skills and approaches towards printed and digital periodicals It was a one-year experiment that started… 5

Harvesting news websites with robots Source: http://mimas.ac.uk/data-harvesting-brings-cost-savings-with-jusp/

The harvesting workflow Contact with publisher Technical instruction Selection Curators Engineers Web harvest Description on access UI So how does it work Selection of news title Then contact with publisher, which may take a lot of time … There is a sampling for quality assurance Who does what; what are the professional profiles involved in this activity? Curators Quality assurance Engineers Cataloguing Preservation Cataloguers Library assistants

Cataloguing… Type: digital document Format Link to the archives Local editions We designed a system to catalogue web archives within the General Catalogue Link with the printed edition record August 20th 2014 Harvesting digital newspapers at the BnF – Clément Oury – IFLA WLIC conference

And access in the archives… The title is accessible through web archives If you clic on a specific date You select then your local edition And here you have the document The result on the collection of the newspaper The choice of the local edition An example of a local edition Harvesting digital newspapers at the BnF – Clément Oury – IFLA WLIC conference August 20th 2014

A guided tour of the news collection Harvesting digital newspapers at the BnF – Clément Oury – IFLA WLIC conference August 20th 2014

Long term preservation in SPAR, BnF’s digital repository Harvesting press websites at the BnF – Clément Oury – IFLA WLIC conference August 20th 2014

Results and lessons learnt

The collections 22 titles 192 local editions Start of harvest Ouest-France 53 July 19, 2012 Le Républicain lorrain 8 December 12, 2012 Le Progrès 18 April 16, 2013 Midi libre 14 May 2, 2013 L’Indépendant 3 Centre Presse 1 La Tribune May 22, 2013 Mediapart July 16, 2013 La Montagne October 10, 2013 Le Populaire du Centre La République du Centre 2 Le Berry Républicain L’Écho Républicain Le Journal du Centre Le Dauphiné libéré 20 April 7, 2014 Les Dernières Nouvelles d'Alsace L'Est Républicain 10 L'Alsace Le Journal de Saône-et-Loire 7 Le Bien Public 4 Vosges Matin The collections Twenty-two titles, representing one hundred ninety-two local editions August 20th 2014 Harvesting digital newspapers at the BnF – Clément Oury – IFLA WLIC conference

Map of the daily regional newspapers Harvested titles Vosges Matin La Liberté de l’Est A good coverage of French territory When there’s a can, there a collected title Harvesting digital newspapers at the BnF – Clément Oury – IFLA WLIC conference August 20th 2014 (n° 1, oct./nov. 2012, p. 60-61)

Main achievements The collections! Technical experimentations of harvest of protected content Creation of links between the General Catalogue and web archives Raising awareness among wider library staff about collecting digital publications Even library assistants are now managing digital documents Apart of course for the collections! Technical improvements: now we know how to collect protected content for websites, we could use this experiment for others contents Also creation of links It was also a way to raise awareness about collecting digital publications Now even library assistants are managing digital documents

The dark side of the crawl News websites’ architecture may change very quickly Requires high reactivity and dedicated time of technical staff Difficulty to recover non-harvested collections Press collections disappear very rapidly from the publisher’s website Some websites are technically NOT possible to harvest with crawling robots But there are bad news Lire Source: http://www.motifake.com/bot-fail-bad-bots-robots-banned-blocked-demotivational-posters-65020.html

The future of the project – and its alternatives

The next steps of the project Extend the harvest to new titles Improve access to collections A dedicated interface? Full-text index of the press corpus? Promote the service towards: Librarians at reference desks Researchers and other users Open remote access From the researchers desktops From regional libraries entitled to receive access to web legal deposit collections Strasbourg and Nancy have allready an access to BnF’ archives.

Success and alternatives Identify alternative ways of collection Deposit from publishers through FTP? Deposit from press aggregators? Build upon the experience of the ebook deposit workflow A successful project… which needs to be complemented For the websites for which web harvest is not feasible, we could set up a legal deposit worflow such as the one of ebooks, With publishers of press agregators So in our opinion…