BnF experiences with harvesting content beyond paywalls BnF - DLWEB - Umbra & Heritrix 3 BnF experiences with harvesting content beyond paywalls Géraldine Camile geraldine.camile@bnf.fr NAS Workshop, Vienna, 27 April 2017

The subscription news sites

Harvest of subscription news sites Since 2012, BnF crawls subscription news sites

Focus on the PDF versions Focus on regional newspapers to ensure collection continuity as microfilming projects for local editions are stopped

The harvesting workflow

The main difficulties News websites architecture may change very quickly Requires high reactivity and dedicated time of technical staff Difficulty to recover non-harvested collections Press collections disappear very rapidly from the publisher’s website No relation between date of the archives and date of the editions Some websites are technically NOT possible to harvest with crawling robots And PDF format is disappearing from the websites

FTP harvest PDF format is still used by the publishers to print the editions Publishers make their files available for BnF on a FTP server The BnF crawls the files with Heritrix daily Publishers choose the rythm of file clean-up

Benefits/inconveniences The benefits: Harvest of all the titles of the groups (against title by title) Decrease of instruction time for the BnF’s engineers Stability of the system It’s easier for the publishers to offer this service than to evolve their websites The inconveniences: Specific indexation and access development to view the files in OpenWayback The filenames are sometimes not meaningful for the title or the editions ftp://transfert-presse-sp.e-i.net/170424/LDL_74BDE_20170424.PDF) The link between the context and the publication is not straight

Merging of the harvest templates (1/2) From 9 harvest templates to 2: By http or html authentication By FTP

Merging of the harvest templates From 9 harvest templates to 2: By http or html authentication By FTP it’s possible to create “sheets”

Towards a deposit? BnF is currently evaluating the possibility to put in place a deposit workflow

ebooks

The BnF approach : deposit against harvest Harvest of parts of websites accessible upon payment In use for online newspapers Entry, preservation and access workflows already in place Why choosing FTP direct deposit? Maintain close relationships with publishers In most cases, ebooks aren’t directly downloadable from the website Allows cataloguing each document Cooperation with publishers and distributors They provide metadata files in ONIX format Which we can reuse! Easy-to-manage ebooks No DRM, no closed-source formats PDF and EPUB (versions 2 and 3)

Bibliographical products Reception FTP platform Extranet for publishers Sorting and first checks Nouveautés Editeurs Metadata Waiting area Preparation for preservation Diffusion Creation of the document in the BnF information system Catalogue M SPAR : negotiated legal deposit track Automatic but rich bibliographic record Gallica intra Muros Livnum_A Channel Livnum_B Channel ADCat-02 Version anglaise EPUB and PDF readers, secured environment Virtual trolley Diffusion zone Complete bibliographic record Preservation Bibliographical products Description