Technology for Long-Term Digital Preservation Long Term Archive Technology and Media ESA/ESRIN, Frascati - Rome: 22 September, 2017 ESA UNCLASSIFIED – For Official Use

ESA Data Preservation System Archival, preservation and bulk extraction of non Copernicus data Archiving function decoupled from data production/dissemination Data extracted as it was ingested – no transformation / conversion

Cold Backup Archive (CBA) ESA UNCLASSIFIED – For Official Use

Cold Backup Archive (CBA) 6 Petabytes 88,000,000 files Second copy of the Preservation System Data Layer Unique data & Unconsolidated data PAC and Station Phase-out project data Live Mission support Swarm (started in mid 2014) Odin / SCISAT / Proba-1 Cryosat-2 (started in March 2015) SMOS (started in May 2016) ESA Science data Disaster Recovery Two copies of the data held at ESRIN ESA EO Internal LAN, not accessible from any network - 10 GB Network Environment Main Tape Library capable of storing 24 Petabytes EO-SIP archiver Native Format archiver Implemented ad no cost in May 2014 with Hardware already available at ESRIN ESA UNCLASSIFIED – For Official Use

CBA Main Tape Library STK SL8500 3000 licensed slots 8 T10000D Drives 10 T10000B Drives 10 9940 Drives 8 Robotic Arms 16 GB Fibre Connection 90 TB Disk based cache ORACLE HSM driven Automated Cartridge System Library High Availability QFS Client to mount the HSM virtual file-system from any client ESA UNCLASSIFIED – For Official Use

CBA Disaster Recovery Tape Library STK SL3000 Three Cartridge Expansion Module 1500 licensed slots 8 T10000B Drives 16 GB Fibre Connection 20 TB Disk based cache ORACLE HSM driven LT0-8 drives to be introduced when available on the market ESA UNCLASSIFIED – For Official Use

CBA Data flow (1) HSM Physical File System Copy Stations NAS Devices FTP Archiving Tool QFS Client Virtual File System Tapes EO-SIP packages Non EO-SIP Data Documentation Auxiliary Data Data written on the Physical File System is automatically copied onto tapes. Data retrieved from the Virtual File System is read from the tapes and written to the Physical File System. A dedicated server handles Input / Output request of data, mounting the Virtual File System trough a client. The Virtual File System is mounted via the 10GB network but routes read and write operations trough 16GB Fibre Optic. ESA UNCLASSIFIED – For Official Use

ngA CBA Data flow (2) EO-SIP Documentation CBA Data Auxiliary Data Native Format Metadata EO-SIP products ingested by the Cold Backup Next Generation Archive (ngA) tool. Bulk Ingestion from Local or remote locations Automated Ingestion from inbox basked EO-SIP Validation Metadata extraction Reporting productName orbitNumber Parameters platformSerialIdentifier accessURL platformShortName archivePath processingMode beginTime processingDate downlinkedTo processorVersion endTime productId imageQualityDegradation productType imageQualityDegradationUnit productVersion instrumentShortName wrsLatitudeGrid orbitDirection wrsLongitudeGrid ESA UNCLASSIFIED – For Official Use

Preservation Element Front-End CBA Data flow (1) Master Archive (DAS) Live missions Ingestion report Cryosat2 Preservation Element Front-End SMOS Cold Backup Archive …. Ingestion report DA TOGLIERE Cryosat-2 (started in March 2015) SMOS (started in May 2016) ESA UNCLASSIFIED – For Official Use

CBA Monitoring Tool Monitoring tool for Tapes and Drive health Connected to Drives and Automated Cartridge System Recommendation of tape content migration Automatic health check on tapes Warnings on read/write errors ESA UNCLASSIFIED – For Official Use

Start of Master Archive Feeding CBA initial backlog Start of Master Archive Feeding ESA UNCLASSIFIED – For Official Use

CBA Storage evolutions 2014 2015 2016 2017 2018 T10KB -> T10KD -> 8 x T10KD ACSLS HA ? Staging/retrieving data from tapes affected archiving performances considerably; Virtual File-system dump took days to complete; Aug 2017 HW Upgrade: T10000 drive family end-of life (tbc) ? Study on possible alternatives are on-going for Main Library Second Library to be equipped with LTO-8 Migration from 8Gbp/s fibre switched to 16 Gbp/s Upgrade from SAM-FS 5.4 to ORACLE HSM Installation of SSD based storage tier Creation of a dedicated LUN for the inodes onto the SSD storage ESA UNCLASSIFIED – For Official Use

Questions ? ESA UNCLASSIFIED – For Official Use