1. Technology for Long-Term Digital Preservation
Long Term Archive Technology and Media
ESA/ESRIN, Frascati - Rome: 22 September, 2017
ESA UNCLASSIFIED – For Official Use

2. ESA Data Preservation System
Archival, preservation and bulk extraction of non Copernicus data
Archiving function decoupled from data production/dissemination
Data extracted as it was ingested – no transformation / conversion

3. Cold Backup Archive (CBA)
ESA UNCLASSIFIED – For Official Use

4. Cold Backup Archive (CBA)
6 Petabytes 88,000,000 files
Second copy of the Preservation System Data Layer
Unique data & Unconsolidated data
PAC and Station Phase-out project data
Live Mission support
Swarm (started in mid 2014)
Odin / SCISAT / Proba-1
Cryosat-2 (started in March 2015)
SMOS (started in May 2016)
ESA Science data Disaster Recovery
Two copies of the data held at ESRIN
ESA EO Internal LAN, not accessible from any network - 10 GB Network Environment
Main Tape Library capable of storing 24 Petabytes
EO-SIP archiver
Native Format archiver
Implemented ad no cost in May 2014 with Hardware already available at ESRIN
ESA UNCLASSIFIED – For Official Use

5. CBA Main Tape Library STK SL8500 3000 licensed slots 8 T10000D Drives
10 T10000B Drives
Drives
8 Robotic Arms
16 GB Fibre Connection
90 TB Disk based cache
ORACLE HSM driven
Automated Cartridge System Library High Availability
QFS Client to mount the HSM virtual file-system from any client
ESA UNCLASSIFIED – For Official Use

6. CBA Disaster Recovery Tape Library
STK SL3000
Three Cartridge Expansion Module
1500 licensed slots
8 T10000B Drives
16 GB Fibre Connection
20 TB Disk based cache
ORACLE HSM driven
LT0-8 drives to be introduced when available on the market
ESA UNCLASSIFIED – For Official Use

7. CBA Data flow (1) HSM Physical File System Copy Stations NAS Devices
FTP
Archiving Tool
QFS Client
Virtual File System
Tapes
EO-SIP packages
Non EO-SIP Data
Documentation
Auxiliary Data
Data written on the Physical File System is automatically copied onto tapes. Data retrieved from the Virtual File System is read from the tapes and written to the Physical File System.
A dedicated server handles Input / Output request of data, mounting the Virtual File System trough a client. The Virtual File System is mounted via the 10GB network but routes read and write operations trough 16GB Fibre Optic.
ESA UNCLASSIFIED – For Official Use

8. ngA CBA Data flow (2) EO-SIP Documentation CBA Data Auxiliary Data
Native Format
Metadata
EO-SIP products ingested by the Cold Backup Next Generation Archive (ngA) tool.
Bulk Ingestion from Local or remote locations
Automated Ingestion from inbox basked
EO-SIP Validation
Metadata extraction
Reporting
productName
orbitNumber
Parameters
platformSerialIdentifier
accessURL
platformShortName
archivePath
processingMode
beginTime
processingDate
downlinkedTo
processorVersion
endTime
productId
imageQualityDegradation
productType
imageQualityDegradationUnit
productVersion
instrumentShortName
wrsLatitudeGrid
orbitDirection
wrsLongitudeGrid
ESA UNCLASSIFIED – For Official Use

9. Preservation Element Front-End
CBA Data flow (1)
Master Archive (DAS)
Live missions
Ingestion report
Cryosat2
Preservation Element Front-End
SMOS
Cold Backup Archive ….
Ingestion report
DA TOGLIERE
Cryosat-2 (started in March 2015)
SMOS (started in May 2016)
ESA UNCLASSIFIED – For Official Use

10. CBA Monitoring Tool Monitoring tool for Tapes and Drive health
Connected to Drives and Automated Cartridge System
Recommendation of tape content migration
Automatic health check on tapes
Warnings on read/write errors
ESA UNCLASSIFIED – For Official Use

11. Start of Master Archive Feeding
CBA initial backlog
Start of Master Archive Feeding
ESA UNCLASSIFIED – For Official Use

12. CBA Storage evolutions
2014
2015
2016
2017
2018
T10KB -> T10KD
-> 8 x T10KD
ACSLS HA ?
Staging/retrieving data from tapes affected archiving performances considerably;
Virtual File-system dump took days to complete;
Aug 2017 HW Upgrade:
T10000 drive family end-of life (tbc) ?
Study on possible alternatives are on-going for Main Library
Second Library to be equipped with LTO-8
Migration from 8Gbp/s fibre switched to 16 Gbp/s
Upgrade from SAM-FS 5.4 to ORACLE HSM
Installation of SSD based storage tier
Creation of a dedicated LUN for the inodes onto the SSD storage
ESA UNCLASSIFIED – For Official Use

13. Questions ?
ESA UNCLASSIFIED – For Official Use