1. 22 September 2017, ESA/ESRIN - Frascati
Technology for Long-Term Digital Preservation - First workshop Gilbert Barrot, ACRI-ST Gaston Briot, adwaïsEO
22 September 2017, ESA/ESRIN - Frascati

2. One significant example: EODAS ESA project
ACRI-ST and its subsidiary adwaisEO are involved in several projects involving data storage
One significant example: EODAS ESA project

3. EO Data Archiving Service (EODAS)
Implements the ESA Master Archive through a dedicated service:
Contract Kicked-off in August 2016
Two copies of the data held > 200 Km apart
Connection to the ESA EO WAN
Checks of the data at all stages from ingestion to delivery
Scalability required for growth of contents
Management of the data information (MetaData) in fully redundant Databases

4. EODAS Consortium
The industrial consortium is made of ACRI-ST (FR), adwäisEO (LU) and KSAT (NO), with the following distribution of roles:
ACRI-ST: Data Archiving (and delivery) Service provider – Prime contractor
adwäisEO: data archiving (and delivery) operation provider with its leading edge IT and connected secured infrastructure in Luxembourg
KSAT: data knowledge provider and overall ingestion validation

5. EODAS drivers Strong drivers from H/Div:
Requirements limited to “Archival and bulk extraction” of non Copernicus data
No operational use of the data
Data extracted as it was ingested – no transformation / conversion
No tight integration to the PDGS of the missions; not for end-user dissemination
Added requirements for Data Ingestion / Evolution / Alignment / Management
Added requirements for Service Management and Reporting
Out of scope: transcription from historical media, consolidation, transformation, conversion

7. Tape transfer to Backup Archive Data retrieval
8 LTO7 drives
500 TB cache
11 PB capacity (LTO7)
Data ingestion
Data evolution
Data alignment
DMS
Tape transfer to Backup Archive
Data retrieval

8. NAS HDD tapes ftp access
listings
Data ingestion
Data evolution
Data alignment
DMS
EODAS central archive
Tiering solution (Quantum)
Disk cache (XCellis)
Tape library (i ANTF)
Weekly tapes transfer or real-time FTP transfer (LM)
Data retrieval (Quantum)
Tiering solution (Quantum)
Disk cache (M440)
Tape library (i ANTF)
Tape storage (vault)
Archive backup

9. Product status evolution during the ingestion process
Data ingestion / archiving process
Ingestion of historical + Live missions (Cryosat-2 and SMOS already operational) data
Products/files from source media to scratch storage
Packaging (AIP)
Copy to archive (disk  tape)
Transfer to backup center (tape or FTP)
Copy in backup archive +
DMS update
Dire que l’EODAS est une archive bande
Product status evolution during the ingestion process
aaaaa

10. Data packaging process (AIP)
One single internal package format to simplify the internal process
Distributed data = same format as original

11. Data integrity
Data integrity verified after each process: copy/packaging/transfer
zip integrity or MD5 checksum

12. Ingestion chain Ingestion chain is based on ACRI-ST DPMC system
Data and Processing Management Core system
 DMS + PMS + orchestrator
Most of EODAS operations are performed in parallel on several processing nodes (up to 136 simultaneous processing threads)

13. DELL (servers) + Quantum (storage/library) + Cisco (network)
Infrastructure
DELL (servers) + Quantum (storage/library) + Cisco (network)

14. Quantum tapes tower + Synology NAS (from ESA)
Infrastructure
Quantum tapes tower + Synology NAS (from ESA)

16. On-line access archives
Disk or tapes?
Volume
Full-tape
Hybrid (e.g. HSM)
On-line access archives
Cold archives
EODAS
Full-disk
Performance

17. Disk vs tapes: the criteria
Disk or tapes?
Disk vs tapes: the criteria
Bulk or random-access ?
Volume ?
Short or long-term data archive ?
Cost ?
Versioning ?

18. Disk or tapes?

19. Many thanks for listening.
Q&A ?