Airbus Archive Technologies Mark Curtis & Phil Capp 22nd September 2017

Introduction Mark Curtis – LTA Architect, Service Manager Phil Capp – Operations Manager History Archive Base Requirements Current Archiving Projects Supported Future 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

History ESA Projects Providing Archive capabilities since 1991 ERS-1, ERS-2 and ENVISAT Missions Originally all on physical media and stored on shelf managed through manual database HDDT, Exabyte, DLT, DAT, Optical Disk Transcribed, using provided infrastructure and tools into Managed Archive System (Robotics and Database) Size of archive at end of project ~1 Pbyte (~20 years of data) Transcribed out of Managed Archive System to transport medium for repatriation Sentinel-1 and Sentinel-2 LTA services (ongoing) SWARM (ongoing) Airbus Projects GEOSTORE ( contains numerous clients ) 16 November, 2018 LTDP Workshop – ESRIN 22/09/17

Archive Base Requirements There are a number of Base Requirements which are seen, by Airbus DS, as crucial considerations when implementing Archive services: Heterogeneous Filesystem Support (GFS) including file level security Fast Access (In-Band Multiple Fibre Channel Connections) Near line and Offline Storage (Robotics and Shelf) Multiple Targets/Media Types Multiple Copies of Files Multiple Versions of Files Data Fault Tolerance – what was copied to tape was what was received on disk Automatic Media Fault Detection and Recovery Data Compression at Hardware Level High Throughput High Availability COTS – Support into the Future 16 November, 2018 LTDP Workshop – ESRIN 22/09/17

Archive Base Requirements There are a number of Base Requirements which are seen, by Airbus DS, as crucial considerations when implementing Archive services: Scalability High Density (Small Footprint) Low Power Consumption and Cooling Open Standards Industry Standard Media Formats Support for Encryption ( embedded ) Longevity of Media (> 5 Years), Obsolescence versus Storage life Protected Archive ( air gapped ) 16 November, 2018 LTDP Workshop – ESRIN 22/09/17

Airbus Adopted Archive Architecture Hierarchical Tiered storage required (many writes few reads) Quantum StorNext Quantum Tape Libraries (i10K, i2K, i6K) Brocade Fabric Controllers Disk Cache (Hitachi Data Systems) – AMS1000, AMS2500, HUS150, G600 Block Storage LTO (LTO-4, LTO-5, LTO-6 LTO-7) 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

Sentinel LTA Service Requirements Sentinel LTA Service Requirements: Daily I/O Capacity: 30 TB (Terabytes) Maximum Single file sizes: 200 GB (Gigabytes) Maximum number of files per day: 10000 Minimum sustained LTA interface transfer rate: 220 MegaBytes/Sec Architecture implementation surpasses service requirements 16 November, 2018 LTDP Workshop – ESRIN 22/09//2017

LTO Roadmap 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

Preferred Archive Architecture 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

LTA Archive Service 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

Current Projects Current Archiving Projects Sentinel-1 LTA Scalar i6K, StorNext, LTO-6 & LTO-7, 0.5 Pbyte Cache (HDS G600) Currently 5 Pbyte Growth ~10TB/day Sentinel-2 LTA Scalar i6K, StorNext, LTO-6, 0.25 Pbyte Cache (HDS HUS150) Currently 2 Pbyte Growth ~4TB/day Geostore (Airbus Services) Scalar i6K, StorNext, LTO-4 & LTO-5, 0.2 Pbyte Cache (HDS HUS150) Currently 0.4 Pbyte Slow growth rate 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

Current Projects Geostore (Airbus Services) Scalar i6K, StorNext, LTO-4, 0.25 Pbyte Cache (HDS HUS150) Currently 0.2 Pbyte No significant growth Multiple clients and business areas Financial Institutions, Insurance Companies, Government Agencies, Local Authorities Data Types: Aerial Photography DTM DSM CIR LIDAR National Vector & Raster Imagery > 100,000,000 files File sizes 1K – 100GB 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

More Faster Longer Updates and Future Updates Full cache replacement for Sentinel-1 LTA Replaced HDS AMS2500 (0.25PB) with HDS G600 (0.5PB) 5 PB Filesystem migrated to new storage ~1 Day interruption to service Future Larger Volumes Higher Throughput Longer Storage More Faster Longer 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

Updates & Future Future (continued)…………… Technologies being looked at: Public Cloud Private Cloud Hybrid Cloud Multiple Sites across Private Networks Object storage to replace traditional block storage The proverbial flash versus disk versus cloud versus tape argument 16 November, 2018 LTDP Workshop – ESRIN 22/09/2017

Thank you