1. Large-scale Archival Storage - a brief overview for the HEP use case -
GDB, 13/9/2017
Germán Cancio – CERN IT/ST

2. Agenda Status of tape market and technology Alternatives to tape Disk
Optical
Holographic

3. Tape Market dominated by LTO consortium (~95%)
IBM, HP, Quantum (drives) + Fujifilm, Sony (media)
Oracle resells LTO drives from IBM
Enterprise tape (IBM+Oracle) ~4%
IBM: Latest: 15TB, 350MB/s – introduced May 2017
Oracle: Latest 8TB, 250MB/s – introduced Sept 2013
Tape drive head technology: From GMR to TMR
GMR has reached its density limits (HDD’s: stopped in 2004)
TMR requires substantial R&D and manufacturing retooling
TS1155 uses TMR, LTO-8 will use it
Large-scale libraries (>=10K slots)
Oracle, IBM, Spectra Logic, now also Quantum

4. DPHEP Collaboration Workshop
LTO and IBM enterprise tape roadmap (source: IBM)
Expected EOY 2017
Released Q2 2017
8/6/2015
DPHEP Collaboration Workshop

5. Oracle enterprise tape roadmap

6. Tape drive head manufacturing (Source: Spectra)

7. Tape drive head manufacturing (Source: Spectra)

8. 30% CAGR probably realistic
08/2017: Sony/IBM demo
(using CoPtCr) 201Gb/in2
~330TB tape
04/2015: Fuji/IBM demo
(using BaFe) 123Gb/in2
~220TB tape
30% CAGR probably realistic
IBM TS Gb/in2

9. Tape Market evolution LTO media shipments decreasing since ~2007
Consolidation, competition of disk and cloud solutions
LTO media units shipped, 200—2016
(source: LTO consortium)
~ 20M units per year (~40 EB)
Media price ~10-15CHF/TB(*), but decay has slowed down (-20%/yr over last 4 years)
Two remaining media manufacturers (TDK exitus 2014)
(*)Media is ~50% of tape TCO (add drive and library HW, maintenance)
SPoF as IBM only remaining (major) manufacturer + R&D of tape drive technology
Will this market sustain to drive (enterprise and LTO) tape research (new heads, new media) and production?

10. Disk (Spinning) disk market: WD (41%), Seagate (37%), Toshiba (22%)
~600EB/year, decreasing since 2010
”Nearline” (high capacity, high quality) drives used in HEP: ~10% of market sales
Increased competition from cloud & SSD for notebooks and enterprise disks
Source: Statista / B.P.S.
Source: Wikipedia

11. Disk Technology 30% CAGR - realistic? Lower reliability than tape
Shingled recording disks (SMR) ~2013
Helium filled disks (more platters) ~2013
HAMR disks not before 2018(?)
Complex technology, laser+new media, reliability+cost are issues
Capacity evolution
14-16TB in 3-12 months
20TB in 2020
100TB by ~2025 with HAMR/HDMR
Pricing for ”nearline” disks decreasing ~14%/year, currently ~35-40CHF/TB
Very shaky price evolution
30% CAGR - realistic?
Lower reliability than tape
UBER cf / for tape
Overhead required (Replica/RAID/EC)
SSD’s vs HDD’s?
Shipped capacity in 2016: ~45EB (roughly 7.5% of shipped HDD capacity – expected to grow to ~20% around 2021)
Large investments required for SSD manufacturing ( B$)
SSD/TB price for capacity in foreseeable future still expected O(10)x of disk/TB price

12. Disk Servers for Archival?
Current CERN EOS disk servers: one ”CPU” node + 2x24 enterprise-class capacity disks. JBOD with 2 replica.
Disk cost is 75%
TCO is ~3x of tape CHF/TB
Looking into optimising CHF/TB to close gap with tape. Ongoing investigations:
Using desktop disks à la BackBlaze (30%-40% cheaper)
No warranty / certification for our use case
Measure reliability of different models/vendors (integrity, failure rates) using SMART and EOS monitoring
Compensate lower reliability with higher redundancy
Review operating procedures: Let disks die rather than replacing
Two servers in production for ALICE (successfully so far)
”Monster” servers for optimising disk-to-infrastructure cost ratio
Testing 192 disks (2 trays with 8x24 HDD’s each) on one server
Up to 1.1PB raw with 6TB disks
Evaluate different file system and redundancy layouts (ZFS pools, RAID, EOS erasure encoding) -> f(capacity, reliability, performance)

13. Disk Servers for Archival?
Roberto Valverde/CERN
Disk Servers for Archival?
Current CERN EOS disk servers: one ”CPU” node + 2x24 enterprise-class capacity disks. JBOD with 2 replica.
Disk cost is 75%
TCO is ~3x of tape CHF/TB
Looking into optimising CHF/TB to close gap with tape. Ongoing investigations:
Using desktop disks à la BackBlaze (30%-40% cheaper)
No warranty / certification for our use case
Measure reliability of different models/vendors (integrity, failure rates) using SMART and EOS monitoring
Compensate lower reliability with higher redundancy
Review operating procedures: Let disks die rather than replacing
Two servers in production for ALICE (successfully so far)
”Monster” servers for optimising disk-to-infrastructure cost ratio
Testing 192 disks (2 trays with 8x24 HDD’s each) on one server
Up to 1.1PB raw with 6TB disks
Evaluate different file system and redundancy layouts (ZFS pools, RAID, EOS erasure encoding) -> f(capacity, reliability, performance)
Roberto Valverde/CERN

14. Optical Customers? Archival Disk – evolution of Blu-Ray
Collaboration between Sony & Panasonic
Max capacity 300GB disks (WORM only)
140MB/s write, 280MB/s read
Reliability (Blu-Ray UBER 10-12) -> erasure coding overhead
Roadmap to 1TB but no timeline nor new products
”1TB by” … now 2020 (was 2010, then )
Consumer market for optical disks vanishing
Behind magnetic storage in capacity, performance, reliability
Media volumes, pricing? ”Cheaper than HDD”
Libraries announced by Sony (Everspan) and Panasonic (Freeze-Ray)
Robots mounting media trays for 4x16 disks (Everspan) / 12 disks (Freeze-Ray)
Up to 14 expansion media 13PB raw (Everspan) -> ~180PB
Up to 64 drives / library (Everspan)
Customers?
Everspan evaluation started by LANL

15. Holographic Record information across media volume, not just surface
CCD readout
reference beam
signal modulation
media
Holographic
Record information across media volume, not just surface
High densities using different recording angles, wavelengths, position on single media location
Potentially, O(GB)/mm3 , fast R/W rates
2015 Demo: 2Tb/in2 -> ~770GB/in3
Prototypes, even ECMA standards
300GB/disk, 20MB/s
Small companies: InPhase (bankrupt in 2011), Akonia Holographics
IBM Almaden Labs (until ~2000); GE
No products on the market, nor any signs of upcoming ones

16. Summary
Tape is still the most cost-effective archival solution for HEP, but…
… concerns about the long-term sustainability of a contracting tape market dominated by a single technology provider
Disk market also contracting but from a wider base in terms of volume and vendors
Possible opportunities by exploring massive ”cheap” disk setups for archival in order to close the 3x cost gap wrt tape
Optical archival storage has recently seen a kind of revival, but no noticeable market impact yet
Will holographic ever surface?

17. References
Storage Technology and Markets (B. Panzer-Steindel, CERN IT CTO)
IBM enterprise and LTO product roadmap
Spectra Logic presentation to the DMF UG, 2017
INSIC consortium tape technology roadmap
LTO consortium 2016 tape capacity shipments
Clipper Group TCO study
ASTC disk technology roadmap, 2016
”Monster” node CERN/IT:
BackBlaze blog: Storage Pod
Sony Everspan specs
Akonia Holographics 2Tbit/in2 press release

18. Source: IBM