CERN IT Department CH-1211 Geneva 23 Switzerland t Experience with NetApp at CERN IT/DB Giacomo Tenaglia on behalf of Eric Grancher Ruben Gaspar Aparicio

CERN IT Department CH-1211 Geneva 23 Switzerland t Outline NAS-based usage at CERN Key features Future plans Experience with NetApp at CERN IT/DB - 2

Storage for Oracle at CERN 1982: Oracle at CERN, PDP-11, mainframe, VAX VMS, Solaris SPARC 32 and : Solaris SPARC with OPS, then RAC 2000: Linux x86 on single node, DAS 2005: Linux x86_64 / RAC / SAN –Experiment and part of WLCG on SAN until : Linux x86_64 / RAC / NFS (IBM/NetApp) 2012: all production primary Oracle databases (*) on NFS (*) apart from ALICE and LHCb online Experience with NetApp at CERN IT/DB - 3

Network topology All 10Gb/s Ethernet Same network for storage and cluster interconnect filer1filer2filer3filer4 serverB serverA serverCserverE serverD Ethernet switch Private 1 Ethernet switch Private 2 Internal HA pair interconnect Private network, both CRS and storage “public network” Ethernet switch Public

Domains: space/filers Total size (TB)Used for backup (TB)# of Filers des-nas shosts2044 gen3974 rac10596 rac11596 castor15418 acc2818 db disk10002 TOTAL Experience with NetApp at CERN IT/DB - 5

Typical setup

Impact of storage architecture on Oracle stability at CERN Experience with NetApp at CERN IT/DB - 7

Key features Flash cache RaidDP Snapshots Compression Experience with NetApp at CERN IT/DB - 8

Flash cache Help to increase random IOPs on disks –Very good for OLTP-like workload Don’t get wiped when servers reboot For databases –Decide what volumes to cache: fas3240>priority on fas3240>priority set volume volname cache=[reuse|keep] 512 GB modules 1 per controller Experience with NetApp at CERN IT/DB - 9

IOPs and Flash cache Experience with NetApp at CERN IT/DB - 10

IOPs and Flash cache Experience with NetApp at CERN IT/DB - 11

Key features Flash cache RaidDP Snapshots Compression Experience with NetApp at CERN IT/DB - 12

Disk and redundancy (1/2) Disks are larger and larger –speed stay ~constant → issue with performance –bit error rate stay constant ( to ), increasing issue with availability With x as the size and α the “bit error rate” Experience with NetApp at CERN IT/DB - 13

Disks, redundancy comparison (2/2) 1 TB SATA desktop Bit error rate 10^-14 RAID 17.68E-02 RAID 5 (n+1)3.29E E E-01 ~RAID 6 (n+2)1.60E E E-13 ~triple mirror8.00E-16 1TB SATA enterprise Bit error rate 10^-15 RAID 17.96E-03 RAID 5 (n+1)3.92E E E-01 ~RAID 6 (n+2)1.60E E E-15 ~triple mirror8.00E GB FC Bit error rate 10^-16 RAID 14.00E-04 RAID 5 (n+1)2.00E E E-02 ~RAID 6 (n+2)7.20E E E-17 ~triple mirror3.60E TB SATA enterprise Bit error rate 10^-15 RAID 17.68E-02 RAID 5 (n+1)3.29E E E-01 ~RAID 6 (n+2)1.60E E E-14 ~triple mirror8E-17 Experience with NetApp at CERN IT/DB - 14 Data loss probability for different disk types and groups

Key features Flash cache RaidDP Snapshots Compression Experience with NetApp at CERN IT/DB - 15

Snapshots Experience with NetApp at CERN IT/DB - 16 T0: take snapshot 1

Snapshots Experience with NetApp at CERN IT/DB - 17 T0: take snapshot 1 T1: file changed

Snapshots Experience with NetApp at CERN IT/DB - 18 T0: take snapshot 1 T1: file changed T2: take snapshot 2

Snapshots for backups With data growth, restoring databases in reasonable amount of time is impossible using “traditional” restore/backup techniques 100TB, 10GbE, 4 tape drives Tape drive restore performance ~120MB/s Restore ~ 58 hours (but it can be much longer) Experience with NetApp at CERN IT/DB - 19

Snapshots and Real Application Testing Capture insert… PL/SQL update … delete … Original Clone Upgrade Replay insert… PL/SQL update … delete … Experience with NetApp at CERN IT/DB - 20

Snapshots and Real Application Testing Capture insert… PL/SQL update … delete … Original Clone Upgrade Replay insert… PL/SQL update … delete … SnapRestore® Replay insert… PL/SQL update … delete … Replay insert… PL/SQL update … delete … Experience with NetApp at CERN IT/DB - 20

Key features Flash cache RaidDP Snapshots Compression Experience with NetApp at CERN IT/DB - 21

NetApp compression factor Experience with NetApp at CERN IT/DB - 22

Compression: backup on disk RMAN File backup 1x tape copy + Disk buffer Raw: ~1700 TiB (576 3TB disks) Usable: 1000 TiB (to hold ~2PiB uncompressed data) Experience with NetApp at CERN IT/DB - 23

Future: OnTap Cluster Mode Non-disruptive upgrades/operations: the immortal cluster Interesting new features –Internal DNS load balancing –Export policies: fine-grained access for NFS exports –Encryption and compression at storage level –NFS 4.1 implementation, parallel NFS Scale-out architecture: up to 24 (512 theoretical) Seamless data moves for capacity, performance rebalancing or hardware replacement Experience with NetApp at CERN IT/DB - 24

Architecture view – Ontap cluster mode Experience with NetApp at CERN IT/DB - 25

Possible implementation Experience with NetApp at CERN IT/DB - 26

Logical components Experience with NetApp at CERN IT/DB - 27

pNFS NFS 4.1 standard (client caching, Kerberos, ACL) Coming with Ontap 8.1RC2 Not natively supported by Oracle yet In RHEL 6.2 Control protocol: provides synchronization among data and metadata server pNFS between client and MDS, get where information is store Storage access protocols: file-based, block-based and object- based pNFS Storage access protocols Experience with NetApp at CERN IT/DB - 28

CERN IT Department CH-1211 Geneva 23 Switzerland t Summary Good reliability –Six years of operations with minimal downtime Good flexibility –Same setup for different uses/workloads Scales to our needs Experience with NetApp at CERN IT/DB - 29

CERN IT Department CH-1211 Geneva 23 Switzerland t Q&A Thanks! Experience with NetApp at CERN IT/DB - 30