Evolution of database services Eva Dafonte Pérez 2014 WLCG Collaboration Workshop

Outline CERN’s databases overview Service evolution HW Storage Service configuration SW New database services Replication Database On Demand (DBoD) service Hadoop + Impala Future plans Summary

CERN’s databases ~100 Oracle databases, most of them RAC Mostly NAS storage plus some SAN with ASM ~500 TB of data files for production DBs in total Example of critical production DBs: LHC logging database ~170 TB, expected growth up to ~70 TB / year But also as DBaaS, as single instances 120 MySQL Open community databases 11 PostgreSQL databases 10 Oracle11g And additional tests setups: Hadoop + Impala, CitusDB Add Impala/Hadoop/CitusDB

Our deployment model DB clusters based on RAC Load balancing and possibility to growth High Availability – cluster survives to node failures Maintenance – rolling interventions Schema based consolidation Many applications share the same RAC cluster Per customer and/or functionality Example: CMS offline database cluster RAC is an ideal environment for distributing load amongst multiple instances accessing the same physical database Real Application Clusters (Oracle) Servers running RHEL NAS (Network-attached Storage) from NetApp 10 Gig Ethernet Between 2 and 5 nodes RAC Instance 1 Clusterware OS RAC Instance 2 Shared Storage Public Network Private Network Storage Network

New systems Installation Service evolution HW Migration SW upgrade Stable services Decomission New systems Installation Preparation for RUN2 Changes have to fit LHC schedule New HW installation in BARN Decommission of some old HW Critical power move from current location to BARN Keep up with Oracle SW evolution Applications’ evolution - more resources needed Integration with Agile Infrastructure @CERN LS1: no stop for the computing or other DB services Additional challenge, no all systems/services are stopped and therefore the DB services cannot be stopped.

Hardware evolution 100 production servers in the BARN Dual 8 core XEON e5-2650 128GB/256GB RAM 3x10Gb interfaces Specific network requirements IP1, ATLAS PIT, Technical Network, routed and non-routed network New generation of storage from NetApp Specific network requirements IP1 (cs network) Atlas pit Technical network Routed network accessible from outside of CERN, Non - routed network on only internal to CERN

Storage evolution NVRAM 1.0 GB 8.0 GB System memory 8GB 64GB CPU FAS3240 FAS8060 NVRAM 1.0 GB 8.0 GB System memory 8GB 64GB CPU 1 x 64-bit 4-core 2.33 Ghz 2 x 64-bit 8-core 2.10 Ghz SSD layer (maximum) 512GB 8TB Aggregate size 180TB 400TB OS controller Data ONTAP® 7-mode C-mode 7-mode: two controllers clustered for HA C-mode: multiple pairs connected (clustered) via back-end 10GbE network, you can have a single storage system with more than two controllers scaling up scaling out

Storage consolidation 56 controllers (FAS3000) & 2300 disks (1400TB storage) … 1 2 7 14 controllers (FAS6220) & 960 disks (1660 TB storage) Storage islands, accessible via private network

Storage setup DB cluster A DB cluster B Power cut for one rack (6th June) Serious outage (~25% of DB servers went down) didn’t break the service Thanks to proper design: DB servers distribution & HA of Oracle RAC clusters Most of the services continues normal operation Sessions connected to failing nodes were lost Root cause – human error

Storage setup  Easy management More capacity Transparent volume move Caching: flash cache and flash pool Performance improvement ~2-3 times more of overall performance  Difficulties finding slots for interventions Flash cache – helps to increase random IOPS on disks

Service configuration - Puppet Following CERN IT’s strategy, IT-DB group adopted Puppet Good occasion to re-think how the services are configured and managed Rely on the same Syscontrol-LDAP* data source for Quattor-managed services Developed custom modules for: Private storage and network configuration Database installation Backups configuration Removed ssh keys and service accounts in favour of kerberos+sudo Improves traceability & manageability RHEL 5 8 RHEL 6 * Syscontrol-LDAP for IT-DB: stores configuration for IT-DB services

New Oracle releases Production was running on 11.2.0.3 (Oracle 11g) 11.2.0.4 Terminal patch set Additional support fees from January 2016 Extended support ends January 2018 (Oracle 12c) 12.1.0.1 First release Next patch set 12.1.0.2 coming in Q3 2014 Educated Guess: users of 12.1.0.1 will have to upgrade to 12.1.0.2 or higher by 2016 No current Oracle version fits well the entire RUN 2

Oracle upgrade Move IT-DB services to Oracle 12c gradually Majority of DB services upgraded to 11.2.0.4 Few candidate services upgraded to 12.1.0.1 ATLARC, LHCBR, PDBR, LEMON, CSDB, CSR, COMPR, TIMRAC Compatibility kept to 11.2.0.3 12c Oracle clusterware deployed everywhere Does not conflict with 11g version of RDBMS Newer 12c releases being/will be tested SW upgrade for aprox 80 databases to 11.2.0.4 or 12.1.0.1 HW migration for 32 critical production databases And 2 additional production database services Create new development DB on 12.1.0.1 – devdb12 September 2013 Upgrade devdb11 DB to 11.2.0.4 October 2013 Move integration DBs to new HW November 2013 Upgrade test and integration DBs December 2013 and January 2014 Test restores & upgrades of prod DBs on new HW Q1 2014 Move to new HW and/or upgrade of production DBs Q1 and Q2 2014

New database services QPSR SCADAR Quench Protection System Will store ~150K rows/second (64GB per redo log) 1M rows/second achieved during catch-up tests Need to keep data for a few days (~ 50 TB) Doubtful if previous HW could have handled that SCADAR Consolidated WinCC/PVSS archive repository Will store ~50-60K rows/second (may increase in the future) The data retention varies depending on the application (from a few days to 5 years)

Replication Plan to deploy Oracle Golden Gate at CERN and Tier1s In order to replace Oracle Streams replication Streams is phased out Some online to offline setups were already replaced by Oracle Active Data Guard Replication Technology Evolution Workshop @CERN in June Migration plan agreed with experiments and Tier1s Centralised GG configuration GG software only at CERN Trail files only at CERN No GG management at T1s LHCb – presentation during LHCb sw week ATLAS – well represented during the workshop Only COOL replication, PVSS will be moved to ADG A C E R Remote site replica Downstream cluster

Database On Demand (DBoD) Openstack Puppetdb (MySQL) Lhcb-dirac Atlassian databases LCG VOMS Geant4 Hammercloud dbs Webcast QC LHC Splice FTS3 DRUPAL CernVM VCS IAXO UNOSAT … https://cern.ch/twiki/bin/view/DB/DBOnDemandManifesto Covers a demand from CERN community not addressed by the Oracle service Different RDBMS: MySQL, PostgreSQL and Oracle Making users database owners Full DBA privileges No access to underlying hardware Foreseen as single instance service No DBA support or application support No vendor support (except for Oracle) It provides tools to manage DBA actions: configuration, start/stop, upgrades, backups & recoveries, instance monitoring

DBoD evolution PostgreSQL since September 2013 Deprecated virtualization solution based on RHEL + OVM HW servers and storage evolution as for the Oracle database services Migration to CERN Agile infrastructure Customized RPM packages for MySQL and PostgreSQL servers High Availability cluster solution based on Oracle clusterware 4 nodes cluster (3 nodes active + 1 as spare) SW upgrades MySQL currently migrating to 5.6 Oracle 11g migrating towards Oracle 12c multi-tenancy Tape backups Clusterware: 12.1.0.1 PostgreSQL and MySQL instances can co-exist, different versions supported

Hadoop Using raw MapReduce for data processing requires: Abstract decomposition of a problem into Map and Reduce steps Expertise in efficient Java programming Impala – SQL engine on top of HDFS Alternative solution to MapReduce Data cache available Easy access: JDBC driver provided Performance benchmarks on synthetic data (early results) Test case: simple CSV  table mapping Full scan 36 million rows (small sample): 5.4M rows/sec Cached: 10x faster Full scan of 3.6 billion rows (100x more): 30M rows/sec IO: ~3.7GB/s with storage throughput ~4GB/s Cache does not help – too large data set for cache Sequential data access with Oracle and Hadoop: a performance comparison (Zbigniew Baranowski – CHEP) Test case – counting exotic muons in the collection Oracle performs well for small clusters but scaling ability is limited by shared storage. Hadoop scales very well. However, writing efficient Map Reduce code is not trivial Using raw MapReduce for data processing requires: abstract decomposition of a problem into Map and Reduce steps expertise in efficient Java programming Test case – no additional tune-ups – Importing data from Oracle into Hadoop (Sqoop, Golden Gate adapter)

Future plans New HW installations SW upgrades Second cluster in BARN Wigner (for Disaster Recovery) SW upgrades First Oracle 12c patch set 12.1.0.2 (Q3 2014) More consolidation – run different DB services on the same machine Study the use of Oracle Golden Gate for near zero downtime upgrades Quattor decommission DBoD High density consolidation Cloning and replication Virtualization as OpenStack evolves Hadoop + Impala Columnar storage (Parquet) Importing data from Oracle into Hadoop Tests with production data (WLCG dashboards, ACC logging, …) Analyze different engines (Shark)

Summary HW, Storage, SW and configuration evolution for the DB service during last year Complex project Many people involved at various levels Experience gained will be very useful for the new installations Careful planning is critical Validation is a key to successful change New systems give more capacity and stability for RUN2 New services provided and more coming Keep looking at new technologies

Q&A

7-mode vs C-mode Private network Private network client access Cluster interconnect Cluster mgmt network client access

Flash cache and flash pool Helps increase random IOPS on disks Warm-up effect Controller operations (takeover/giveback) invalidate the cache Flash pool Based on SSD drives Availability to cache random overwrites Heat map in order to decide what stays and for how long in SSD cache Most basic difference is that Flash cache is a PCI-e card that sits in the NetApp controller, whereas Flash pool are SSD drives that sits in the NetApp shelves. Flash cache memory is always going to have a much higher potential for throughput than Flash pool memory that sits on SAS stack. Planned downtime – flash cache data can be copied to the HA pair´s partner node BUT Unplanned downtime – flash cache data is lost!! However flash pool will be accessible on the other controller (aggregate will fail over to the other controller). Another important difference is flash pool’s availability to cache random overwrites Writing to SSD is up to 2x quicker than writing to rotating disk, which means that the consistency point occurs faster. (Random overwrites are the most expensive and slowest operation that we can perform to a rotating disk, so it is in our interest to accelerate them) Flash cache utilizes a first-in first-out algorithm Flash pool utilizes a temperature map algorithm Write to disk read overwrite Eviction scanner Insert into SSD write Every 60 secs & SSD consumption > 75% hot warm neutral cold evict

Hadoop Sequential data access with Oracle and Hadoop: a performance comparison Zbigniew Baranowski – CHEP 2013 Test case – counting exotic muons in the collection Oracle performs well for small clusters but scaling ability is limited by shared storage Hadoop scales very well However, writing efficient Map Reduce code is not trivial

Hadoop + Impala 4 nodes each: Intel Xeon L5520 @2.27GHz (Quad) 24GB RAM > 40TB storage (sum of single HDDs capacity) Pros: Easy to setup – works „out of the box” Acceptable performance Promising scalability Cons: No indexes SQL is not everything on RDBMS! No indexes is not a big deal because for small data sets data is cached pretty well and for bigger data sets indexes are millstones around DB’s neck. What to do with PL/SQL procedures, jobs, APEX applications? Providing SQL does not solve all the problems with migrating from RDBMS to i.e. Impala but helps a lot!