LHC Logging Cluster Nilo Segura IT/DB

Agenda ● Hardware Components ● Software Components ● Transparent Application Failover ● Service definition

Hardware Components ● Two Sun Fire V240 – Dual CPU 1Ghz, 4Gb memory – Dual internal disks, dual power supply ● One Sun Storedge 3510FC – 2Gb fiber channel architecture – 12x146Gb 10k RPM FC disks – Two Raid controllers with 1GB cache – Can accept up to 2x3510 Jbod expansion trays ● Both machines share the same set of disks – The 3510 can accept up to 8 hosts directly attached (or up to 4 with a redundant config.).

Software Components ● Sun Cluster 3.1 Update 1 ● Solaris 9 ● Oracle RDBMS (Real Application Cluster) ● Oracle Distributed Lock Manager ● Veritas Volume Manager 3.5 ● Sun certification completed – checking correct level of patches – shutting down one of the nodes – disconnecting one of the nodes from the disk system – etc..

High Availability ● The purpose of the cluster installation is to offer 365x24 access to the database ● No single point of failure – Two nodes, two disk system, two.... ● Recovery/Availability offered by the Oracle software (Real Application Cluster) – Transactions are recovered by the surviving instance ● Tested the following cases – Listener down (re-connection immediate) – Listener up but instance down (re-connection immediate) – Machine down (re-connection takes longer, 3minutes connecting from a Linux client due to TCP driver timeout) ● Timeout can be tweaked but...

Transparent Application Failover ● For SELECT operations, if the connection is lost, the session is resumed transparently in the surviving node – Tested and working, the session stops for a few seconds and then resumes withouth the user issuing a new connect request – Not tested from a JDBC Thin driver.... it will work with the JDBC OCI driver ● Sessions modifying data will still lose the connection and need to re-connect – As expected, the current transaction will be rolled-back ● Possibility of LOAD BALANCING at the level of the connect string – Not enabled for the moment, perhaps later

Service definition - General ● Service to run 365x24, backups will not interrupt the database access – Export + hot backups – Oracle Recovery Manager will reduce the backup window time ● Problems with the service to be reported to and/or Oracle GSM telephone (depending on the criticality of the – Same mechanism used for SUNSLPS and LEP Database servers ● However, the system can still collapsed due to other reasons (network outage, power failure, gremlins....) so applications must be able to react to these events (local buffering?) – Instance failure when recovering a distributed transaction – Surviving instance tried to recover and crashed in the same point

Service Definition - Patches ● We may need to interrupt the service for updates... – If all goes well, one day (scheduled) interruption per year – We should be able to apply Solaris patches one node at a time ● Moving applications from one node to another – Oracle offers apparently Rolling upgrade features in their RAC patches ● Some patches that touch common structures used by all the instances will still require database downtime ● But : critical security patches may need to be applied at any given moment (following Sun and/or CERN SecurityChief requests) – Removed all unneeded Solaris services to avoid potential problems ● Private firewall for all the database servers ala AIS ?

lhclog=(DESCRIPTION= (FAILOVER=on) (LOAD_BALANCE=off) (ADDRESS= (PROTOCOL=TCP) (HOST=sunlhclog01.cern.ch) (PORT=1521) ) (ADDRESS= (PROTOCOL=TCP) (HOST=sunlhclog02.cern.ch) (PORT=1521) ) (CONNECT_DATA= (SERVICE_NAME=LHCLOGDB) (FAILOVER_MODE= (TYPE=SELECT) (METHOD=BASIC) )

lhclog=(DESCRIPTION= (FAILOVER=on) (LOAD_BALANCE=off) (ADDRESS= (PROTOCOL=TCP) (HOST=sunlhclog01.cern.ch) (PORT=1521) ) (ADDRESS= (PROTOCOL=TCP) (HOST=sunlhclog02.cern.ch) (PORT=1521) ) (CONNECT_DATA= (SERVICE_NAME=LHCLOGDB) (FAILOVER_MODE= (TYPE=SELECT) (METHOD=PRECONNECT) )

Database ● Space will be managed automatically by Oracle – No need to specify extent size – Unlimited number of extents