CS526 Dr.Chow1 HIGH AVAILABILITY LINUX VIRTUAL SERVER By P. Jaya Sunderam and Ankur Deshmukh

CS526 Dr.Chow2 WHAT IS LINUX VIRTUAL SERVER? Highly scalable server built on cluster of real servers. Users see only a single IP address. Front end of the real servers is the load balancer. Schedules requests to the different servers and make parallel services of the cluster to appear as a virtual service on a single IP address. Scalability is achieved by transparently adding or removing a node in the cluster. Forms the basic building block of HA-LVS.

CS526 Dr.Chow3 CONTD..

CS526 Dr.Chow4 CONCEPT OF HIGH AVAILABILITY What is High Availability? Providing high availability of services for critical commercial applications Provided by detecting node or daemon failures and reconfiguring the system appropriately so that the workload can be taken over by the remaining nodes in the cluster. How it is used in LVS? Principle : One load balancer is the master while the other is slave. If master fails then the slave takes up the master’s pending tasks. Also checks the health status of the real servers.

CS526 Dr.Chow5 LVS to HA-LVS The LVS by itself is not configured to be highly available. Patch with other services to make it highly available. The high availability can be provided by using any of these services in conjunction with each other: 1. mon 2. heartbeat 3. ldirectord 4. Coda fault tolerant file system

CS526 Dr.Chow6 THE OVERALL PICTURE

CS526 Dr.Chow7 FAILURES INTO ACCOUNT BY HA-LVS HA-LVS handles two types of failures Director Failure: When the load balancer fails or crashes Handled by having a redundant director available One director maintains all the session and IP information On failover this information is handed over to the redundant director 2. Real server Failure: When the service to the real server fails or crashes Handled by using a signal running on the director that checks the health of real servers after every constant time interval. If a certain service is down the it is removed from the directors “ipvsadm” table. When the service comes back again the it is again added back to the director’s “ipvsadm” table

CS526 Dr.Chow8 HEARTBEAT/FAKE The "fake" software is used for the backup to takeover the IP addresses of the load balancer when the load balancer fails The "heartbeat" code is used to detect the status of the load balancer to activate/deactivate the "fake" on the backup server. Two heartbeat daemons run on the primary and the backup, they heartbeat the message like "I'm alive" each other through the serial line periodically. When the heart code daemon of the backup cannot hear the "I'm alive" message from the primary in the defined time, it activates the fake to take over the Virtual IP address to provide the load-balancing service. When it receives the "I'm alive" message from the primary later, it deactivate the fake to release the Virtual IP address, and the primary comes back to work again.

CS526 Dr.Chow9 MON General purpose resource monitoring system. Used for monitoring services and triggering alerts in events of failure which are viewed as two different tasks. In case of HA-LVS used to check the real servers health i.e. to check which services are down and which services are running, by sending alerts after certain time intervals. The fping.monitor is configured to detect whether the server nodes is alive every t seconds, and the relative service monitor is also configured to detect the service daemons on all the nodes every m seconds. An alert was written to remove/add a rule in the linux virtual server table while detecting the server node or daemon is down/up. Therefore, the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.

CS526 Dr.Chow10 CODA Coda is a fault-tolerant distributed file system so that the files are easily available and easy to manage. It was developed by the Carnegie-Melon University in It has many features that are very desirable for network file systems. 1. disconnected operation for mobile computing 2. Is freely available under a liberal license 3. high performance through client side persistent caching 4. server replication 5. security model for authentication, encryption and access control 6. continued operation during partial network failures in server network 7. network bandwidth adaptation 8. good scalability 9. well defined semantics of sharing, even in the presence of network failures

CS526 Dr.Chow11 IMPLEMENTATION Heartbeat implementationHeartbeat implementation Overview of /etc/ha.d/ha.cf Key parameters usedKey parameters used Keep aliveKeep alive Dead timeDead time Warn timeWarn time UdpUdp Medium –Ethernet,serial consoleMedium –Ethernet,serial console Node –Name of machine in clusterNode –Name of machine in cluster Watchdog –Reboots system if heart doesn’t beat for a specified amt of timeWatchdog –Reboots system if heart doesn’t beat for a specified amt of time

CS526 Dr.Chow12 Heartbeat

CS526 Dr.Chow13 CONTINUED.. Mon implementation ServerServer ClientClient SERVER SERVER Runs on Load balancerRuns on Load balancer Monitors services for availability, sends alarms upon failuresMonitors services for availability, sends alarms upon failures CLIENT CLIENT Runs on the real serverRuns on the real server Sends commands to mon daemon and shows the results of the corresponding repliesSends commands to mon daemon and shows the results of the corresponding replies

CS526 Dr.Chow14 Mon

CS526 Dr.Chow15 References Masters thesis of C Prakash at UCCS thesis of C Prakash at UCCS