1. Linux Virtual Server Jim Lawson VAGUE/University of Vermont jim@jimlawson.org / Jim.Lawson@uvm.edu jim@jimlawson.orgJim.Lawson@uvm.edu

2. What is a load balancer? ● Front-end appliance for a web (or other service) farm ● Allows you to “scale out” rather than “scale up” ● Several vendors supply products in this space (Cisco, F5, Foundry, others)

3. What is LVS? ● Linux Virtual Server ● (“IPVS” inside the kernel) ● Kernel-space load balancer ● Fast, efficient, reliable ● Somewhat feature-limited compared to commercial options

4. What is keepalived? ● Provides: – Health-checking for “realservers” - takes malfunctioning servers out of the pool – Failover for director/load- balancer

5. LVS NAT config

6. LVS DR config

7. Why DR (direct-route?) ● Director only needs to handle request portion of traffic. ● In typical HTTP, request is small (typically <1K) – GET /index.html HTTP/1.1 – Host: www.uvm.edu ● Response is sent directly to gateway – HTTP/1.1 200 OK... –... Content-size: 22947

8. Why DR (direct-route?) ● Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) 123multimedia (dot) com 06/06/2005: – I have 38 realservers behind my director, incoming traffic (to director) goes up to 20Mb/s, outgoing (from realservers LVS-DR setup) up to 60Mb/s. I have about 1200 sites hosted. 36 virtual_server entries in keepalived.conf, 30 VIPs. There's no noticable load on the poor PIII/700 director that's handling the traffic.

9. Why not DR? ● ARP problem – Realservers have to be configured to not “ARP” for the VIP – only the director should respond to ARP requests for that IP ● Linux: 2.2, 2.4 kernels need “hidden” arp patch ● 2.6 kernels only need arp_ignore and arp_announce set in /proc. ● Most other unixes (unices?): NOARP works fine ● Windows (since NT4SP2): ifconfig -arp

10. Why not DR? ● If you forget to set NOARP (or hidden/arp_announce) before you bring the VIP up on the realserver, the realserver will receive all traffic bound for the VIP! – To avoid this, put the VIP config in a special startup script which always sets the proper flags in /proc – In general, it is a good idea to have 1 VIP per service or pool

11. LVS scheduling algorithms ● rr (round-robin) ● lc (pick server with least # connections) ● wrr, wlc – weighted versions of above ● For load balanced caching proxy servers: – DH (destination hash, static, based upon destination IP) – LBLC (locality-based least connection; like DH but dynamic)

12. lc example graph (rrd image courtesy Salvatore D. Tepedino) LVS with 2 realservers, serving http all day

13. lc vs rr ● Round-robin keeps servers “more or less” evenly balanced ● Least-connections is very good at keeping them evenly balanced ● BUT... “thundering herd problem” – Newly added or recovered realservers have no active connections! Guess where everyone gets sent?

14. Keepalived ● Monitors services for availability – Built-in checks: http, https, smtp, ldap, “tcp” – Custom scripts are easy to plug in – Threaded Health Checks ● When services go down, servers are removed from pool and users are automatically redirected to remaining available nodes

15. What about the director? ● It's a single point of failure ● Solution: keepalived VRRP – Virtual Router Redundancy Protocol – RFC 2338, election protocol, multicast – Similar to Cisco's HSRP – active/passive – Can have VIPs staggered between directors for active/active config

16. What about the director? ● Active connection state (client IP realserver) is communicated via IPVS syncd – Active server informs passive server about new associations – Runs over crossover cable, or LAN ● During a failover, “gratuitous ARP” is sent ● Failback: set PREEMPT_DELAY

17. CIT LVS config