23 September 2004 Evaluating Adaptive Middleware Load Balancing Strategies for Middleware Systems Department of Electrical Engineering & Computer Science Vanderbilt University, Nashville Jaiganesh Balasubramanian Ossama Othman Douglas C. Schmidt Lawrence Dowdy 8 th EDOC Conference, Monterey, California

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University R&D Motivation: Scaleable Distributed Systems Heterogeneous environments Large number of bursty clients Stringent QoS requirements, e.g.: –24x7 availability –Low latency & high throughput Examples –Online trading systems –Mission-critical systems for critical infrastructure e.g., air transportation, power grid control Our R&D goal is to assure the scalability of these types of distributed systems

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University General Approach: Scalability via Load Balancing Goal: Improve the efficiency & scalability of distributed systems Approach: Partition & balance application tasks across many computers via load distribution strategies Load balancing be implemented at multiple layers, e.g.: –Network layer –OS layer –Middleware layer

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Our Approach: Middleware Load Balancing Capabilities of middleware –Control end-to-end resources & QoS –Leverage hardware/software advances –Handle new environments & requirements –Provide reusable COTS services Motivation for middleware load balancing –Can take into account distributed system state, system runtime behavior, & request content –Application-level control over load balancing policies Hardware Domain-Specific Middleware Services Common Middleware Services Distribution Middleware Host Infrastructure Middleware Operating Systems & Protocols Applications

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University TAO & Cygnus: Our Middleware Environment Based on The ACE ORB (TAO) – an open-source, high-performance, real-time CORBA Object Request Broker –deuce.doc.wustl.edu/ Download.html Cygnus is a middleware load balancing service implemented atop TAO TAO also provides many other middleware services LOADBALANCING RT-CORBA EVENT SERVICESECURITYTRANSACTIONS DYNAMIC/STATIC SCHEDULING AV STREAMS

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Cygnus Load Balancing Architecture Multiple Clients can invoke requests –Potentially non-deterministic –Duration called a session Members –Multiple instances of the same object implementation Object Groups –Collections of members among which loads will be distributed equitably Load Balancer –Transparently distributes requests to members within an object group

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Load Monitor –Provides load feedback Load Analyzer –Determines location and member load conditions Member Locator –Binds client to appropriate object group member Load Alert –Facilitates load control (load shedding) Load Manager –Mediates all interactions between load balancer components Cygnus Load Balancing Components

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Types of Load Balancing Strategies Non-adaptive strategies –Use static information to make load balancing decisions e.g., cyclic selection of servers Adaptive strategies –Use dynamic system information to make load balancing decisions e.g., runtime host load information, CPU utilization –Have runtime parameters that must be set properly

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Cygnus Load Balancing Strategies Non-adaptive strategies –RoundRobin & Random strategies Adaptive strategies –LeastLoaded strategy rejects new sessions so that load is balanced RejectThreshold controls runtime rejection decision –LoadMinimum strategy migrates client sessions from one machine to another to balance load MigrationThreshold controls runtime migration decision

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Cygnus Adaptive Load Balancing Interactions Acceptance strategies Migration strategies

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Evaluating Load Balancing Strategies with LBPerf We use LBPerf to empirically determine appropriate load balancing strategies for applications with various workload characteristics Open-source tool suite for evaluating middleware load balancing strategies Can tune different configurations of middleware load balancing –e.g., systematically choose different load balancing strategies & runtime parameters associated with those strategies Evaluate strategies using different metrics –e.g., throughput, latency, CPU utilization, overhead, etc. Client code generated by LBPerf Server code generated by LBPerf Load balancer code with option to change strategies & run-time parameters Client workloads can vary!!

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Summary of Testbed & Experiments Emulab is a NSF-sponsored testbed for conducting large-scale network/system experiments – LBPerf experiments conducted with 16 clients & 32 servers RedHat Linux 7.1, Pentium III 850Mhz & 600Mhz machines LBPerf experiments run in Emulab –Overhead measurements –Throughput measurements –CPU utilization –Migration threshold values –Reject threshold values

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Overhead of Cygnus (1/2) CPU-intensive requests Single server Single-threaded clients generate CPU-intensive requests on one server Repeat experiment with & without load balancer Measure throughput, defined as # of requests processed per- second by server Without load balancer With load balancer

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Overhead of Cygnus (2/2) Single-threaded clients generate CPU-intensive requests on one server Repeat experiment with & without load balancer Measure throughput, defined as # of requests processed per- second by server Overhead of using Cygnus is negligible

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Load Balancing Effectiveness (1/3) Load balancer code with option to change strategies & run-time parameters CPU-intensive requests Single-threaded clients generate CPU- intensive requests on 4 servers Repeat experiment, choosing each of four strategies Measure –Throughput, defined as # of requests processed per- second by server –CPU utilization at each server

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Load Balancing Effectiveness (2/3) Single-threaded clients generate CPU- intensive requests on 4 servers Repeat experiment, choosing each of four strategies Measure –Throughput, defined as # of requests processed per- second by server –CPU utilization at each server Adaptive strategies Non-adaptive strategies Adaptive strategies behave better than non-adaptive strategies with non-uniform CPU intensive workloads

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Load Balancing Effectiveness (3/3) Adaptive strategies ensure uniform utilization of servers Single-threaded clients generate CPU- intensive requests on 4 servers Repeat experiment, choosing each of four strategies Measure –Throughput, defined as # of requests processed per- second by server –CPU utilization at each server

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Effect of Migration Threshold (1/2) Single threaded clients generate CPU- intensive requests on 4 servers Use LoadMinimum strategy Repeat experiments for different MigrationThreshold values Measure throughput, defined as # of requests processed per-second by server Load balancer code with LoadMinimum strategy Change MigrationThreshold values CPU-intensive requests

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Single threaded clients generate CPU- intensive requests on 4 servers Use LoadMinimum strategy Repeat experiments for different MigrationThreshold values Measure throughput, defined as # of requests processed per-second by server Migration’s not always effective Measuring Effect of Migration Threshold (2/2)

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Effect of Reject Threshold (1/2) Single threaded clients generate CPU- intensive requests on 4 servers Use LeastLoaded strategy Repeat experiments for different RejectThreshold values Measure throughput, defined as # of requests processed per-second by server Load balancer code with LeastLoaded strategy Change RejectThreshold values CPU-intensive requests

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Measuring Effect of Reject Threshold (2/2) Single threaded clients generate CPU- intensive requests on 4 servers Use LeastLoaded strategy Repeat experiments for different RejectThreshold values Measure throughput, defined as # of requests processed per-second by server Reject sessions to ensure QoS for admitted sessions

Balasubramanian et allEvaluating Load Balancing Strategies Vanderbilt University Concluding Remarks Cygnus adds little overhead for load balancing, while providing increased scalability & utilization Adaptive load balancing strategies can perform much better than non-adaptive strategies in presence of non-uniform loads Different run-time parameters associated with adaptive strategies must be tuned to perform well Future work: Devise self-adaptive load balancing architectures –Take QoS specifications as input –Tune run-time parameters dynamically to achieve those specifications