Transparent Adaptive Resource Management for Middleware Systems Jaiganesh Balasubramanian Ossama Othman Dr. Douglas C. Schmidt {jai, ossama, schmidt}@dre.vanderbilt.edu Department of Electrical Engineering and Computer Science Vanderbilt University, Nashville

Distributed Systems Typical issues with distributed systems Heterogeneous environments Concurrency Large bursts of client requests 24/7 availability Stringent QoS requirements Examples of distributed systems E-commerce Online trading systems Mission critical systems Vanderbilt University

Motivation Development and maintenance of QoS-enabled distributed systems Non-trivial Requires expertise that application developers often lack Solution: Middleware (e.g. CORBA) Can shield distributed system developers from the complexities involved with developing distributed applications Can facilitate manipulation of QoS requirements and management of resources Vanderbilt University

Load Balancing Load balancing can improve the efficiency and scalability of a distributed system Load balancing service can be implemented in the following layers Network layer OS layer Middleware layer Why choose middleware-layer load balancing? Allows additional components to be added to distributed systems with minimal impact to performance Improves availability due to inherent redundancy Can be composed with other services (e.g. fault tolerance, security, etc) Can take into account request content Vanderbilt University

Distributed System Resource Management Managing resources transparently is important for legacy DRE systems “Transparency” is hard “Transparency is essential Middleware in use may need to be enhanced to support this functionality Such an enhancement can be found in a middleware-based load balancing service Vanderbilt University

Common Scenario Multiple clients making request invocations Potentially non-deterministic Members Multiple instances of the same object implementation Object groups Collections of members among which loads will be distributed equitably Logically a single object Load balancer Transparently distributes requests to members within an object group Vanderbilt University

TAO Load Balancer (Cygnus) Cygnus makes it easier to develop distributed applications by providing application transparency, scalability, flexibility, adaptability and interoperability. Has the following built-in strategies: - RoundRobin - Random - LeastLoaded EVENT SERVICE TRANSACTIONS SECURITY AV STREAMS DYNAMIC/STATIC SCHEDULING LOAD BALANCING RT-CORBA 1.0 Vanderbilt University

Cygnus Load Balancing Strategies Client binding granularity Per-session, client permanently forwarded to a replica Per-request, requests forwarded on client’s behalf On-demand, client can be rebound to another replica when necessary Vanderbilt University

Cygnus Load Balancing Architectures Load balancing architecture comprised of a combination of client binding granularity and balancing policy Given the strategies just described, there are six possible architectures Three common architectures Non-adaptive per-session Adaptive per-request Adaptive on-demand Vanderbilt University

Cygnus Load Balancer Components Load Monitor Provides load feedback Load Analyzer Determines location and member load conditions Member Locator Binds client to appropriate object group member Load Alert Facilitates load control (load shedding) Load Manager Mediates all interactions between load balancer components Vanderbilt University

Cygnus Load Monitor Facilitates feedback Monitor and report loads Load monitoring is location-oriented The loads at a given location are monitored and reported Contrast with loads on a given object group member Vanderbilt University

Cygnus Load Analyzer Load Analyzer Decides which member will receive the next client request Determines load condition at each location Induces load shedding Extensible load balancing strategies Set via the PropertyManager interface Each object group may use a different strategy Vanderbilt University

Cygnus Member Locator Implements the Interceptor design pattern Transparently forwards client requests to member retrieved from the load analyzer In CORBA, redirection induced via standard GIOP LOCATION_FORWARD message Conforming client side ORB will transparently re-issue request to member chosen by load balancer Vanderbilt University

Cygnus Load Alert Facilitates “control” aspect of load balancing Load shedding Only used for adaptive load balancing strategies Forwards client requests back to load balancer Vanderbilt University

Cygnus Load Manager Mediates interactions between all load balancer components Manages object groups Manages load monitor and load alert location maps Acts as a specialized event channel Load events are published by load monitors Load events are consumed by load analyzers Vanderbilt University

Adaptive Load Balancing Interactions Vanderbilt University

Load Balancing Strategies (1/2) Adaptive Load Balancing Strategies Use run-time system information, example host load information to make load balancing decisions. Adaptive strategies integrated in Cygnus are LeastLoaded, which allows locations to receive loads until a threshold value is reached. Once a threshold is reached , subsequent requests are transferred to the location with the lowest load. LoadMinimum, which calculates the average load at all locations and if the load at a particular location is greater than the average load and greater than the least loaded location by a certain threshold, then the load at the particular location is transferred to the least loaded location. Vanderbilt University

Load Balancing Strategies (2/2) Non-adaptive load balancing strategies Makes load balancing decisions based on some static information. Do not use run-time system state information while making load balancing decisions. Non-adaptive load balancing strategies integrated into Cygnus are RoundRobin and Random. RoundRobin strategy keeps a list of locations and iterates through that list to select the next location that will receive requests from the client session. Random strategy keeps a list of locations and selects a random member to receive the requests from the next client session. Vanderbilt University

Load Metrics Cygnus supports the following load metrics Request-per-second, which calculates the average number of requests arriving per second at each server. CPU run queue length, which returns the load as the average number of processes waiting in the OS run time queue over a configurable time period in seconds, normalized over the number of processors. CPU utilization, which returns the load as the CPU usage percentage. Vanderbilt University

Evaluating load balancing strategies Load balancing strategies can be of the type adaptive or non-adaptive Load balancing strategies can be employed various run time parameters and load metrics. Determining the appropriate load balancing strategies for different classes of distributed applications is hard without the guidance of comprehensive performance evaluation models, systematic benchmarks and empirical results. Vanderbilt University

Introducing LBPerf LBPerf is an open source benchmarking tool suite for evaluating middleware load balancing strategies. Extensible C++ middleware framework. Supports range of adaptive and non-adaptive load balancing strategies. Tune different configurations of middleware load balancing , including choosing different load balancing strategies and run-time parameters associated with those strategies. Evaluate strategies using different metrics like throughput, latency, CPU utilization. Vanderbilt University

Workload Characterization (1/2) Empirical evaluations not useful, unless the workload is representative of the context in which the load balancer is deployed. Workload model could be of the type: Closed analytical network models Simulation models Executable models Our benchmarking experiments are based on the executable workload models. Vanderbilt University

Workload Characterization (2/2) Executable models can be classified as : Resource type, where workloads are characterized by the type of resource being consumed. Service type, where the workloads can be characterized by the type of service being performed on behalf of the clients. Session type, where the workloads are characterized by the type of requests initiated by a client and serviced by a server in the context of a session. Our benchmarking experiments focused on generating session type workloads. Vanderbilt University

Experiment Single threaded clients generating CPU intensive requests on the servers Experiments repeated for different strategies including RoundRobin, Random, LoadMinimum and LeastLoaded. Measure throughput, which is the number of requests processed per second by the server and the CPU utilization, which is the CPU usage percentage. Vanderbilt University

Run-time configurations Load reporting interval, which is the time interval between successive load information updates from the server to the load balancer. Reject threshold value of the LeastLoaded strategy, which is the load at which the servers will stop receiving extra loads. Critical threshold value of the LeastLoaded strategy, which is the load at which the servers will start shedding loads. Migration threshold value of the LoadMinimum strategy, which is the difference between the most loaded machine and the least loaded machine to trigger a migration of load from the most loaded machine to the least loaded machine. Dampening value, which is the fraction of the newly reported load that will be considered for fresh load balancing decisions. Vanderbilt University

Vanderbilt University

Vanderbilt University

Vanderbilt University

Vanderbilt University

Vanderbilt University

Results Analysis Our results indicate that: Adaptive load balancing strategies perform better than non-adaptive load balancing strategies in the presence of non-uniform loads. LeastLoaded strategy is better than the other three strategies under such loading conditions Reducing the number of client migrations is necessary for achieving maximum performance. In other words, client session migrations are not always effective. Need to maintain system utilization to enable more predictive behavior of the strategies. Vanderbilt University

Future Work Extend LBPerf to evaluate other types of workloads. Use the empirical results as a learning process to understand the nuances of the different run-time behaviors of these adaptive load balancing strategies. Use observations from the learning process to train an operational phase by developing self-adaptive load balancing strategies that can dynamically tune the run-time parameters according to the load being experienced. Define stateful load balancing model Decentralized load balancing Reduced network overhead Improved reliability Integrating load balancer service into the CCM model. Vanderbilt University

Concluding Remarks Cygnus load balancing architecture and model is Flexible Extensible load balancing strategies Allows both non-adaptive and adaptive (including self-adaptive) strategies to be used Freedom to implement in a variety of ways Centralized Decentralized / Federated / Cooperative Hierarchical Generic Supports multiple object groups Not application-specific Familiar Uses many of the same group management concepts found in existing fault tolerance models Vanderbilt University

References Ossama Othman, Jaiganesh Balasubramanian, Douglas C. Schmidt “ The Design of an Adaptive Load Balancing and Monitoring Service” http://www.dre.vanderbilt.edu/~jai/IWSAS_2003.pdf Jaiganesh Balasubramanian, Ossama Othman, Douglas C. Schmidt “Evaluating the performance of middleware load balancing strategies” http://www.dre.vanderbilt.edu/~jai/EDOC_2004.pdf Download Cygnus, LBPerf and TAO from http://deuce.doc.wustl.edu/Download.html Vanderbilt University