1. Reinforcement Learning Based Virtual Cluster Management
Anshul Gangwar
Dept. of Computer Science and Automation,
Indian Institute of Science

2. Virtualization: Cloud Computing:
The ability to run multiple operating systems on a single physical system and share the underlying hardware resources*.
Cloud Computing:
“The provisioning of services in a timely (near on instant), on-demand manner, to allow the scaling up and down of resources”**.
* VMware white paper, Virtualization Overview
** Alan Williamson, quoted in Cloud BootCamp March 2009

3. The Traditional Server Concept
SERVER m
SERVER n
App 1
30 %
OS 1
App 2
40 %
OS 2
App 3
25 %
OS 3
App 4
30 %
OS 4
App 5
20 %
OS 5
App m
28 %
OS m
App n
50 %
OS n
Machine provisioning done for peak demands
Processors are under utilized during off-peak hours
Wastage of resources
Need technology and algorithms that can allow allocation of only as many resources as are required
9 A.M. TO 5 P.M.
M-F
All Other
Times
Rate of Server Accesses
Time

4. Server Consolidation Process
SERVER m
SERVER n
App 1
30 %
OS 1
App 2
40 %
OS 2
App 3
25 %
OS 3
App 4
30 %
OS 4
App 5
20 %
OS 5
App m
28 %
OS m
App n
50 %
OS n
Consolidation Process
Allows shutting down of
idle PMs, saving
operational costs
SERVER 1
SERVER 2
SERVER m
VM 1
VM 2
VM 3
VM 4
VM 5
VM m
VM n
App 1 on Guest OS
30 %
App 2 on Guest OS
40 %
App 3 on Guest OS
25 %
App 4 on Guest OS
30 %
App 5 on Guest OS
20 %
App m on Guest OS
28 %
App n on Guest OS
50 %
Hypervisor
Hypervisor
Hypervisor

5. Load Distribution Hypervisor Hypervisor Hypervisor Hypervisor
VM 1
VM 2
VM 3
VM 4
VM 5
VM m
VM n
App 1 on Guest OS
30 %
App 2 on Guest OS
40 %
App 3 on Guest OS
25 %
App 4 on Guest OS
30 %
App 5 on Guest OS
20 %
App m on Guest OS
28 %
App n on Guest OS
50 %
Hypervisor
Hypervisor
Hypervisor
SERVER 1
SERVER 2
SERVER m
VM 1
VM 2
VM 3
VM 4
VM 5
VM m
VM n
App 1 on Guest OS
30 %
App 2 on Guest OS
20 %
App 3 on Guest OS
45 %
App 4 on Guest OS
30 %
App 5 on Guest OS
25 %
App m on Guest OS
28 %
App n on Guest OS
50 %
Hypervisor
Hypervisor 5
Hypervisor

6. Live Migration Migrate VM 5 Hypervisor Hypervisor Hypervisor
VM m
VM n
App 1 on Guest OS
30 %
App 2 on Guest OS
20 %
App 3 on Guest OS
45 %
App 4 on Guest OS
30 %
App 5 on Guest OS
25 %
App m on Guest OS
28 %
App n on Guest OS
50 %
Hypervisor
Hypervisor 6
Hypervisor
SERVER 1
SERVER 2
SERVER m
Migrate VM 5
SERVER 1
SERVER 2
SERVER m
VM 1
VM 2
VM 5
VM 3
VM 4
VM m
VM n
App 1 on Guest OS
30 %
App 2 on Guest OS
20 %
App 5 on Guest OS %
App 3 on Guest OS
45 %
App 4 on Guest OS
30 %
App m on Guest OS
28 %
App n on Guest OS
50 %
Hypervisor 6
Hypervisor
Hypervisor

7. Dynamic Resource Allocation
Dynamic Workload requires Dynamic Resource Management
Allocation of resources to VMs in each PM
Resources such as CPU, memory etc.
Allocation of VMs to PMs
Minimize number of operational PMs
Modern VMs (e.g. Xen) allow
Resource allocation within PM
Dynamic allocation of VM to PM through Live Migration
Required:
Architecture and mechanisms for
Determining resource allocation to VM within PM
Determining deployment of VMs on PMs
So that:
Capital and Operational costs are minimized
Application Performance is maximized

8. Two Level Controller Architecture
PMA : Performance Measurement Agent
RAC : Resource Allocation Controller
VM Placement Controller
RAC Determined Resource Requirements
Performance Measures
Migration Decisions PM
PMA
RAC PM
PMA
RAC
PMA
RAC PM VM VM VM VM VM VM
Note: PMs(Physical Machines/servers) are assumed to be homogeneous.

9. Problem Definition
VM Placement Controller has to make optimal migration decisions at regular intervals which results in
Low SLA Violations
Reduction in number of busy PMs
SLA(Service Level Agreement) are Performance Guaranties which Data Center Owner negotiates with User. These Performance Guaranties can include average response time, maximum delay, maximum downtime etc.
Idle PMs can be switched off/run in low power mode

10. Issues in Server Consolidation/ Distribution
There are various issues involved in Server Consolidation
Interference of VMs : Bad behaviors of one application in a VM adversely affect(degraded performance) the other VMs on same PM
Delayed Effects : Resource configurations of a VM show effects after some delay
Migration Cost : Live Migrations involves cost(performance degradation)
Workload Pattern : is not deterministic or known apriori
These difficulties motivates us to use Reinforcement Learning Approach

11. Reinforcement Learning(RL)
The agent-environment interaction in RL
The goal of the agent is to maximize the cumulative long term
reward based on the immediate reward rn+1.

12. Reinforcement Learning(RL)
The agent-environment interaction in RL
The goal of the agent is to maximize the cumulative long term
reward based on the immediate reward rn+1.
RL has two major benefits
Doesn't requires model of the system
Capture delayed effects in decision making
Can take action before problem arises

13. Problem Formulation for RL Framework: System Assumptions
M PMs: Assumed to be homogeneous
N VMs: Each assumed to be running one application whose performance metrics of interest are throughput and response time
Response time implied at server level only (not as seen by user)
Workload per VM is assumed to be cyclic
Resource requirement assumed to be equal to workload
Time Period
Phase 2
Cyclic Workload model:
Time period assumed to be divided
Into phases
Rate of Server Accesses
Phase 1
Time

14. Problem Formulation for RL Framework
N VMs , M PMs , M > 1 and P Phases in Cyclic Workload Models
State sn at time n is (Phase of Cyclic Workload Models , allocation vector of VMs)
Action an at time n is (VM id which is migrating , to PM id which it is migrating)
Reward r(sn,an) at time n is defined as
r( 𝒔 𝒏 ; 𝒂 𝒏 ) = 𝒔𝒄𝒐𝒓𝒆 𝒏+𝟏 −𝒑𝒐𝒘𝒆𝒓𝒄𝒐𝒔𝒕 𝒏+𝟏 𝒊𝒇 𝒔𝒄𝒐𝒓𝒆 𝒊 >𝟎;𝒊=𝟏,…,𝑵, −𝟏 𝒐𝒕𝒉𝒆𝒓𝒘𝒊𝒔𝒆 ;
𝒑𝒐𝒘𝒆𝒓𝒄𝒐𝒔𝒕 𝒏 =𝒃𝒖𝒔𝒚 𝒏 ∗𝒇𝒊𝒙𝒆𝒅𝒄𝒐𝒔𝒕 ;
𝒔𝒄𝒐𝒓𝒆(𝒏)= 𝑵 𝒊=𝟏 𝑵 𝒘𝒆𝒊𝒈𝒉𝒕 𝒊 ∗ 𝒔𝒄𝒐𝒓𝒆 𝒊 (𝒏) ;
𝒔𝒄𝒐𝒓𝒆 𝒊 𝐧 = 𝑻𝒉𝒓𝒐𝒖𝒈𝒉𝒑𝒖𝒕 𝒐𝒇 𝑽𝑴 𝒊(𝒏) 𝑨𝒓𝒓𝒊𝒗𝒂𝒍 𝑹𝒂𝒕𝒆 𝒐𝒇 𝑽𝑴 𝒊(𝒏) 𝒊𝒇 𝒓𝒆𝒔𝒑𝒐𝒏𝒔𝒆 𝒕𝒊𝒎𝒆 𝒊 ≤ 𝑺𝑳𝑨 𝒊 −𝟏 𝒐𝒕𝒉𝒆𝒓𝒘𝒊𝒔𝒆 ;
𝑺𝑳𝑨 𝒊 is maximum allowable response time for VM i

15. RL Agent implemented in CloudSim
CloudSim is a Java based Simulation tool for Cloud Computing
We have implemented the following additions to it
Response Time and Throughput calculations
Cyclic Workload Model
Interference Model
RL Agent which takes Migration Decisions
Implements Q-Learning with ε non-greedy policy
Full State Representation without batch updates
Full State Representation with batch updates of batch size 200
Implements Q-Learning with Function Approximation and ε non-greedy policy
Function Approximation without batch updates
Function Approximation with batch updates of batch size 200
Used CloudSim for implementation of all our algorithms

16. Workload Model Graphs for Experiments
Workload Model 1(wm2)
Workload Model 2(wm2)
Graphs shows the cyclic workload graphs with 5 phases which repeats itself periodically

17. Experiment Setup
5 VMs, 3 PMs and 5 Phases in cyclic workload model (shown in previous slide)
Migration decisions are taken at end of Phase
Experiments:
One VM have workload model wm1 and others have workload model wm2
Two VMs have workload model wm1 and others have workload model wm2
For each Experiment (1) and (2), following scenarios
All costs are negligible except power
High interference cost with VM 4 and 5 interfering(high performance degradation due to interference of VMs)
High migration cost with VM 4 and 5 interfering (high performance degradation due to VM migration)
High migration cost and Interference cost

18. Policy Generated after Convergence
Initial state is all VMs on PM 1 and Phase 1
i.e. (1;((1,2,3,4,5),(),()))
All costs are negligible except power
VM 1
VM 2
VM 3
VM 4
VM 5
0.5
utilization
Migrate VM 1 to PM 1
Migrate VM 1 to PM 1
0.1
Migrate VM 1 to PM 2
Phase
Migrate VM 1 to PM 2

19. Policy Generated after Convergence
Initial state is all VMs on PM 1 and Phase 1
i.e. (1;((1,2,3,4,5),(),()))
High migration cost and Interference cost
VM 3 and 4 are interfering
VM 1
VM 2
VM 3
VM 4
VM 5
0.5
utilization
0.1
Migrate VM 1
Phase

20. Results with Full State Representation
Algorithm verified to converge most of the times in steps in case 1 and 80 steps in case 2
Algorithm verified to converge every time in steps in case 1 and 115 steps in case 2.

21. Features Based Function Approximation
Full state representation leads to state space explosion
10 PMs , 10 VMs , 5 Phases results in number of states to be in order of 1011
Will require huge memory and long time to converge to a good policy
Need approximation methods
Approximate 𝑄 𝑠;𝑎 as 𝑄 𝑠;𝑎 ≈ 𝜃 𝑇 σ s;a where σ s;a is a d-dimensional feature (column vector) corresponding to the state-action tuple 𝑠;𝑎 and 𝜃 is a d-dimensional tunable parameter

22. Features for Function Approximation
Let there be 5 VMs, 3 PMs and 2 Phases in Cyclic Workload Model
State = [1,(1,2,3),(4),(5)] and Action = (4,3)
Next State allocation = [1,(1,2,3),(),(4,5)]
Phase Indicator of Cyclic Workload Model (utilization level)
Pairwise Indicator whether VMs (i,j) allocated on same PM in
Next State(interference)
Fraction of Busy PMs in
Next State
(Power Savings)
(1,2) (1,3) (1,4) (3,5) (4,5) …
total k features

23. Features for Function Approximation
start index
Migrate VM 1
Migrate VM 2
Migrate VM 4
No Migration f1 f2 f4 f6
… … … … … …
k features
k features
k features
k features
Position of fi features captures the migration cost for migrating VM i
Features except f4 are zero vectors
Store only f4 features and its start index
Perform multiplication and addition operations only for k features starting from start index
k features are corresponding to three bullet points on previous slide
Above idea reduces number of multiplication and addition operations by around five times the number of VMs

24. Results with Function Approximation
Feature based Q-Learning with Function Approximation Algorithm
Features are found to be non-differentiating with some state-action (s;a) tuples. For Example
Consider state-action tuples
((5;(1,2,3,4),(5),()); (1,2)) and ((5;(2,3,4),(1,5),()); (1,1))
Above state-action tuples are differentiated by pairwise indicators only
In first case pairwise indicators (1,5) and in second case (1,2);(1,3);(1,4)
Clearly from next slide action (1,2) is good for state (5;(1,2,3,4),(5),())
while action (1,1) is bad for state (5;(2,3,4),(1,5),())
Which implies pair (1,3) and (1,4) is bad allocation and pair (1,5) is
good but they are equivalent in deployment

25. Optimal Policy Initial state is all VMs on PM 1 and Phase 1 VM 1
i.e. (1;((1,2,3,4,5),(),()))
High migration cost and Interference cost
VM 3 and 4 are interfering
VM 1
VM 2
VM 3
VM 4
VM 5
0.5
utilization
0.1
Migrate VM 1
Phase

26. Conclusion and Future Work
We conclude with this project that
RL Algorithm with Full State Representation works very well but has problem of huge state space
For these features to work for Function Approximation, we have to add more features for interference of VMs
Which results in huge Feature Set
Same problem as before
Future work would involve following three issues
Features must be able to well differentiate between (s;a) tuple
Fast Convergence of algorithm
Scalability of Algorithm

27. References References
Virtualization and Cloud Computing . Norman Wilde. Thomas Huber
VCONF : A Reinforcement Learning Approach to Virtual Machines Auto-Configuration
L A Prashanth and Shalabh Bhatnagar. Reinforcement learning with function approximation for trafic signal control.

28. Thank You !