SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. CPU Utilization Control in Distributed Real-Time Systems Chenyang Lu Department of Computer Science and Engineering

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 2 Why CPU Utilization Control?  Overload protection  CPU over-utilization  system crash  Nightmare for mission-critical applications and “always-on” E-businesses  Meet deadlines  CPU utilization < schedulable utilization bound

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 3 End-to-End Task Model in Distributed Real-Time Systems  Periodic task T i = a chain of subtasks {T ij } located on different processors  Subtasks run at a same rate  Task rate can be adjusted within a range  Higher rate  higher utility Remote Invocation Subtask T1T1 T2T2 T3T3 T 11 T 12 T 13 P1P1 P2P2 P3P3

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 4  B i : Utilization set point of processor P i (1 ≤ i ≤ n)  u i (k): Utilization of P i in k th sampling period  r j (k): Rate of task T j (1 ≤ j ≤ m) in k th sampling period subject to rate constraints: R min,j  r j (k)  R max,j (1 ≤ j ≤ m) Problem Formulation

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 5 Challenge: Uncertainties  Execution times?  Unknown sensor data or user input  Request arrival rate?  Aperiodic events  Bursty service requests  Disturbance?  Denial of Service Attacks Control-theoretic approaches to adaptive software  Robust performance in face of workload uncertainty

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 6 Single-Processor Solution: Feedback Control Real-Time Scheduling  Adaptation based on single-input-single-output control Monitor Processor OS Application Sensor Inputs Set point U s = 69% Task Rates R 1 : [1, 5] Hz R 2 : [10, 20] Hz Middleware ActuatorController u(k) {r(k+1)} FCS C. Lu, X. Wang, and C. Gill, Feedback Control Real-Time Scheduling in ORB Middleware, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS'03), May 2003.

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 7 What’s New in Distributed Systems?  Need to control utilization of multiple processors  Utilization of different processors are coupled with each other due to end-to-end tasks  Replicating FCS on all processors does not work!  Constraints on task rates T1T1 T2T2 T3T3 T 11 T 12 T 13 P1P1 P2P2 P3P3

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 8 EUCON: Multi-Input-Multi-Output Control Model Predictive Controller Distributed System (m tasks, n processors) Utilization Monitor Rate Modulator RM UM RM Feedback Loop Precedence Constraints Subtask Control Input Measured Output C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and Distributed Systems, 16(6): , June 2005.

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 9 Control Theoretic Methodology 1.Derive a dynamic model of the controlled system 2.Design a controller 3.Analyze stability

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 10 Dynamic Model: One Processor  S i : set of subtasks on P i  c jl : estimated execution time of T il running on P i  may not be correct  g i : utilization gain of P i  unknown ratio between actual and estimated change in utilization  models uncertainty in execution times

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 11 Dynamic Model: Multiple Processors  G: diagonal matrix of utilization gains  F: subtask allocation matrix  models the coupling among processors  f ij = c jl task T j has a subtask T jl on processor P i  f ij = 0 if T j has no subtask on P i u(k) = u(k-1) + GF  r(k-1) T1T1 T2T2 T 11 P1P1 P2P2 T 21 T 22 T3T3 T 31

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 12 Model Predictive Control  Advanced control technique for coupled MIMO control problems with actuator constraints.  Minimize a cost function over an interval in the future.  Compute an input trajectory that minimizes cost subject to actuator constraints.  Predict cost based on a system model and feedback.  Combines optimization, model-based prediction, and feedback.

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 13 Model Predictive Controller At a sampling instant  Compute inputs in future sampling periods  r(k),  r(k+1),...  r(k+M-1) to minimize a cost function  Cost is predicted using (1) feedback u(k-1) (2) approximate dynamic model  Apply  r(k) to the system At the next sampling instant  Shift time window and re-compute  r(k+1),  r(k+2),...  r(k+M) based on feedback

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 14 Model Predictive Controller in EUCON Difference with reference trajectory Desired trajectory for u(k) to converge to B Constrained optimization solver

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 15 Stability Analysis  Stability: system converges to equilibrium point from any initial condition  Equilibrium point = utilization set points B  If stable, utilization of all processors converge to their set points whenever feasible  Derive stability condition  tolerable range of G  tolerable variation of execution times  Stability analysis establishes analytical guarantees on utilization despite uncertainty

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 16 Simulation: Stable System execution time factor = 0.5 (actual execution times = ½ of estimates)

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 17 Simulation: Unstable System execution time factor = 7 (actual execution times = 7 times estimates)

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 18 Stability  Stability: system converges to desired utilizations from any initial condition  Derive stability condition  tolerable range of execution times Analytical assurance on utilizations despite uncertainty Overestimation of execution times prevents oscillation Predicted bound for stability actual execution time / estimation

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 19 FC-ORB Middleware Feedback lane Remote request lanes Priority Manager Rate Modulator Model Predictive Controller Remote request lanes Utilization Monitor Measured Output Control Input Priority Manager Rate Modulator Utilization Monitor Priority Manager Rate Modulator Utilization Monitor X. Wang, C. Lu and X. Koutsoukos, Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time Systems Symposium (RTSS'05), December 2005.

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 20 Workload Uncertainty time-varying execution times disturbance from periodic tasks

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 21 Processor Failure 1.Norbert fails. 2.move its tasks to other processors. 3.reconfigure controller 4.control utilization by adjusting task rates

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 22 Summary: Model Predictive Control  Robust utilization control for distributed systems  Handles coupling among processors  Enforce constraints on task rates  Analyze tolerable range of execution times

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu. 23 References  Centralized control: EUCON  C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and Distributed Systems, 16(6): , June 2005.Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks,  Decentralized control: DEUCON  X. Wang, D. Jia, C. Lu and X. Koutsoukos, DEUCON: Decentralized End-to-End Utilization Control for Distributed Real-Time Systems, IEEE Transactions on Parallel and Distributed Systems, 18(7): , July 2007.DEUCON: Decentralized End-to-End Utilization Control for Distributed Real-Time Systems  Middleware: FC-ORB  X. Wang, C. Lu and X. Koutsoukos, Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time Systems Symposium (RTSS'05), December 2005.Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control  Controllability and feasibility  X. Wang, Y. Chen, C. Lu and X. Koutsoukos, On Controllability and Feasibility of Utilization Control in Distributed Real-Time Systems, Euromicro Conference on Real-Time Systems (ECRTS'07), July 2007.On Controllability and Feasibility of Utilization Control in Distributed Real-Time Systems  Project page: