1. Budget-based Control for Interactive Services with Partial Execution 1 Yuxiong He, Zihao Ye, Qiang Fu, Sameh Elnikety Microsoft Research

2. Motivation Interactive services specify stringent SLA on response time Long response time causes user dissatisfaction and revenue loss Important to bound response time (e.g. mean, 95-percentile) Address two challenges Adapt to dynamic and changing environment Achieve high response quality G OAL : Develop a self-managed scheduling system to meet response time target while achieving high quality. 2

3. Existing Techniques (1) Static admission control approach – Define a fixed queue length limit; drop requests when queue is full. Issues – Only works under a static system. – Determining an appropriate queue-length for every setting and load is challenging. Small queue length => underutilize resources Large queue length => long response time – Can not adapt to dynamic and changing environment. 3

4. Existing Techniques (2) Classic feedback control approach: – Feedback control on queue length Decrease queue length when response time is above target – Issue Dropping requests results in degraded quality Does not consider partial execution of requests 4

5. Partial Execution & Response Quality Incomplete execution of requests may still return meaningful partial results Many interactive services support partial execution – Web search, web server, video streaming, finance server Quality profile – A function maps request execution time to response quality 5

6. Our Contributions Propose a budget-based control model for interactive services with partial execution – Use feedback control to meet response time target – Apply optimization procedure to improve response quality Exploit partial execution and request quality profile Evaluation – Implementation at Bing search server – Simulation on finance server 6

7. Budget-based Control Model Control Variable – Budget: amount of computation time for all pending requests Control mechanism – Determine the budget based on response time feedback – Control budget to meet response time Optimization procedure – Given a budget, assign processing time to requests – Exploit partial results of a query – Scheduling to improve quality 7

8. Control Mechanism Basic idea – If response time is larger than target, smaller budget – If response time is smaller than target, larger budget Criteria – Meet response time target accurately and quickly – Incur little runtime overhead. 8

9. Control Mechanism: Background Integral control – Adjust budget based on the difference between the observed and target response time – Advantage: eliminate steady-state error – Limitation: response is slow (long settling time) Adaptive control – Model estimator + Linear quadratic optimal controller – Advantage: quick adaptation, fast response – Limitation: computationally expensive, stead-state error 9

10. Control Mechanism: Hybrid Control Combine the integral and adaptive control Run adaptive control periodically in a coarse- grain time interval Use integral control for execution of each request for fine-grain adjustment Meet our goal – Quick and accurate adaptation – Little runtime overhead. 10

11. Optimization Procedure Objective: maximize total response quality Input: budget, pending requests Output: assigned processing time to requests Optimization procedure depends on applications

12. Bing index server Core part of Bing search – For a user query, match and rank docs, return top results Concave quality profile – First-half of request execution receives higher quality gain than the second half. 12

13. Run the portion of requests with higher gain Prevent long requests from starving short ones Combine two techniques – Reservation at light load: Reserve time for later requests in the queue based on mean service demand – Equal sharing at heavy load: Allocate resource equally among requests Optimization Procedure for Index Server 13

14. Evaluation Implemented and evaluated at Bing index server – Meet response time target and achieve high quality Simulation study on finance server – Double system throughput at desired quality 14

15. Bing Index Server Implementations – BudgetIS Feedback control on budget Hybrid control + optimization procedure – QueueIS Feedback control on queue length Evaluation – Production trace 15

16. Compare Queue v.s. Budget Approach Mean response time = 35ms Budget approach Meet response time accurately Achieve high quality 16

17. Conclusion Propose a budget-based control optimization model for interactive services with partial execution – Hybrid control mechanism to meet response time target – Optimization procedure to improve response quality Evaluation – Implemented and evaluated at Bing index server Meet response time target and achieve high quality – Simulation study on finance server Double system throughput at desired quality 17

18. Thank you & Questions? 18