1. Integrated Resource Management for Cluster-based Internet Services
Shen, Tang, Yang, and Chu
11/27/2018
Integrated Resource Management for Cluster-based Internet Services
Kai Shen
Dept. of Computer Science
Univ. of Rochester
Hong Tang, Tao Yang*, Lingkun Chu
Dept. of Computer Science
Univ. of California, Santa Barbara
*: Ask Jeeves, Inc.
OSDI 2002

2. Shen, Tang, Yang, and Chu
11/27/2018
Background
Large-scale resource-intensive Internet services hosted on server clusters.
Yahoo, MSN, Google, Teoma/Ask Jeeves …
Challenges/requirements for resource management:
Scalability and robustness;
Online users require interactive responses;
Resource (CPU, IO)–hungry service processing and large user traffic require efficient resource utilization;
Fluctuating user traffic requires adaptive management;
Supporting differentiated services to different types of user requests.
11/27/2018
OSDI 2002
OSDI 2002

3. Architecture of Targeted Services: Document Search Engine
Shen, Tang, Yang, and Chu
11/27/2018
Architecture of Targeted Services: Document Search Engine
Index servers
(partition 1)
Query
caches
Firewall/
Web switch
Local-area
network
Index servers
(partition 2)
Web server/
Query handlers
Index servers
(partition 3)
Doc servers
11/27/2018
OSDI 2002
OSDI 2002

4. “Neptune” Project Overview
Shen, Tang, Yang, and Chu
11/27/2018
“Neptune” Project Overview
Programming and runtime support to aggregate and replicate stand-alone service components.
Building blocks for scalable and robust service constructions:
Functionally-symmetric clustering architecture;
Integrated resource management – quality, efficiency, and differentiation;
Replication management.
11/27/2018
OSDI 2002
OSDI 2002

5. Architecture of Targeted Services: Document Search Engine
Shen, Tang, Yang, and Chu
11/27/2018
Architecture of Targeted Services: Document Search Engine
Index servers
(partition 1)
Query
cache
Neptune runtime
SAP
Neptune runtime
SAP
Firewall/
Web switch
Local-area
network
Index servers
(partition 2)
Web server/
Query handlers
Index servers
(partition 3)
Doc servers
11/27/2018
OSDI 2002
OSDI 2002

6. Neptune Deployments Service deployments:
Web document searching;
BLAST – protein sequence similarity matching;
Prototype database services – online discussion group, auction.
Production system at search engines Teoma/Ask Jeeves since 2000:
search indexes of more than 450M Web documents;
over 800 multiprocessor servers;
tens of millions of search queries per day.
11/27/2018
OSDI 2002

7. Outline Project Overview Integrated Resource Management
Shen, Tang, Yang, and Chu
11/27/2018
Outline
Project Overview
Integrated Resource Management
Multiple Resource Management Objectives
Two-level Mechanism
Trace-driven Performance Evaluation on a Linux Cluster
Related Work and the Conclusion
11/27/2018
OSDI 2002
OSDI 2002

8. Quality-aware Resource Utilization Efficiency
Shen, Tang, Yang, and Chu
11/27/2018
Quality-aware Resource Utilization Efficiency
Throughput: measure resource utilization efficiency.
Service response time: measure client-perceived service quality.
Aggregate service yield: measure quality-aware resource utilization efficiency.
Fulfillment of each service request generates quality-aware service yield – a function of service response time.
Service yield function – specified by service providers (flexibility).
System goal – maximizing aggregate service yield:
11/27/2018
OSDI 2002
OSDI 2002

9. Sample Service Yield Functions
Shen, Tang, Yang, and Chu
11/27/2018
Sample Service Yield Functions
Response
time
Deadline
Constant
yield
<A> Maximizing throughput
(with a deadline)
Service yield
QoS yield
QoS yield
Deadline
Full
yield
Response
time
<B> Minimizing mean response time
(with a deadline)
Service yield
Full-yield
deadline
Deadline
Drop
penalty
Full
yield
Response
time
<C> A hybrid metric
Service yield
QoS yield
11/27/2018
OSDI 2002
OSDI 2002

10. Service Differentiation
Shen, Tang, Yang, and Chu
11/27/2018
Service Differentiation
Service class – a category of service accesses that enjoy the same level of QoS support.
Client identities: paid vs unpaid, consumers vs corporate partners.
Service types or data partitions: order placement vs catalog browsing.
Service differentiation in Neptune
Differentiated service yield function.
Proportional resource allocation guarantee.
11/27/2018
OSDI 2002
OSDI 2002

11. Two-level Resource Management
Shen, Tang, Yang, and Chu
11/27/2018
Two-level Resource Management
11/27/2018
OSDI 2002
OSDI 2002

12. Cluster-level: Partitioning or Not?
Shen, Tang, Yang, and Chu
11/27/2018
Cluster-level: Partitioning or Not?
Periodic Server Partitioning [Zhu2001]:
Determine resource allocation at each epoch.
Partition the server pool among service classes.
Neptune – does not partition servers at cluster-level:
Random polling-based load balancing to evenly distribute requests for each service class to all nodes  service differentiation inside each node.
Advantages:
Functional-symmetry and decentralization  robustness and scalability.
Better handling of system state changes: demand spikes and node failures.
Disadvantage:
Less isolation for misbehaved service classes.
11/27/2018
OSDI 2002
OSDI 2002

13. Node-level Request Scheduling
Shen, Tang, Yang, and Chu
11/27/2018
Node-level Request Scheduling
Drop requests likely
generating zero yield
Search for
under-allocated
service class
Found ?
Schedule the
Yes
Schedule for
high aggregate yield No
11/27/2018
OSDI 2002
OSDI 2002

14. Scheduling for High Aggregate Yield
Shen, Tang, Yang, and Chu
11/27/2018
Scheduling for High Aggregate Yield
Offline optimal scheduling is NP-complete.
Policy
Priority (the smaller the higher)
EDF
Relative deadline;
YID
Relative deadline divided by expected yield;
Greedy
Expected resource consumption divided by expected yield;
Adaptive
Dynamically switch between YID (in under-load) and Greedy (in overload).
11/27/2018
OSDI 2002
OSDI 2002

15. Evaluation Settings Evaluation platform Workload I: trace-driven
Shen, Tang, Yang, and Chu
11/27/2018
Evaluation Settings
Evaluation platform
A cluster of Linux servers connected by switched Ethernet.
Workload I: trace-driven
Document search on a 2.5GB memory-mapped search index.
Based on 1.5M search queries selected from an one-week access trace at Ask Jeeves search in January 2002.
“Service yield”-based priority order: Gold > Silver > Bronze.
Workload II:
CPU-spinning micro-benchmark.
Poisson process arrival; exponentially-distributed service processing time.
QoS yield
11/27/2018
OSDI 2002
OSDI 2002

16. Evaluation on Scheduling Policies (16 nodes aggregate)
Shen, Tang, Yang, and Chu
11/27/2018
Evaluation on Scheduling Policies (16 nodes aggregate)
Performance Metric:
(A) Underload
(B) Overload 6%
EDF
60%
YID
Loss percent
Greedy
Loss percent
Adaptive
45% 4%
30%
EDF
Lost percent 2%
Lost percent
YID
15%
Greedy
Adaptive
Aggregated yield (normalized)
Aggregated yield (normalized)
Aggregated yield (normalized)
Aggregated yield (normalized) 0% 0%
Aggregated yield (normalized) 0%
25%
50%
75%
100%
Aggregated yield (normalized)
100%
125%
150%
175%
200%
Arrival demand
Arrival demand
EDF and YID perform better than Greedy during system under-load; Greedy performs better during system overload.
Adaptive dynamically switches between YID and Greedy to achieve good performance under both situations.
11/27/2018
OSDI 2002
OSDI 2002

17. Shen, Tang, Yang, and Chu
11/27/2018
Service Differentiation during a Demand Spike and a Node Failure (8 nodes)
Bronze demand
In percentage to total system resource
CPU demand/acquisition
Bronze acquisition
Silver demand
100%
Silver acquisition
Gold demand
Gold acquisition
80%
60%
40%
20%
Resource demand/acquisition
Resource demand/acquisition 0% 50
100
150
200
250
300
Timeline (seconds)
“Service yield”-based priority order: Gold > Silver > Bronze.
20% proportional resource guarantee for low-priority Bronze class.
Demand spike for the Silver class between time 50 and 150.
One node fails at time 200 and recovers at 250.
11/27/2018
OSDI 2002
OSDI 2002

18. Performance Scalability
Shen, Tang, Yang, and Chu
11/27/2018
Performance Scalability
<A> Differentiated Search
<B> Micro-benchmark 20 20
Aggregated yield (normalized)
Aggregated yield (normalized)
Demand 200%
Demand 200%
Demand 125%
Demand 125% 15
Demand 75% 15
Demand 75% 10 10 5 5
Aggregate yield (normalized)
Aggregate yield (normalized) 5 10 15 20 5 10 15 20
Number of service nodes
Number of service nodes
11/27/2018
OSDI 2002
OSDI 2002

19. Shen, Tang, Yang, and Chu
11/27/2018
Related Work
Software infrastructure for cluster-based Internet services – TACC [Fox1997], MultiSpace [Gribble1999], Porcupine [Saito1999], Ninja [von Behren2002].
QoS and service differentiation in computer networks – Weighted Fair Queuing [Demers1990; Parekh1993], Leaky Bucket, LIRA [Stoica1998], [Dovrolis1999].
QoS or real-time scheduling at the single host level – [Huang1989], [Haritsa1993], [Waldspurger1994], [Mogul1996], LRP [Druschel96], [Jones97], Eclipse [Bruno1998], Resource Container [Banga1999], [Steere1999].
Resource management and QoS for Web servers – [Almeida1998], [Pandey1998], [Abdelzaher1999], [Bhatti1999], [Chandra2000], [Li2000], [Voigt2001].
Resource management for clustered servers – LARD [Pai1998], Cluster Reserves [Aron2000], [Sullivan2000], DDSD [Zhu2001], [Chase2001].
11/27/2018
OSDI 2002
OSDI 2002

20. Conclusion Multiple resource management objectives:
Shen, Tang, Yang, and Chu
11/27/2018
Conclusion
Multiple resource management objectives:
quality-aware resource utilization efficiency
service differentiation
Two-level resource management mechanism:
non-partitioning at the cluster level
adaptive scheduling at the node level
Trace-driven evaluations.
Future work – other types of service qualities.
11/27/2018
OSDI 2002
OSDI 2002