1. Systems Issues for Scalable, Fault Tolerant Internet Services
Yatin Chawathe
Eric Brewer
To appear in Middleware ’98

2. Motivation Proliferation of network-based services
Two critical issues must be addressed by Internet services:
System scalability
Incremental and linear scalability
Availability and fault tolerance
24x7 operation

3. A Reusable SNS Framework
Clusters of workstations are ideal for Internet services [FGC+97]
But, clusters are difficult to manage
To ensure linear scalability, service must distribute load across the cluster
Service must grow the cluster with increasing load
Partial failures within a cluster complicate fault management
Isolate common requirements of cluster-based
Internet apps into a reusable substrate --
the Scalable Network Services (SNS) framework

4. Architecture ... ... Outside World Worker Worker Driver Worker
Internal
Network
SNS
Manager
Worker Driver
Worker
Worker Driver
Worker
...

5. Workers
Workers are grouped into classes. Within a class, workers are identical
Workers can receive tasks from the outside world, or from other workers
Workers have a simple serial interface for tasks
The originator sends a task to the consumer by specifying the class and inputs for the task
Tasks are atomic and restartable
Worker Drivers present a narrow interface between the SNS substrate and the worker application

6. Centralized SNS Manager
SNS Manager is intentionally centralized
makes it easier to reason about and implement the various policies
“all” we need to do is ensure the fault tolerance of the manager, and make sure it is not a performance bottleneck
Three key functions
Resource location
Load balancing and scalability
Fault tolerance

7. Resource Location Persistent Connection Find Register Found
Worker
Worker Driver
Worker
Worker Driver
Persistent
Connection
Find
Register
Found
Multicast Beacons
Multicast Beacons
Multicast Beacons
SNS
Manager

8. Load Balancing Load measurement and reporting
Each worker examines incoming requests and estimates the “load” that would be generated
Simplest load metric: queue length at workers
Workers periodically report their current load to the SNS Manager
SNS Manager maintains load history and aggregates load reports from all workers
Load reports are piggybacked on manager beacons to rest of the system

9. Load Balancing Each worker performs local load balancing decisions
Use lottery scheduling -- # of tickets are inversely proportional to worker load
Stale load reports can cause oscillations
Use a correction factor based on the number of requests that were sent since last load report

10. Auto-launch for Scalability
Worker replication to handle short traffic bursts
Multiple workers handle requests in parallel
If load on a class of workers gets too high, the SNS Manager launches a new one
Overflow pool for long bursts
non-dedicated set of machines (e.g. users’ desktop machines)
when all dedicated nodes are exhausted, harness an overflow node; release it after burst subsides
useful for incremental scalability

11. Fault Tolerance Starfish Fault tolerance Two mechanisms:
“Peer” monitoring as opposed to primary/secondary fault tolerance
Two mechanisms:
Timeouts and retries
Preemptive detection and component restart
Reliance on soft state simplifies crash recovery

12. Fault Tolerance Worker Worker Driver Worker Worker Driver Worker
AmRestarting
ReRegister
SNS
Manager
SNS
Manager
SNS
Manager
SNS
Manager
SNS
Manager

13. Example Applications TranSend Wingman TopGun Mediaboard MARS
Web proxy for on-the-fly content distillation
Wingman
The world’s only graphical web browser for the 3COM PalmPilot
TopGun Mediaboard
PDA groupware: shared electronic whiteboard for the 3COM PalmPilot
MARS
MBone archive server

14. Evaluation

15. Evaluation

16. Evaluation
Workers 4
& 5started
Worker 3
started
Worker 2
started

17. Summary
Reusable architecture substrate for building Internet service applications
Application developers program their services to a well-defined narrow interface
SNS takes care of resource location, spawning, load balancing, fault tolerance
Number of interesting applications on top of the SNS substrate
Next step: SNSv NINJA