1. Lecture 18: Scalable Web Services
COSC6376 Cloud Computing
Lecture 18: Scalable Web Services
Instructor: Weidong Shi (Larry), PhD
Computer Science Department
University of Houston

2. Outline
Scalable services

3. Building scalable web services
A relatively easy problem?
Why?
HTTP: stateless, request-response protocol
decoupled, independent requests
How?
divide and conquer
replicate, partition, distribute, load balance
In the end you want to build a Web 2.0 app that can serve millions of users with ZERO downtime

4. The Variables
Scalability - Number of users / sessions / transactions / operations the entire system can perform
Performance – Optimal utilization of resources
Responsiveness – Time taken per operation
Availability - Probability of the application or a portion of the application being available at any given point in time
Downtime Impact - The impact of a downtime of a server/service/resource - number of users, type of impact etc
Cost
Maintenance Effort
High: scalability, availability, performance & responsiveness Low: downtime impact, cost & maintenance effort
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

5. The Factors Platform selection Hardware Application Design
Database/Datastore Structure and Architecture
Deployment Architecture
Storage Architecture
Abuse prevention
Monitoring mechanisms
… and more
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

6. Lets Start …
We will now build an example architecture for an example app using the following iterative incremental steps –
Inspect current Architecture
Identify Scalability Bottlenecks
Identify SPOFs and Availability Issues
Identify Downtime Impact Risk Zones
Apply one of -
Vertical Scaling
Vertical Partitioning
Horizontal Scaling
Horizontal Partitioning
Repeat process
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

7. Step 1 – Lets Start … Appserver & DBServer
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

8. Step 2 – Vertical Scaling
Appserver, DBServer
CPU
CPU
RAM
RAM
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

9. Step 2 - Vertical Scaling
Introduction
Increasing the hardware resources without changing the number of nodes
Referred to as “Scaling up” the Server
Advantages
Simple to implement
Disadvantages
Finite limit
Hardware does not scale linearly (diminishing returns for each incremental unit)
Requires downtime
Increases downtime Impact
Incremental costs increase exponentially
Appserver, DBServer
CPU
CPU
CPU
CPU
RAM
RAM
RAM
RAM
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

10. Step 3 – Vertical Partitioning (Services)
Introduction
Deploying each service on a separate node
Positives
Increases per application Availability
Task-based specialization, optimization and tuning possible
better cache performance
Reduces context switching
No changes to App required
Flexibility increases
AppServer
DBServer
Example
mail.blah.com
images.blah.com
shopping.blah.com
my.blah.com
etc. etc.
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

11. Vertical Partitioning (Services)
Disadvantages
lower peak capacity
sub-optimal resource utilization
coarse load balancing across servers/services
finite Scalability
management costs
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

12. Understanding Vertical Partitioning
The term Vertical Partitioning denotes –
Increase in the number of nodes by distributing the tasks/functions
Each node (or cluster) performs separate Tasks
Each node (or cluster) is different from the other
Vertical Partitioning can be performed at various layers (App / Server / Data / Hardware etc)
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

13. Step 4 – Horizontal Scaling (App Server)
Introduction
Increasing the number of nodes of the App Server through Load Balancing
Referred to as “Scaling out” the App Server
Load Balancer
AppServer
AppServer
AppServer
DBServer
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

14. Understanding Horizontal Scaling
The term Horizontal Scaling denotes –
Increase in the number of nodes by replicating the nodes
Each node performs the same Tasks
Each node is identical
Typically the collection of nodes maybe known as a cluster
Also referred to as “Scaling Out”
Horizontal Scaling can be performed for any particular type of node (AppServer / DBServer etc)
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

15. Load Balancer – Hardware vs Software
Hardware Load balancers are faster
Software Load balancers are more customizable
With HTTP Servers load balancing is typically combined with http accelerators
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

16. Load Balancer – Session Management
Sticky Sessions
Sticky Sessions
Requests for a given user are sent to a fixed App Server
Observations
Asymmetrical load distribution
Downtime Impact – Loss of session data
User 1
User 2
Load Balancer
AppServer
AppServer
AppServer
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

17. Load Balancer – Session Management
Central Session Storage
Central Session Storage
Central Session Store
Introduces SPOF
An additional variable
Session reads and writes generate Disk + Network I/O
Also known as a Shared Session Store Cluster
Load Balancer
AppServer
AppServer
AppServer
Session Store
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

18. Load Balancer – Session Management
Clustered Session Management
Easier to setup
No SPOF
Network I/O increases exponentially with increase in number of nodes
In very rare circumstances a request may get stale session data
User request reaches subsequent node faster than intra-node message
Intra-node communication fails
Clustered Session Management
Load Balancer
AppServer
AppServer
AppServer
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

19. Load Balancer – Session Management
Recommendation
Use Clustered Session Management if you have –
Smaller Number of App Servers
Fewer Session writes
Use a Central Session Store elsewhere
Use sticky sessions only if you have to
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

20. Load Balancer – Removing SPOF
Active-Passive LB
In a Load Balanced App Server Cluster the LB is an SPOF
Active-Active nevertheless assumes that each LB is independently able to take up the load of the other
If one wants ZERO downtime, then Active-Active becomes truly cost beneficial only if multiple LBs (more than 3 to 4) are daisy chained as Active-Active forming an LB Cluster
Users
Load Balancer
Load Balancer
AppServer
AppServer
AppServer
Active-Active LB
Users
Load Balancer
Load Balancer
AppServer
AppServer
AppServer
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

21. Step 4 – Horizontal Scaling (App Server)
At the end of Step 4
Positives
Increases Availability and Scalability
No changes to App required
Easy setup
Negatives
Finite Scalability
Load Balanced App Servers
DBServer
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

22. Step 5 – Vertical Partitioning (Hardware)
Introduction
Partitioning out the Storage function using a SAN
Positives
Allows “Scaling Up” the DB Server
Boosts Performance of DB Server
Negatives
Increases Cost
Load Balanced App Servers
DBServer
SAN
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

23. Step 6 – Horizontal Scaling (DB)
Introduction
Increasing the number of DB nodes
Referred to as “Scaling out” the DB Server
Options
Shared nothing Cluster
Real Application Cluster (or Shared Storage Cluster)
Load Balanced App Servers
DBServer
DBServer
DBServer
SAN
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

24. Shared Nothing Cluster
Each DB Server node has its own complete copy of the database
Nothing is shared between the DB Server Nodes
This is achieved through DB Replication at DB / Driver / App level or through a proxy
Supported by most database software natively or through 3rd party software
DBServer
DBServer
DBServer
Database
Database
Database
Note: Actual DB files maybe stored on a central SAN
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

25. Replication Considerations
Master-Slave
Writes are sent to a single master which replicates the data to multiple slave nodes
Replication maybe cascaded
Simple setup
No conflict management required
Multi-Master
Writes can be sent to any of the multiple masters which replicate them to other masters and slaves
Conflict Management required
Deadlocks possible if same data is simultaneously modified at multiple places
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

26. Replication Considerations
Asynchronous
Guaranteed, but out-of-band replication from Master to Slave
Master updates its own db and returns a response to client
Replication from Master to Slave takes place asynchronously
Faster response to a client
Slave data is marginally behind the Master
Requires modification to App to send critical reads and writes to master, and load balance all other reads
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

27. Replication Considerations
Synchronous
Guaranteed, in-band replication from Master to Slave
Master updates its own db, and confirms all slaves have updated their db before returning a response to client
Slower response to a client
Slaves have the same data as the Master at all times
Requires modification to App to send writes to master and load balance all reads
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

28. Replication Considerations
Replication at RDBMS level
Support may exists in RDBMS or through 3rd party tool
Faster and more reliable
App must send writes to Master, reads to any db and critical reads to Master
Replication at Driver level
Driver layer ensures
writes are performed on all connected DBs
Reads are load balanced
Critical reads are sent to a Master
In most cases RDBMS agnostic
Slower and in some cases less reliable
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

29. Real Application Cluster
All DB Servers in the cluster share a common storage area on a SAN
All DB servers mount the same block device
The filesystem must be a clustered file system (eg GFS)
Currently only supported by Oracle Real Application Cluster
Can be very expensive (licensing fees)
DBServer
DBServer
DBServer
SAN
Database
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

30. Load Balanced App Servers
Recommendation
Try and choose a DB which natively supports Master- Slave replication
Use Master-Slave Async replication
Write your layer to ensure
writes are sent to a single DB
reads are load balanced
Critical reads are sent to a master
Load Balanced App Servers
DBServer
DBServer
DBServer
Writes & Critical Reads
Other Reads
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

31. Step 6 – Horizontal Scaling (DB)
Our architecture now looks like this
Positives
As Web servers grow, Database nodes can be added
DB Server is no longer SPOF
Negatives
Finite limit
Load Balanced App Servers
DB Cluster DB
SAN
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

32. Step 7 – Vertical / Horizontal Partitioning (DB)
Introduction
Increasing the number of DB Clusters by dividing the data
Options
Vertical Partitioning - Dividing tables / columns
Horizontal Partitioning - Dividing by rows (value)
Load Balanced App Servers
DB Cluster DB
SAN
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

33. Vertical Partitioning (DB)
Take a set of tables and move them onto another DB
Eg in a social network - the users table and the friends table can be on separate DB clusters
Each DB Cluster has different tables
Application code or Driver code or a proxy knows where a given table is and directs queries to the appropriate DB
Can also be done at a column level by moving a set of columns into a separate table
App Cluster
DB Cluster 1
Table 1
Table 2
DB Cluster 2
Table 3
Table 4
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

34. Vertical Partitioning (DB)
Negatives
One cannot perform SQL joins or maintain referential integrity
Finite Limit
App Cluster
DB Cluster 1
Table 1
Table 2
DB Cluster 2
Table 3
Table 4
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

35. Horizontal Partitioning (DB)
Take a set of rows and move them onto another DB
Eg in a social network – each DB Cluster can contain all data for 1 million users
Each DB Cluster has identical tables
Application code or Driver code or a proxy knows where a given row is and directs queries to the appropriate DB
Negatives
SQL unions for search type queries must be performed within code
App Cluster
DB Cluster 1
Table 1
Table 2
Table 3
Table 4
DB Cluster 2
Table 1
Table 2
Table 3
Table 4
1 million users
1 million users
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

36. Horizontal Partitioning (DB)
Techniques
FCFS
1st million users are stored on cluster 1 and the next on cluster 2
Round Robin
Least Used (Balanced)
Each time a new user is added, a DB cluster with the least users is chosen
Hash based
A hashing function is used to determine the DB Cluster in which the user data should be inserted
Value Based
User ids 1 to 1 million stored in cluster 1 OR
all users with names starting from A-M on cluster 1
Except for Hash and Value based all other techniques also require an independent lookup map – mapping user to Database Cluster
This map itself will be stored on a separate DB (which may further need to be replicated)
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

37. Step 7 – Vertical / Horizontal Partitioning (DB)
Our architecture now looks like this
Positives
As App servers grow, Database Clusters can be added
Note: This is not the same as table partitioning provided by the db (eg MSSQL)
We may actually want to further segregate these into Sets, each serving a collection of users (refer next slide
Load Balanced App Servers
Lookup
Map
DB Cluster DB
DB Cluster DB
SAN
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

38. Step 8 – Separating Sets
Now we consider each deployment as a single Set serving a collection of users
Global
Lookup
Map
Global Redirector
Load Balanced App Servers
Load Balanced App Servers
Lookup
Map
Lookup
Map
DB Cluster DB
DB Cluster DB
DB Cluster DB
DB Cluster DB
SAN
SAN
SET 1 – 10 million users
SET 2 – 10 million users
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

39. Creating Sets The goal behind creating sets is easier manageability
Each Set is independent and handles transactions for a set of users
Each Set is architecturally identical to the other
Each Set contains the entire application with all its data structures
Sets can even be deployed in separate datacenters
Users may even be added to a Set that is closer to them in terms of network latency
Building a Scalable Architecture for Web Apps. Bhavin Turakhia

40. Step 8 – Horizontal Partitioning (Sets)
Our architecture now looks like this
Positives
Infinite Scalability
Negatives
Aggregation of data across sets is complex
Users may need to be moved across Sets if sizing is improper
Global App settings and preferences need to be replicated across Sets
Global Redirector
App Servers
Cluster
App Servers
Cluster
DB Cluster
DB Cluster
DB Cluster
DB Cluster
SAN
SAN
SET 1
SET 2
Building a Scalable Architecture for Web Apps. Bhavin Turakhia