1. Enhance Features and Performance of Content Switches
9/18/2018
Enhance Features and Performance of Content Switches
Chandra Prakash
Department of Computer Science University of Colorado at Colorado Springs
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS
Chandra Prakash/Enhance Features of CS

2. Chandra Prakash/Enhance Features & Performance of CS
Outline of the Talk
Introduction
Existing Content Switch related products/techniques
Basic Architecture of Content Switch (CS)
TCP Delayed Binding and proposed improvements
Performance results of various schemes for improving TCP Delayed Binding
Handle multiple requests in HTTP Keep-Alive connection
Enhancements to CS
Handling muitlple packets of a HTTP request
Handling different data encoding formats
Improved XML rule matching
High-Availability of Linux CS Cluster
Conclusion
Future Work
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

3. Content Switch-based Cluster
RIP1
Real Server1
RIP2
Real Server2
Internet
WAN/ LAN
VIP
RIP3
CIP
Virtual Server/
Content Switch
Real Server3
Client
CIP: Client IP Address
VIP: Virtual IP Address
RIP: Real Server IP Address
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

4. Existing Content Switch Related Products
Alteon's series (A180e/A184) products support URL-based Server load balancing
F5's Big-IP product supports load balancing and contents switching
Foundry network's ServerIron product supports URL, Cookie, and SSL Session ID-based switching
Intel’s XML accelerator products can distribute Web load based on XML tag values
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

5. Existing Content Switch Related Techniques
MAC address translation (MAT)
MAC multicast
Half network address translation (HNAT), also known as NAT in Linux Virtual Server Project (
Full network address translation (FNAT)
IP tunneling
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

6. MAC Address Translation
server to client traffic
client traffic
real server 1
ethernet IP:
loopback IP:
LAN
client traffic
real server 2
ethernet IP:
loopback IP:
virtual server
VIP:
real server 3
ethernet IP:
loopback IP:
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

7. Chandra Prakash/Enhance Features & Performance of CS
MAC Multicast
server to client traffic
client traffic
client traffic
real server 1
IP1
LAN
client traffic
real server 2
IP2
switch
client traffic
MAC Multicast group
real server 3
IP3
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

8. Chandra Prakash/Enhance Features & Performance of CS
Half NAT
client traffic
server to client traffic
client traffic
real server 1
ethernet IP:
Default GW:
server to client traffic
real server 2
ethernet IP:
Default GW :
virtual server
VIP:
LAN
real server 3
ethernet IP:
Default GW :
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

9. Chandra Prakash/Enhance Features & Performance of CS
Full NAT
server to client traffic
server to client traffic
client traffic
client traffic
real server 1
IP1
real server 2
IP2
virtual server
VIP
real server 3
IP3
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

10. Chandra Prakash/Enhance Features & Performance of CS
IP Tunneling
server to client traffic
client traffic
real server 1
IP1
client traffic
real server 2
IP2
virtual server
VIP
packet destined
for
real server
encapsulation
at virtual server
VIP
RIP
VIP
client packet
real server 3
IP3
decapsulation
at real server
VIP
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

11. Basic Operations of Content Switching
CS: Content Switching
CS Rule Editor CS
Rules
Incoming
Packets
Packet Classification
Header
Content Extraction
CS Rule Matching Algorithm
Packet Routing (Load Balancing)
Network Path Info
Forward
Packet To
Servers
Server Load Status
Load Balancing Repository
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

12. TCP Delayed Binding(Basic Scheme)
client
content switch
server
SYN(CSEQ)
step1
SYN(DSEQ)
step2
ACK(CSEQ+1)
ACK(DSEQ+1)
step3
DATA(CSEQ+1)
ACK(DSEQ+1)
step4
SYN(CSEQ)
step5
SYN(SSEQ)
step6
ACK(CSEQ+1)
step7
ACK(SSEQ+1)
step8
DATA(CSEQ+1)
ACK(SSEQ+1)
DATA(DSEQ+1)
step9
DATA(SSEQ+1)
ACK(CSEQ+LenR+1)
ACK(CSEQ+lenR+1)
step10
ACK(DSEQ+
lenD+1)
ACK(SSEQ+lenD+1)
lenR: size of http request. .
lenD: size of return document
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

13. Pre-Allocate Scheme if Guess is Correct
client
Pre-allocated
server
content switch
SYN(CSEQ)
step1
SYN(CSEQ)
SYN(SSEQ)
step2
SYN(SSEQ)
ACK(CSEQ+1)
ACK(CSEQ+1)
step3
ACK(SSEQ + 1)
ACK(SSEQ+1)
step4
ACK(SSEQ+1)
step5
step6
DATA(SSEQ+1)
ACK(CSEQ+lenR+1)
ACK(CSEQ+LenR+1)
ACK(SSEQ+
lenD+1)
ACK(SSEQ+lenD+1)
DATA(CSEQ+1)
DATA(CSEQ+1)
Guess routing decision based on IP/Port#/History
Advantage:
Faster than TCP delay binding.
Possible direct route between client and server
Reduce session processing overhead no need to convert server sequence # .
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

14. Pre-allocate Scheme if Guess is Wrong
Pre-allocated
server
client
content switch
SYN(CSEQ)
step1
SYN(CSEQ)
step2
SYN(SSEQ)/ ACK(CSEQ+1)
SYN(SSEQ)/ ACK(CSEQ+1)
step3
ACK(SSEQ + 1)
ACK(SSEQ+1)
DATA(CSEQ+1)/ ACK(SSEQ+1)
step4
DATA(CSEQ+1)/ACK(SSEQ+1)
step5
DATA(SSEQ+1)
Server sent HTTP 404
RST
step6
Right server
step7
SYN(CSEQ)
SYN(RSEQ)/ ACK(CSEQ+1)
step8
Sequence # conversion needed
for right server now
ACK(RSEQ+1)
step9
step10
DATA(CSEQ+1)/ACK(RSEQ+1)
DATA(SSEQ+1)/ACK(CSEQ+LenR+1)
DATA(RSEQ+1)/ACK(CSEQ+lenR+1)
step11
ACK(SSEQ+lenD+1
step12
ACK(RSEQ+lenD+1)
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

15. Migrate (Data, CSEQ, DSEQ)
Filter Process Scheme
client
Filter Process run on server
content switch
server
SYN(CSEQ)
step1
step2
SYN(DSEQ)/ACK(CSEQ+1)
ACK(DSEQ+1)
step3
DATA(CSEQ+1)/ACK(DSEQ+1)
step4
step5b
Migrate (Data, CSEQ, DSEQ)
SYN(CSEQ)
step5 a
SYN(SSEQ)/ ACK(CSEQ+1)
step6
ACK(SSEQ+1)
step7
DATA(CSEQ+1)/ACK(SSEQ+1)
step8
step9
DATA(SSEQ+1)
ACK(CSEQ+lenR+1)
DATA(DSEQ+1)
ACK(CSEQ+LenR+1)
ACK(DSEQ+
lenD+1)
ACK(SSEQ+lenD+1)
step10
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

16. Chandra Prakash/Enhance Features & Performance of CS
Performance Metrics
Processing time vs document size for GET request
Processing time vs document size for POST request
Obtained results for individual schemes using Webbench by varying the delay and number of threads sending request
Plot of max sustainable requests/sec vs. number of rules
Plot of max throughput in bytes/sec vs. number of rules
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

17. Benchmark Configuration
fladnag.uccs.edu - content switch Linux
vinci.uccs.edu real server Linux
gandalf.uccs.edu real server Linux
dilbert.uccs.edu - client Windows NT 4.0
For plot of processing time vs. document size.
used a Perl script that sends GET and POST requests
with varying request and response sizes.
For response time and throughput measurement.
used Webbench
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

18. Processing Time vs. Response Size for GET Request
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

19. Processing Time vs. Request Size for POST Request
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

20. Comparison of Overall Webbench Requests/Second Metric
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

21. Comparison of Overall Webbench Throughput (Bytes/Second) Metric
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

22. Request/Sec vs. Number of CS Rules
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

23. Throughput in Bytes/Sec vs. Number of CS Rules
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

24. Handling Multiple Requests in a Keep-Alive Connection
Determine when new request arrives
Verify that previous request has been completely received
TCP payload size is > 0
Key assumption is only one outstanding request is sent at a time by client, i.e., requests are not pipelined.
Reuse connections
Store each connection control information in a hash table keyed by real server address, once it is established.
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

25. Keep-Alive Connection Hash Table
For each real server, hash table entry stores following parameters:
rs_addr
cli_str_seq
cli_str_ack_seq
rs_last_next_seq
rs_last_ack_seq
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

26. Client to Real Server Sequence Translation in a Keep-Alive Connection
Sequence number of packet sent by client and forwarded to current real server
rs_last_ack_seq + (cli_cur_seq - cli_str_seq)
Here cli_cur_seq is the sequence number of currently forwarded client packet.
Acknowledgment number of packet sent by client and forwarded to current real server
rs_last_next_seq + (cli_cur_ack_seq - cli_str_ack_seq)
Here cli_cur_ack_seq is the acknowledgment number of currently forwarded
client packet.
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

27. Real Server to Client Sequence Translation in a Keep-Alive Connection
Sequence number of packet sent by real server and forwarded to client
cli_str_ack_seq + (rs_cur_seq - rs_last_next_seq)
Here rs_cur_seq is the sequence number of currently forwarded real server packet.
Acknowledgment number of packet sent by real server and forwarded to client
cli_str_seq + (rs_cur_ack_seq - rs_last_ack_seq)
Here rs_cur_ack_seq is the acknowledgment number of currently forwarded
real server packet
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

28. Handling Multiple Packets of a HTTP Request
Request may span over multiple TCP segments which requires queuing of incoming packets
Determine when request is completely received which requires parsing of HTTP header content, e.g., “Content-Length” tag in requests like PUT and POST
Keeping in sync with client and server TCP
HTTP request fragmentation example, where TCP Segment n contains:
POST /cgi-bin/cs622/purchase.pl HTTP/1.0\r\n
Referer:
Connection: Keep-Alive\r\n
Content-type: application/x-www-form-urlencoded\r\n
Content-length: 7
and TCP Segment n+1 contains:
53\r\n
data (753 bytes)
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

29. Handling Different Data Encodings in XML Document
Typically there are two encoding techniques
text/xml
consist of plain ascii text with no specical encoding
x-www-form-urlencoded
Consist of text where special characters are encoded as “%XX”, where XX is the hexadecimal value of the special character.
For example, newline and left anchor (‘<‘) characters are encoded as "%0A" and "%3C” respectively.
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

30. Improved Rule Specification
<purchase>
<customerID> </customerID>
<item>
<productID> </productID>
<unitPrice>5000</unitPrice>
</item>
<productID> </productID>
<unitPrice>200</unitPrice>
</purchase>
Many tags with the same name make rule specification ambiguous, e.g, the item tag in above XML sample document
A rule specification like “purchase:1:item:2:uniPrice:1 > 200” allows to access unitPrice tag of the second item
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

31. High-Availability of Linux Content Switch (HA-LCS)
Address issues related to fault tolerance of
Virtual server
real servers or services on the real server
high-availability of data files (e.g. HTML docs)
The setup is based on existing configuration of high-availability of LVS with with following key software components:
Heartbeat (for fault tolerance of virtual server)
Mon (for fault tolerance of services of real server)
Coda (for high-availability of data files)
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

32. Chandra Prakash/Enhance Features & Performance of CS
HA-LCS Architecture
user
real server 1
mon
heartbeat
Coda file
system
primary
real server 2
mon
heartbeat
backup
virtual server cluster
LAN
real server 3
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

33. Chandra Prakash/Enhance Features & Performance of CS
HA-LCS Configuration
fladnag.uccs.edu - content switch (primary) Linux
walden.uccs.edu - content switch(secondary) Linux
vinci.uccs.edu real server 1 (coda client) Linux
gandalf.uccs.edu - real server 2 (coda client) Linux
wait.uccs.edu coda server Linux
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

34. Unique Constraints Imposed in HA-LCS as Compared to HA-LVS
In LCS, switching rules based on application content are hard wired in kernel rule module. To change a switching rule requires:
modify rule module code to reflect changed rule
compile modified rule module
remove old rule module
insert new old module
In LVS, switching rules based a simple load balancing policy and can be changed via built in commands
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

35. HA-LCS Configuration Notes
Heartbeat
Setting up “ha.cf” and “haresources” file
Note: The IP address specified in the “haresources” file should not be configured via OS
Mon
Setting up “mon.cf” and real server failure/startup handler script “wk_up.ksh”
Coda
Setting up coda server using “vice-setup” and configuring client
When creating “coda volume” on server the INSTALL.linux setup file says create volume as:
createvol_rep coda:root E /vicepa, where “coda:root” is root volume
On the other hand, online coda help says root volume be set as “coda.root”.
While creating coda admin, do not specify user id as 1, even though instructions say one can use coda admin user id as 1.
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

36. Performance of Different File Systems with HA-LCS
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

37. Chandra Prakash/Enhance Features & Performance of CS
Conclusion
Implemented three schemes to study improvement on TCP delayed binding. Pre-allocate scheme gave best results followed by basic and filter scheme.
Implemented scheme to handle multiple requests in a given connection coming in a non-pipelined fashion.
Proposed ways to handle request sent in a pipelined fashion in a Content Switch
Addressed issues related to content switch processing:
handling multiple packets in a request,
improving rule matching,
handling different data encoding formats.
Implemented a highly-available Linux Content Switch (LCS) system
Identified key issues related to fault tolerance in LCS and implemented the solutions.
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS

38. Chandra Prakash/Enhance Features & Performance of CS
Future Work
Improve the reliability of LCS by moving the content switch processing from IP layer to Transport layer.
Enhance load balancing policies by considering network path status and server load.
Improve performance by reusing connections in a connection pool to avoid setup overhead
Utilize latest protocols, e.g., ASAP/ENRP, for managing fault-tolerant clusters.
9/18/2018
Chandra Prakash/Enhance Features & Performance of CS