1. Content Switch Research Projects at UCCS Network Research Lab
C. Edward Chow
Department of Computer Science
University of Colorado at Colorado Springs

2. Outline of the Talk Overview Content Switch Operation and Architecture
Improving Content Switch Processing: HA-LCS, More Server Load
Design of Secure Web Switch (LSWS)
Performance of Linux-based SWS
Status of IXP-based SWS
Future Directions (Integrate SSL accelerator/Traffic manager; Content Switch for Security)
Related Research (JESI dynamic cache server; Secure Collective Defense/A2D2; Secure Storage Networks; Multipath Routing; Wireless Security; Optimizing Wireless Multiplexing/Diversity)

3. Content Switch (CS) server1 client server2 . . server9
home.htm
Content Switch
server2
client
uccs.jpg .
Index.htm .
rocky.mid
server9
Switches capable of routing packets based on packet headers and content of layers 2-7.
Examples:
Direct web traffic based on pattern of URLs, host tags, cookies. (load balancing, firewall)
Detect/remove spam /virus; load balance IMAP sessions
Web switches and Intel XML Director/accelerator are special cases of content switch.

4. What Services It Can Provide
Enabling premium services for e-commerce, ISP, and Web hosting providers
Load Balancing and High Available Server Clusters: Web, E-commerce, , Computing, File, SAN
Policy-based networking, differential/QoS services.
Firewall, Strengthening DoS protection, cache/firewall load-balancing
‘Flash-crowd' management
Spam Protection, Virus Detection/Removal
Applet Authentication/Filtering

5. Basic Operations of Content Switching
CS: Content Switching
CS Rule Editor CS
Rules
Incoming
Packets
Packet Classification
Header
Content Extraction
CS Rule Matching Algorithm
Forward
Packet To
Servers
Packet Routing (Load Balancing)
Network Path Info
Server Load Status

6. Content Switch ArchitectureCS
Rules
Step2. CS processor a. Extract content/Match CS rules b.Route request c. Setup Sequence# modification on server side port
Real Server1
Case B: Step 1. Controller finds
there is no entry in Hash Table,
Route request to content switch processor
pkt
Modification
info
Step 3. At server side port,
Return pkts are modified Sequence#/IP addr/Chksum Route back to client
Hash
Table
Client

7. Efficient Software Architecture
Tasks: Million packets with thousand of rules to match and load balancing algorithms to run.
How to assign tasks to the (network) processors and threads?
Packet Extraction (Understand header formats, XML parsing)
Content Switching Rule Matching
Packet Routing (Load Balancing, Bandwidth Control)
How Much Packet Processing Should Controllers Do?
What a controller can do?
A Typical Parallel Processing Problem?

8. TCP Delay Binding (Splicing)
client
content switch
server
SYN(CSEQ)
step1
SYN(DSEQ)
step2
ACK(CSEQ+1)
ACK(DSEQ+1)
step3
DATA(CSEQ+1)
ACK(DSEQ+1)
step4
SYN(CSEQ)
step5
SYN(SSEQ)
step6
ACK(CSEQ+1)
step7
ACK(SSEQ+1)
step8
DATA(CSEQ+1)
ACK(SSEQ+1)
DATA(DSEQ+1)
step9
DATA(SSEQ+1)
ACK(CSEQ+LenR+1)
ACK(CSEQ+lenR+1)
step10
ACK(DSEQ+
lenD+1)
ACK(SSEQ+lenD+1)
step11
DATA(?) 2nd request
ACK(?)
lenR: size of http request. .
lenD: size of return document

9. Improve Content Switching
Setup CS-Real Server connections ahead of time (Persistent HTTP Connections). NetScale  Reduce TCP 3-way handshake time
Pre-allocate Server Scheme (Guess Real Server based on the TCP Sync)
Sequence# modification on every return pkt  Need to recompute checksum also.
Filter Scheme (Offload Sequence# modification/rule matching to real servers).
Buffering/Pipeline (aggregate) Requests

10. Pre-Allocate Server Scheme
client
Pre-allocated
server
content switch
SYN(CSEQ)
step1
SYN(CSEQ)
SYN(SSEQ)
step2
SYN(SSEQ)
ACK(CSEQ+1)
ACK(CSEQ+1)
step3
ACK(SSEQ + 1)
ACK(SSEQ+1)
step4
ACK(SSEQ+1)
step5
step6
DATA(SSEQ+1)
ACK(CSEQ+lenR+1)
ACK(CSEQ+LenR+1)
ACK(SSEQ+
lenD+1)
ACK(SSEQ+lenD+1)
DATA(CSEQ+1)
DATA(CSEQ+1)
Guess routing decision based on IP/Port#/History
Advantage:
Faster than TCP delay binding.
Possible direct route between client and server
Reduce session processing overhead no need to convert server sequence # .

11. Degenerated to TCP Delayed Binding If Guess is Wrong
Pre-allocated
server
client
content switch
SYN(CSEQ)
step1
SYN(CSEQ)
step2
SYN(SSEQ)/ ACK(CSEQ+1)
SYN(SSEQ)/ ACK(CSEQ+1)
step3
ACK(SSEQ + 1)
ACK(SSEQ+1)
DATA(CSEQ+1)/ ACK(SSEQ+1)
step4
DATA(CSEQ+1)/ACK(SSEQ+1)
step5
DATA(SSEQ+1)
Server sent HTTP 404
FIN(CSEQ+lenR+1))
step6
Right server
step7
SYN(CSEQ)
SYN(RSEQ)/ ACK(CSEQ+1)
step8
Sequence # conversion needed
for right server now
ACK(RSEQ+1)
step9
step10
DATA(CSEQ+1)/ACK(RSEQ+1)
DATA(SSEQ+1)/ACK(CSEQ+LenR+1)
DATA(RSEQ+1)/ACK(CSEQ+lenR+1)
step11
ACK(SSEQ+lenD+1
step12
ACK(RSEQ+lenD+1)

12. Migrate (Data, CSEQ, DSEQ)
Filter Process Scheme
client
Filter Process run on server
content switch
server
SYN(CSEQ)
step1
step2
SYN(DSEQ)/ACK(CSEQ+1)
ACK(DSEQ+1)
step3
DATA(CSEQ+1)/ACK(DSEQ+1)
step4
step5b
Migrate (Data, CSEQ, DSEQ)
SYN(CSEQ)
step5 a
SYN(SSEQ)/ ACK(CSEQ+1)
step6
ACK(SSEQ+1)
step7
DATA(CSEQ+1)/ACK(SSEQ+1)
step8
step9
DATA(SSEQ+1)
ACK(CSEQ+lenR+1)
DATA(DSEQ+1)
ACK(CSEQ+LenR+1)
ACK(DSEQ+
lenD+1)
ACK(SSEQ+lenD+1)
step10

13. Pre-allocate performance plot
Series 1 - Basic scheme with no rule matching module inserted, i.e., using default IPVS. Series 2 - Basic scheme with the rule matching module inserted. Series 3 - Pre-allocate scheme with all hits, i.e., where all pre-allocate guesses were correct. Series 4 - Pre-allocate scheme with all misses, i.e., where all pre-allocate guesses were wrong.

14. Handling multiple requests in a Keep-Alive connection
Determine when new request arrives
Verify that previous request has been completely received
Request data size is > 0
Key assumption is only one outstanding request is sent at a time by client, i.e., requests are not pipelined
Reuse connections
Store each connection control information in a hash table keyed by real server address, once it is established.

15. HA-LCS Architecture LAN user real server 1 heartbeat Coda file system
mon
heartbeat
Coda file
system
primary
real server 2
mon
heartbeat
backup
virtual server cluster
LAN
real server 3

16. HA-LCS Configuration
fladnag.uccs.edu - content switch (primary) Linux
walden.uccs.edu - content switch(secondary) Linux
vinci.uccs.edu real server 1 (coda client) Linux 2.4.3
gandalf.uccs.edu - real server 2 (coda client) Linux 2.4.3
wait.uccs.edu coda server Linux 2.4.3

17. Unique Constraints Imposed in HA-LCS as Compared to HA-LVS
In LCS, switching rules based on application content are hard wired in kernel rule module. To change a switching rule requires:
modify rule module code to reflect changed rule
compile modified rule module
remove old rule module
insert new old module
In LVS, switching rules based a simple load balancing policy and can be changed via built in commands

18. Related Load Balancing Research Results
Modified Apache status module to report
Total bytes to be transferred by child processes
Average document transfer speed
Modified LB-DNS to receive server status and bandwidth probing results.
LB-DNS returns IP-address of the best server based a weight contributed by both server load and bandwidth.
Modified WebStone benchmark to test the performance of load balancing web server clusters.

19. Load balancing Systems
Bandwidth Probe Results
Modified Web Server 1
Statistics Gathering
Daemon
Server Delay
Server Ranking
/tmp/StatFile
Modified Web Server n
LBA: Modified
DNS
Request for
Web pages

20. Connection Rate: LBA vs. Round-Robin
Round robin only run once

21. About SSL Secure Sockets Layer (SSL) protocol
developed by Netscape Communications to ensure private and authenticated communications
put into the public domain for Internet community

22. Session Establishment
ClientHello
ServerHello
Certificate
Establish protocol version, session- id, cipher suite, compression method.
Certificate Request
ServerHelloDone
Certificate Verify
ChangeCipherSpec
Handshake
Optionally send server certificate and request client certificate
Change CipherSpec and finish handshake.
Send client certificate response if requested.
Client
Server

23. Session Reusability
ClientHello
ServerHello
Establish protocol version, session-id ( Including previous session info), cipher suite, compression method.
ChangeCipherSpec
Handshake
Change CipherSpec and finish handshake.
Client
Server
If the Client wants to reuses the same session, it sends the previous session id in the clientHello message.
If the server wants to reuse the same session, it sends the same session id back in the serverHello

24. OpenSSL
OpenSSL is based on the SSLeay library developed by Eric A. Young and Tim J. Hudson.
Open Source toolkit implementing the Secure Socket Layer (SSL v2/v3) and Transport Layer Security (TLS v1) protocols as well as a full-strength general purpose cryptography library
Important Libraries
SSL
OpenSSL ssl library implements SSL v2/v3 and TLS v1
Crypto
OpenSSL crypto library implements a wide range of cryptographic algorithms used in various Internet standards. The services provided by this library are used by OpenSSL implementations of SSL, TLS, and they have also been used to implement SSH, OpenPGP, and other cryptographic standards.

25. OpenSSL Command Interface
The Openssl program is a command line tool for using the various cryptography functions of OpenSSL's crypto library from the shell. It can be used for
Creation of RSA, DH and DSA key parameters
Creation of X.509 certificates, and Certificate Revocation List (CRL)
Calculation of Message Digests o Encryption and Decryption with Ciphers
SSL/TLS Client and Server Tests
Handling of S/MIME signed or encrypted mail

26. Secure Content Switch(SCS)
Secure content switch is a transparent proxy that can translate between encrypted and unencrypted data transport on socket connections.
Need for secure network access and high performance
e-commerce transactions require security
Need high performance for better Quality of Service
Solution: just plug in SCS between client and the server to add Secure Socket Layer (SSL) support.

27. Goals of Secure Content Switch
In addition to the above, we need to be able to route requests based on packet content to a set of backend real servers.
Design Considerations
Our real servers can be located at different places
Efficiency must not be ruined
Easy to understand/write content switching rules.
Dynamic rule update
Session Reusability

28. Design of Secure Content Switch

29. Architecture of Secure Content Switch
SCS
Dispatcher
Module
Content
Extraction
Module
Web
Browser
packet Info
header/
content
Child
Module
Rule
Module
routing
decision
The web browser makes a request to the secure content switch.
The dispatcher module in the secure content switch forwards the request to the secure content switch child module. In the dynamic forking version of SCS the dispatcher module forks a child process. In Preforking version of SCS the dispatcher module forwards request to a free child.
The secure content switch child module performs the handshake with the client and reads in the request.
The secure content switch child module then sends the request to the Rule module, which performs rule matching and returns the name of the server by which the request can be served.
The secure content switch child forwards the request to the real server based on the routing decision …
Real
Server 1
Real
Server 2
Real
Server n

30. Secure Content Switch using Dynamic Forking
Request From Web Browser to SCS
Dispatcher
Module
fork ()
SCS
Child Module
Negotiate SSL Session No
Existing SSL
Session
Decrypt Object
Using SSL Session
Information
SSL Request
Yes
Yes
Encrypt the Object Per Session Info and Send over HTTPS to Web Browser
Retrieve Object From the Server
Using Standard HTTP
Extract Object Info/Sent To Rule Matching Module
Rule Matching; Sent routing decision back to Child Module

31. Secure Content Switch using Prefork Processes
Request From Web Browser to SCS
Dispatcher
Module
Assign
Assign
Preforked SCS
Child Process n
Preforked SCS
Child Process 2
Preforked SCS
Child Process 1
Negotiate SSL Session No
Existing SSL
Session
Decrypt Object
Using SSL Session
Information
SSL Request
Yes
Yes
Encrypt the Object Per Session Info and Send over HTTPS to Web Browser
Retrieve Object From the Server
Using Standard HTTP
Extract Object Info/Sent To Rule Matching Module
Rule Matching; Sent routing decision back to Child Module

32. E-Commerce Example: 1. Client
Client submits via HTTP/Post (or SOAP) the following purchase in XML:
<purchase>
<customerName>CCL</customerName>
<customerID> </customerID>
<item><productID> </productID>
<productName>IBM Thinkpad T21</productName>
<unitPrice>5000</unitPrice>
<noOfUnits>10</noOfUnits>
<subTotal>50000</subTotal>
</item>
<item><productID> </productID>
<productName>Intel wireless LAN PC Card</productName>
<unitPrice>200</unitPrice>
<subTotal>2000</subTotal>
<totalAmount>52000</totalAmount>
</purchase>

33. E-Commerce Example: 2. Content Switch
Content switch receives the packet.
Recognize it is a http post request from http request line POST /purchase.cgi HTTP/1.1
Recognize it is an XML document from the meta header content-type: TEXT/XML
Parsing XML content
Extract values of tag sequences: purchase/totalAmount CCL purchase/customerName
Rule 1 is matched and packet is routed to one of highSpeedServers. Rule 1: if (xml.purchase/totalAmount > 5000) routeTo(highSpeedServers); Rule 2: if (xml.purchase/customerName == CCL) routeTo(specialCustomerServers);

34. Java-based Rule Editor
Detect conflicts in content switch rule set
Convert rules into LCS rule module.

35. Design of Rule Module
How can we update rules dynamically? Make rule matching as a separate process
Other Design considerations:
Should the rule module run on other machine?
E.g., thread on IXP microengine.
What is going to be the impact on performance ?

36. Current Design and Operation of Rule Module
Rule module can run as a separate process on the same or different machine.
Rule module is an iterative server because we found that encryption and decryption are the bottlenecks from the performance results (not rule matching).
To update rule set
Shutdown the rule module, compile the rule module with new rule set, and start up the rule module
During this period, packets are routed to default rule module.

37. SCS Processing Establish Connection with Rule Module
Send Url, Src portno, Src IP, HTTP Headers, Data (if any) to Rule Module
succeed
Decrypt SSL
Packet Data
Fail IS
(Method == Post)
Establish Connection with Default Rule Module
Succeed No
Yes
IS (content type == x-www-form-urlencoded) No
Perform rule matching and send back Real Server Name, Address and Port # on which Real Server is listening
Fail
Default Rule
Module Missing
Yes
Decrypt the data
And populate the rules with values (if any)

38. Linux-based SCS Test bed (LACS)

39. Configuration of Machines Used in Testbed
Machine Spec
IP Address
O/S
webserver
a) calvin.uccs.edu DELL Dimension-4100, 933 MHz, 512MB b) oblib.uccs.edu
HP Vectra VL
512 MHz, 512MB
 (Content switch)  
Redhat 7.2
( )
Apache
a) dilbert.uccs.edu
b) wait.uccs.edu
c) wind.uccs.edu
 (Client)
a)WinNT-4.0
b)&c)Win-2000, Adv. Server
N/A
a) eca.uccs.edu
b) frodo.uccs.edu
c) bilbo.uccs.edu
d) odorf.uccs.edu
e) walrus.uccs.edu
f) wallace.uccs.edu
 HP Kayak, 233 MHz, 96MB RAM
(Real Server)
Redhat 7.1
( )
Apache

40. SSL Processing Overhead
Average SSL req./sec is 14.7
Average HTTP req./sec is 180
dilbert.uccs.edu
eca.uccs.edu

41. Dynamic vs. Preforking SCS
The performance of the Pre-forking SCS is better than Dynamic Forking SCS
eca.uccs.edu
dilbert.uccs.edu
oblib.uccs.edu
frodo.uccs.edu

42. Dynamic vs. Preforking SCS
I found out that after one stage the child’s are created and killed immediately there by they are serving a single request, because the Pre-forked server is designed to see that child’s are created ahead of time and kill them if there are more number of child’s, I found that during the test that the child’s are being killed reason could be that more child’s are free which implies number of requests sent by the web-bench are irregular, there by affecting the overall performance of the pre-forked SSLProxy
The performance of the Dynamic forking SCS is better than Pre-forked SCS
Reason ?
What is the advantage of using cluster ?
eca.uccs.edu
dilbert.uccs.edu
calvin.uccs.edu
frodo.uccs.edu

43. Performance of Prefork SCS on varying # of Startup Children
Startup children => no of child Processes spawned ahead of time
It is Suggested always to keep the Startup Children Small if you don’t expect heavy traffic
Having about 25 pre spawned children is better if the traffic load is heavy
eca.uccs.edu
dilbert.uccs.edu
calvin.uccs.edu
frodo.uccs.edu

44. Impact of Rules on the performance of Dynamic SCS
Clearly there is no impact of rules on the performance of Dynamic Forking Secure Content Switch
eca.uccs.edu
dilbert.uccs.edu
calvin.uccs.edu
frodo.uccs.edu

45. Impact of Rules on the performance of Dynamic Non-SCS
Clearly there is some impact of Rules on the the Performance of Dynamic Forking Non-Secure content Switch
the smaller the rule set, better the performance
No heavy impact of the performance of the Secure content Switch with increase in the number of rules

46. Impact of the # of Real Servers on the Performance of Dynamic SCS
Clearly there is no impact of the # of Real Server on the the Performance of Dynamic Forking Secure content Switch
Secure Content Switch is the bottleneck ??
calvin.uccs.edu
eca.uccs.edu
dilbert.uccs.edu
frodo.uccs.edu

47. Impact of Real Servers on the Performance of Dynamic Non-SCS
Performance is not directly proportional to # of Real Servers !!
Clearly there is impact of Real Server on the the Performance of Dynamic Forking Non-SSL Secure content Switch
Performance was found to degrade when there is only one real server

48. Performance of SCS in Local Node situation
Local Node => Web Server runs on the Content Switch machine, therefore the content Switch can serve the requests by routing requests internally
calvin.uccs.edu
No Performance degradation for SSL version of Secure content Switch
dilbert.uccs.edu
Apache Web Server

49. Interest Problem Encountered: Conflicts among Different Servers on Same Machine
While measuring results using WebBench for local node situation for pre-fork non-secure content switch
With local Apache running on port 8000, SCS on 80.
After serving a few requests SCS stopped serving requests. Why is that?
If we stop Apache web server, SCS starts serving requests again. If we resume the Apache server, after serving a few requests SCS stops serving requests again.
Potential server file locks were ruled out.

50. Performance of SCS When Rule Module Runs Locally
Pre-fork SCS Overtakes Dynamic forking SCS
Dynamic forking SCS Performance was degraded by 100%
Others Variations of SCS did not suffer much
calvin.uccs.edu
eca.uccs.edu
dilbert.uccs.edu
Rule
frodo.uccs.edu

51. Performance of SCS When Rule Module Runs Remotely
calvin.uccs.edu
No major change in performance w.r.t rule module running locally
eca.uccs.edu
dilbert.uccs.edu
Rule
frodo.uccs.edu

52. Status of IXP-based SCS
Ported OpenSSL0.6 to VxWork.
Have an IXP-based SCS running on Fast Ethernet ports (rule modules can be on Linux machines or on StrongArm)
Working on Gigabit port driver with help from CCL.
Working on distributing tasks to microengines
Testing Goahead web server for uploading object files (such as rule module, configuration files).

53. Future Direction: Parallel SCS using SSL Accelerators
Use parallel processing to reduce bottleneck of SSL processing.
Intel just donated 8 CEA7110 SSL Accelerators and 4 CEA7280 XML Directors to our Lab.
Plan to use CEA7110 for Parallel SCS.
Compare performance of XML directors with that of our SCS.
Experiment with proposed server id parameter feature in OpenSSL 0.7

54. Future Directions
Using User Mode Linux (UML) for quick kernel-based content switch development (with Spam Mail/Virus detection/removal)
Implement packet rewrite/page rewrite techniques.
Perform fair comparison of all clustering techniques implemented.

55. Related Research Projects
Dynamic Cache Server using JESI (Java Edge Server Side Include).
Secure Collective Defense
Intrusion tolerance
Modified Secure DNS
Proxy server/Alternate Route
A2D2 (Automatic Anti-DDOS System):
Explore the use of multiple connection relay servers
Efficient Intrusion Handling System
Multipath Routing.
VOIP/802.11a/b; Wireless Security (PEAP vs. TTLS).
Optimizing Multiplexing/Diversity Trade-off for Improving Wirless Network Access (proposed Medium size NSF project with Virginia Tech)

56. ESI features
Can be generated and cached by the cache server according to the web page designer

57. JSP (Java Server Pages) custom tag specifications
To use the JSP custom tags, we need to define three separate components:
JSP file.
tag library descriptor (tld).
tag handler class.
simple-taglib.tld
<name>hello</name>
<tagclass>
cwp.tags.HelloWorldTag
</tagclass>
Helloworld.jsp
taglib
uri="simple-taglib.tld”
prefix="jspx" %>
<jspx:hello />
HelloWorldTag.java
public class HelloWorldTag extends TagSupport { }

58. UCCS JESI

59. Autonomous Anti-DDoS Network Security Testbed A D )
Normal
Output
route
Attack
rate limiting
Access
RealServer
100Mpbs Switch
Page 1
Autonomous Anti-DDoS Network Security Testbed A 2 D )
(New Proposed Testbed - V1)
Monday, July 01, 2002
Private Subnet
Public
Network
RealPlayer Server
Computer B
Computer C
Computer D
IP:
NM:
GW:
eth0
Pluto (C3)
as Linux Router
IP:
NM:
GW:
eth1
Titan (C2)
iptables firewall
IP:
GW:
IP:
Saturn (C1)
Real Player Client
Computer A
Internet
IP:
DMZ
Alternate
output
during
attack
Detect attack not
filtered by firewall,
instruct firewall to
start rate limiting
against specific
DDoS attack
Snort

60. Conclusion We have developed
two versions of Linux kernel based Web Switches (LWS).
a HA-LWS with coda file system.
An application level Secure Web Switch (LSWS).
A preliminary IXP-based web switch.
Network Bandwidth Measurement/Server load tools
Very interested in joint research/development efforts.

61. Discussion Feedback Suggestions
Potential Research/Development Collaboration: