1. The TIME-WAIT state in TCP and its Effect on Busy Servers Theodore Faber University of Southern California
Xindian Long

2. Outline TIME-WAIT state and its function
Performance problems on busy web server
Solution: move the TIME-WAIT to clients
End points negotiation
TCP, HTTP modification
Experiments
Conclusion

3. Delayed Packet Problem

4. TIME-WAIT State
Blocking connections between the same address/port pair
Holding a Time-Wait State at one endpoint
2MSL
The end doing active close holds the TIME-WAIT State

5. TIME-WAIT State

6. TIME-WAIT state in the server
The server actively closes the connection, and maintains the TIME-WAIT state
Server sending data
TCP connection closure as end-of-transaction marker
Simple protocol, fast response
Otherwise:
Knowing the content length
Extra end-of-transaction marker
requires masking & restoring the marker
HTTP, FTP servers use TCP connection closure as an unambiguous end-of-transaction marker

7. Performance Problems on Busy Server
Too many connections in TIME-WAIT State
TCB Consumes memory
Slows active connections
A shorter MSL weaken the protection
Solutions like persistent connection are not enough

8. Solution: TIME-WAIT in Clients
Blocking at one end is enough
Move the TIME-WAIT to the Client
End points negotiating
TCP Modification
HTTP Modification
Scales better than persistent connection

9. TIME-WAIT Negotiation
Non-simultaneous connection establishment
SYN (TW-Negotiate)
Client
Server
SYN, Ack (TW-Negotiate set to its choice)
Ack (TW-Negotiate set to the same value)
If Acceptable
If not Acceptable
RST
If the server does not respond with TW-negotiate option, the current TCP semantic is applied.

10. Either Contains No Option
TIME-WAIT Negotiation
Simultaneous connection establishment
SYN (TW-Negotiate)
SYN (TW-Negotiate)
SYN-RCVD
SYN-RCVD
endpoint1
endpoint2
Value Known
Value To Use
Either Contains No Option
No Option
Same IP Address
That IP Address
Different IP Address
Larger IP Address

11. TIME-WAIT Negotiation
Advantage
Makes post-connection memory requirement explict
Transparent to applications
No hidden performance penalty
Disadvantage
Significant change to TCP state machine
Information from connection establishment affects the closure
Negotiating at closure reduces endpoints’ control over their resource
Allow either endpoint to decline the connection if the overhead is unacceptable.
Only move to the TIME-WAIT those clients willing to accept the overhead
Significant programming and testing effort
Correctness proof based on TCP state machine would be invalidated

12. TCP Level Solution Modify the clients’ TCP implementation
Sends a RST packet to the server
Puts itself into a TIME-WAIT state
Final Ack TW
RST TW
closed
Client
Server

13. Modified TCP State Machine

14. TCP Level Solution
Only works for system that accepts <RST> in TIME-WAIT state
Performance limited by the way <RST> is processed
Changes the meaning of <RST> packet

15. Application Level Solution
Decouple the end-of-connection with end-of-transaction indication
HTTP 1.1 Modification
Content Length as the end of transaction
New Extension Request: CLIENT_CLOSE
The client opens a connection
Sending series of requests
Collecting the response
Sending a Client-Close, closing connection
The server closes its end without responding

16. CLIENT_CLOSE Extention
[Conection: close] line
Client
Server
data
CLIENT_CLOSE
Detect the end
of data
close
FIN .
Last request:
Connection: close
data
close
FIN .
Client
Server

17. Application Level Solution
CLIENT-WAIT notify the server about the client close
Requires change only on the client side
Conforms to HTTP 1.1
Requires no change to other protocols
Creates no new security vulnerability
ONLY reduces the load by HTTP

18. Experiments Environment Three Tests Implemented under SUNOS 4.1.3
Evaluation using custom benchmark program and WebSTONE benchmark
640 Mb/s Myrinet LAN
Three Tests
Demonstration of Worst-case Server Loading
HTTP Experiment
Time wait avoidance and Persistent Connections

19. Demonstration of Worst-Case Server Loading
Test Configuration
A Server
Two clients doing simultaneous bulk transfer
Server loaded with TIME-WAIT state by a fourth machine
Constructing a worst case scenario
Client connections are put at the end of the list of TCBs
Two clients are used to neutralize the simple caching behavior
Two distinct clients allow bursts from different clients to interleave
The first experiment was designed to determine if TCB load reduces server throughput and if our modifications alleviate that effect;

20. Demonstration of Worst-Case Server Loading

21. HTTP Load Experiments Two workstations each running 20 clients
File Size: 9KB-5MB
System
Throughput Mb/s
Conn/s
TCP Mem (Kb)
Unmodified
20.97
49.09
722.7
TCP Mods
26.40
62.02
23.1
HTTP Mods
31.73
74.70
23.4
8 clients on 4 workstations, small files
System
Throughput Mb/s
Conn/s
TCP Mem (Kb)
Unmodified
Fails
TCP Mods
1.14
223.8
16.1
HTTP Mods
222.4

22. TIME-WAIT Avoidance and Persistent Connection
2 Client machine, 1 Server
5 HTTP request bursts
10 request / Connection
TIME-WAIT avoidance methods increase the per-connection throughput as client load increases
Throughput vs. Clients

23. TIME-WAIT Avoidance and Persistent Connection
Connection Rate vs. Clients

24. TIME-WAIT Avoidance and Persistent Connection
Memory vs. Clients

25. Conclusion TCP With TCP With CLIENT_CLOSE TIME-WAIT Client HTTP
Negotiation <RST> Extension
Reduces TIME-WAIT Loading Yes Yes Yes
Compatible With Current Protocols Yes Yes Yes
Changes Are Effective If Only No Yes Yes
The Client Is Modified
Allows System To Prevent Yes No Yes
TIME-WAIT Assassination
No Changes To Transport Protocol No No Yes
No Changes To Application Protocols Yes Yes No
Adds No Packet Exchanges To Yes No No
Modified Protocol
TIME-WAIT Allocation Is A Requirement Yes No No
of Connection Establishment