HTTP-Tester- A web appliance testing tool Kiran Komaravolu Dept of Computer Science and Engg Lehigh University

What is the need to test Web Appliances Need to use best components. Need to use best components. Compare performance of different web devices. Compare performance of different web devices. Verify if a device is performs as per specifications. Verify if a device is performs as per specifications. Choosing between different service providers. Choosing between different service providers.

Related Work The Ethereal network analyser ( The Ethereal network analyser ( TCPDUMP/PCAP ( TCPDUMP/PCAP ( Httpflow (Grevnin and Davison) Httpflow (Grevnin and Davison) Simultaneous proxy evaluation. (Davison). Simultaneous proxy evaluation. (Davison). Medusa Proxy. (Koletsou and Voelker). Medusa Proxy. (Koletsou and Voelker).

Http-Tester Replays captured requests to an existing server to one that is being evaluated. Replays captured requests to an existing server to one that is being evaluated. No active deployment needed. No active deployment needed. Does not affect existing services. Does not affect existing services. Can be used to evaluate any Web appliance running HTTP. Can be used to evaluate any Web appliance running HTTP.

HTTP-Tester Topology setup Topology setup

HTTP-Tester Components Components The Sniffer. The Sniffer. Logs all packets onto a packet database. Logs all packets onto a packet database. Request Replicator. Request Replicator. Replays all requests to the “tested device” Replays all requests to the “tested device” The Post Processor. The Post Processor. Reads the packet logs, extracts HTTP requests and responses and correlates them. Reads the packet logs, extracts HTTP requests and responses and correlates them.

HTTP HTTP Message Structure HTTP Message Structure A command line. (GET, POST, OK etc.) A command line. (GET, POST, OK etc.) GET / HTTP 1.0 GET / HTTP 1.0www.google.com Zero or more header lines. Zero or more header lines. Content-Type: text/html; charset=iso Content-Type: text/html; charset=iso Control Return + Line Feed. Control Return + Line Feed. Optional Message Body Optional Message Body Response message Response message

HTTP Transactions. HTTP 1.0 HTTP 1.0 No persistent connections. No persistent connections. A new connection is opened up for every request made, and is closed down after response sent. A new connection is opened up for every request made, and is closed down after response sent. HTTP 1.1 HTTP 1.1 Supports Persistent Connections. Supports Persistent Connections. Client may make a series of requests on a connection. Server returns responses in same order. Client may make a series of requests on a connection. Server returns responses in same order. Supports chunk encoding. Supports chunk encoding.

The Sniffer Blindly captures all relevant network packets and logs them. Blindly captures all relevant network packets and logs them. The pcap packet capture library has been used to read packets off the wire. The pcap packet capture library has been used to read packets off the wire. Berkeley DB package has been used to log the packets into a ‘db’ database. Berkeley DB package has been used to log the packets into a ‘db’ database. The job of efficient memory and disk usage has been delegated to DB. The job of efficient memory and disk usage has been delegated to DB.

Request Replicator Replays all requests flowing into original device onto the tested device. Replays all requests flowing into original device onto the tested device. Works in parallel with the sniffer module. Works in parallel with the sniffer module. Looks for TCP SYN packets and determines the beginning of a request flow. And opens up a new connection to the second device. Looks for TCP SYN packets and determines the beginning of a request flow. And opens up a new connection to the second device. The payload every next packet sent to on this connection is sent to the second device. The payload every next packet sent to on this connection is sent to the second device.

Request Replicator. This way the second server sees the same workload as the first one. This way the second server sees the same workload as the first one. The replies sent by the server are ignored. They are flushed out as the socket buffers overflow. The sniffer module though logs the TCP packets. The replies sent by the server are ignored. They are flushed out as the socket buffers overflow. The sniffer module though logs the TCP packets.

Post Processing. Extracts HTTP requests/responses from the packet logs. Extracts HTTP requests/responses from the packet logs. Every request is mapped onto its response. The same request response pair on the second connection is also identified. Every request is mapped onto its response. The same request response pair on the second connection is also identified. Httpflow routines used to extract HTTP requests from packet stream. Httpflow routines used to extract HTTP requests from packet stream.

Identifying Requests. We know which way requests flow. We know which way requests flow. Assumption: First TCP packet of every request has one of Assumption: First TCP packet of every request has one of GET, HEAD,OPTIONS, DELETE, POST, PUT. End of headers is determined by a CRLF. End of headers is determined by a CRLF. content-length field is specified in case of PUT/POST requests. content-length field is specified in case of PUT/POST requests.

Identifying responses Response headers extrated by looking for a CRLF. Response headers extrated by looking for a CRLF. End of Response message determined in three ways: End of Response message determined in three ways: Server sends a FIN or RST packet. Server sends a FIN or RST packet. Response headers contain a “content-length” field. Response headers contain a “content-length” field. Response is chunk encoded. Each chunk has its own length. Zero length chunk indicates end of stream. Response is chunk encoded. Each chunk has its own length. Zero length chunk indicates end of stream.

Correlating requests and responses HTTP 1.0 HTTP 1.0