IP Security (IPSec) Encapsulating Security Payload (ESP) Dr Milan Marković

Introduction  ESP is used to provide integrity check, authentication, and encryption to IP datagrams.  However, some restrictions apply:  Integrity check and authentication are used together.  Replay protection is selectable only in conjunction with integrity check and authentication.  Replay protection can be selected only by the receiver.  Encryption can be selected independently of other services.  It is highly recommended that, if encryption is enabled, integrity check and authentication be turned on.  If only encryption is used, intruders could forge packets in order to mount cryptanalytic attacks.

Introduction  Although both authentication (with integrity check) and encryption are optional, at least one of them is always selected, otherwise you would not be using ESP.  ESP is identified by protocol number 50, as assigned by the IANA.  If both encryption and authentication with integrity check are selected, then the receiver first authenticates the packet and, only if this step was successful, proceeds with decryption.  This mode of operation saves computing resources and reduces the vulnerability to denial of service attacks.

ESP packet format

 Payload data The payload data field is mandatory. It consists of a variable number of bytes of data described by the next header field. This field is encrypted with the cryptographic algorithm selected during SA establishment. If the algorithm requires initialization vectors, these are also included here. The ESP specification requires support for the DES algorithm in CBC mode (DES-CBC transform). Often, other encryption algorithms are also supported, such as triple- DES and CDMF, in the case of IBM products.  Padding Most encryption algorithms require that the input data must be an integral number of blocks. Also, the resulting ciphertext (including the padding, pad length and next header fields) must terminate on a 4-byte boundary, so the next header field is right- aligned. For this reason, padding is included. It can also be used to hide the length, of the original messages. However, this could adversely impact the effective bandwidth. Padding is an optional field (but needed for some algorithms).  Pad length This 8-bit field contains the number of the preceding padding bytes. It is always present, and the value of 0 indicates no padding.

ESP packet format  Next header The next header is an 8-bit mandatory field that shows the data type carried in the payload, for example an upper- level protocol identifier such as TCP. Thevalues are chosen from the set of IP protocol numbers defined by the IANA.  Authentication data This field is variable in length and contains the ICV calculated for the ESP packet from the SPI to the next header field inclusive. The authentication data field is optional. It is included only when integrity check and authentication have been selected at SA initialization time. The ESP specifications require two authentication algorithms to be supported: HMAC with MD5 and HMAC with SHA-1. Often the simpler keyed versions are also supported by IPsec implementations.  Note: The encryption covers the payload data, padding, pad length and next header fields.  Note: The IP header is not covered by the ICV.

ESP in transport mode  In this mode, the ESP header is inserted right after the IP header.  If the datagram already has IPsec header(s), then the ESP header is inserted before any of those.  The ESP trailer and the optional authentication data are appended to the payload.  ESP in transport mode provides neither authentication nor encryption for the IP header.  This is a disadvantage, since false packets might be delivered for ESP processing.  The advantage of transport mode is the lower processing overhead.  As in the case of AH, ESP in transport mode is used by hosts, not gateways.  Gateways are not required to support transport mode.

ESP in transport mode

ESP in tunnel mode  A new IP packet is constructed with a new IP header and ESP is then applied, as in transport mode.  Since the original datagram becomes the payload data for the new ESP packet, it is completely protected, if both encryption and authentication are selected. However, the new IP header is still not protected.  The tunnel mode is used whenever either end of a security association is a gateway. Thus, between two firewalls the tunnel mode is always used. For example two security gateways may operate an ESP tunnel which is used to secure all traffic between the networks they connect together.  Gateways often also support transport mode. This mode is allowed when the gateway acts as a host, that is in cases when traffic is destined to the gateway itself. For example, SNMP commands could be sent to the gateway using transport mode.  In tunnel mode the outer header's IP addresses does not need to be the same as the inner headers' addresses.  Hosts are not required to support tunnel mode.  The advantages of tunnel mode are total protection of the encapsulated IP datagram and the possibility of using private addresses.  However, there is an extra processing overhead associated with this mode.

ESP in tunnel mode

Combining IPSec protocols  The AH and ESP protocols can be applied alone or in combination.  To make things more complicated, the AH and ESP SAs do not need to have identical endpoints.  The combinations of IPsec protocols are realized with SA bundles.  There are two approaches for an SA bundle creation:  Transport adjacency: Both security protocols are applied in transport mode to the same IP datagram. This method is practical for only one level of combination.  Iterated (nested) tunneling: The security protocols are applied in tunnel mode, in sequence. After each application, a new IP datagram is created and the next protocol is applied to it. This method has no limit in the nesting levels. However, more than three levels are impractical.  These approaches can be combined. For example, an IP packet with transport adjacency IPsec headers can be sent through nested tunnels.  When designing a VPN, one should limit the number of IPsec processing stages. In our view, three stages is the limit beyond which further processing has no benefits. Two stages are sufficient for almost all cases.

Combining IPSec protocols  Note: in order to be able to create an SA bundle in which the SAs have different endpoints, at least one level of tunneling must be applied. Transport adjacency does not allow for multiple source/destination addresses, because only one IP header is present.  The practical principle of the combined usage is that, upon the receipt of a packet with both protocol headers, the IPsec processing sequence should be authentication followed by decryption. It is common sense not to bother with decryption of packets of uncertain origin.  Following the above principle, the sender first applies ESP and then AH to the outbound traffic. In fact, this sequence is an explicit requirement for transport mode IPsec processing.  Turning on ESP authentication makes sense only when the ESP SA extends beyond the AH SA, e.g., ESP could be used end-to-end, while AH only goes as far as the remote gateway. In this case, not only does it make sense to use ESP authentication, but it is highly recommended to do so to avoid spoofing attacks within the intranet.  As far as the modes are concerned, transport mode is usually used between the endpoints of a connection and tunnel mode is usually used between two machines when at least one of them is a gateway.

Case 1: End-to-end security  Two hosts are connected through the Internet (or an intranet) without any IPsec gateway between them.  They can use ESP, AH or both.  Either transport or tunnel mode can be applied.  The combinations required to be supported by any IPsec implementation are the following:  Transport Mode  AH alone  ESP alone  AH applied after ESP (transport adjacency)  Tunnel Mode  AH alone  ESP alone

Case 1: End-to-end security

Case 2: Basic VPN support  This is the simplest IPsec VPN.  The gateways G1 and G2 run  the IPsec protocol stack.  The hosts in the intranets are not required to support IPsec.  In this case, the gateways are required to support only tunnel mode, either with AH or ESP.

Case 2: Basic VPN support

Combining tunnels between gateways  Although gateways are required to support either an AH tunnel or ESP tunnel, it is often desirable to have tunnels between gateways that combine the features of both IPsec protocols.  The order of the headers is user selectable by setting the tunnel policy.  A combined tunnel between gateways does not mean that iterated tunneling takes place.  Since the SA bundle comprising the tunnel have identical endpoints, it is inefficient to do iterated tunneling.  Instead, one IPsec protocol is applied in tunnel mode and the other in transport mode, which can be conceptually thought of as a combined AH-ESP tunnel.  An equivalent approach is to IP tunnel the original datagram and then apply transport adjacency IPsec processing to it.  The result is that we have an outer IP header followed by the IPsec headers in the order set by the tunnel policy, then the original IP packet.

Combining tunnels between gateways

Case 3: End-to-end security with VPN support  This case is a combination of cases 1 and 2 and does not raise new IPsec requirements for the machines involved.  The big difference from case 2 is that now the hosts are also required to support IPsec.  In a typical setup, the gateways use AH in tunnel mode, while the hosts use ESP in transport mode.  An enhanced security version could use a combined AH- ESP tunnel between the gateways.  In this way, the ultimate destination addresses would be encrypted, the whole packet traveling the Internet would be authenticated and the carried data double encrypted.  This is the only case when three stages of IPsec processing might be useful, however, at a cost; the performance impact is considerable.

Case 3: End-to-end security with VPN support

AH tunneling of ESP transport  Let us look in more detail at the common combination of using AH tunneling to protect ESP traffic in transport mode.  Consider that host H1 sends an IP packet to host H2.  Here is what happens:  Host H1 constructs the IP packet and applies ESP transport to it. H1 then sends the datagram to gateway G1, the destination address being H2.  Gateway G1 realizes that this packet should be routed to G2. Upon consulting its IPsec databases (SPD and SAD) G1 concludes that AH in tunnel mode must be applied before sending the packet out. It does the required encapsulation. Now the IP packet has the address of G2 as its destination, the ultimate destination H2 being encapsulated.  Gateway G2 receives the AH-tunneled packet. It is destined to itself, so it authenticates the datagram and strips off the outer header. G2 sees that the payload is yet another IP packet (that one sent by H1) with destination H2, so it forwards to H2. G2 does not care that this packet has an ESP header.  Finally H2 receives the packet. As this is the destination, ESP-transport processing is applied and the original payload retrieved.

AH tunneling of ESP transport

Case 4: Remote access  This case applies to remote hosts that use the Internet to reach a server in the organization protected by a firewall.  Between the remote host H1 and the firewall G2, only tunnel mode is required. The choices are the same as in case 2.  Between the hosts themselves, either tunnel mode or transport mode can be used, with the same choices as in case 1.  A typical setup is to use AH in tunnel mode between H1 and G2 and ESP in transport mode between H1 and H2.  It is also common to create a combined AH-ESP tunnel between the remote host H1 and the gateway G2.  In this case H1 can access the whole intranet using just one SA bundle, whereas it only could access one host with one SA bundle.

Case 4: Remote access

HVALA NA PAŽNJI