Measuring packet forwarding behavior in a production network Lars Landmark
Temporarily file
Initial interest Multicast and Unicast, can we measure differences in packet forwarding behavior in a production network?
Packet forwarding within a router Fast switching (fast path) – Fast switched packets benefit from parallel processing, cache lookup without the need of consulting main router tables and forwarding at interrupt level. Process switching (slow path). – Packets not eligible for fast switching must wait in the central processor input queue until they are scheduled for processing. Hence, process switched packets are associated with a higher router delay.
Today routers mainly "fast switch" their traffic applying dedicated hardware. However "process switching" which involves the router OS may occur occasionally before caches in hardware modules are updated. In case of failures or miss configuration, large portion of traffic is process switched.
Measurement Technique for Jitter Measurement packets were time stamped at source and destination using DAG cards, and transmitted with a 10ms interval. Given equal packet forwarding within each router, difference in jitter is caused by cross traffic. Variation in jitter is caused by: – Cross traffic, queuing time – Dissimilar packet forwarding Equal spaced modes due to constant processing time.
TEST-BED The measurements were performed over UNINETT, the Norwegian national research and educational network (NREN) which connects universities, colleges and research institutions to the Internet. The UNINETT core interconnects the main Norwegian cities with 100, 10 and 2.5 gigabit per second (Gbps) links in ring structures. Capacity on access links to institutions varies from 1 Gbps and upwards.
Topology for our measurement path Network equipment involved in our measurement path. We acquired two measurement set: – DAG. – Linux and the application Crude.
Measured E2E delay Our E2E delay measurements show similarity to uniform distribution, which is not typical seen for queuing delay. This delay distribution is caused by internal scheduling from one or more routers. Multicast show similar distribution to unicast, but is added 50 microseconds delay.
Observed Jitter for multicast and unicast for DAG cards No difference between Multicast and Unicast. Jitter modes are spaced by 60 micro-seconds. Large amount of process switched traffic was revealed after a closer router inspection. The modes on both side of zero show the number of packets in queue for being process switched. The mode width illustrates cross traffic.
Jitter observation acquired from DAG and Linux measurement. Linux and DAG provide similar measurement results when for Jitter.
Observed Jitter after the replacement of the misbehaved router No process switched traffic observed after the replacement of the misbehaved router.
Conclusion Flawed routers with suboptimal packet forwarding is inferred by applying commonly available measurement techniques and analysis methods. Such flawed routers are not necessarily detected by ordinary network management routines since no alarms are generated even though packets may be lost. Our multi-mode jitter observation turns out to reveal the difference between fast and process switched packets. The main mode presents fast switched packets, while additional modes show the probability of being process switched. The time distance between the modes represent the processing time for process switching. Two measurement methods were also explored, software acquisition at the Linux application layer and acquisition with high precision hardware (DAG cards). DAG and Linux delivered similar probability distributions for jitter measurements, suggesting that low cost software tools may be sufficient in this context.