Network Processors Harsh Chilwal

900MHz Voice 1G 900MHz 1800MHz Voice 2G MHz Smart Phone Full web service 3G MHz Voice Tiny Internet 2.5G 12 kb/s Data Rate Evolution : Cellular phone generation

Evolution : 3G cellular phones base station (BS) mobile station (MS) base station controller (BSC) 12Kb/secon d 5Mb/second 100Mb/second Network 100 MS 10 BS

base station (BS) mobile station (MS) base station controller (BSC) 1Mb/second 500Mb/second 50Gbit/second Network Evolution : 3G cellular phones 500 MS 100 BS NP

DS0 Year Bandwidth (Mb/s) 64K 1.5M DS DS3 44Mb OC12 622Mb x24 x28 x12 10Gb x16 10,000 OC ,000 x4 OC768 40Gb 2005 Evolution : Networks NP DS= Digital signalOC = Optical carrier

Networking Trends  Increasing Networking Traffic.  New sophisticated protocols are being introduced at rapid pace.  Need for supporting new applications to provide new services.  Convergence of voice and data networks introducing a lot of changes in the communication industry.  Increasing TTM Pressures  Decreasing product life cycles.

General Purpose Processor based Software Router  Benefits  Flexible for upgrading the system  Easy for supporting additional interfaces  Quick to develop new products with short TTM.  The core processor performs all the routing functionalities  Drawbacks  Not able to scale up for higher bandwidths, maximum up to OC-12 speeds only  Can support complex network operations viz., traffic engineering, QoS, etc  with a major reduction in performance

ASIC based Routers » Benefits  Provide wire-speed performances  provided high speed » Drawbacks  Lacks flexibility; difficult to meet changing market needs/demands  Long design cycles increases TTM reduces PLC.  Change in design or failure in design involves more risks  Need to replace the ASIC to provide new functionality  Complex network operation are still executed in software

Network Processor based boxes  Promises to provide performance and flexibility  Comprises of many packet processing elements supporting multiple threads  Achieves higher performance by pipelining and parallel processing both in terms of threads and packet processing elements  Brings-in flexibility by due software programming  Easy to add features

Network Processor

Basic Architecture of Network Processors

Basic architecture (contd.) Dispatcher Merger CP2CP1CP3CP4 Look-A-Side Co-processors Risc Com – Engine Multiple Streams

Intro: Systems and Protocols: Relation with Standards IETF / Forces WG: Data / Forwarding Plane Control Plane NPF: Service Layer System Wide No awareness where things are Functional Layer Awareness where things are Operational Layer Interface Management ITU-T/ANSI/ATM Forum: ATM IEEE Ethernet IETF/Protocols IPv4 MPLS PPP/L2TP IPv6 MIBs Protocols Systems

OSI Network Architecture DATA Application Pre. Session Transport Network Data Link Physical DATAAH DATAPH DATASH DATATH DATANH DATADH DATAPH Application Pre. Session Transport Network Data Link Physical Network AB

Typical Applications  WAN/LAN Switching and Routing, Multi- service Switches, Multi-layer switches, Aggregators  Web caching, Load balancing, Web switching, Content based load balancers  QoS solutions  VoIP Gateways  2.5G and 3G wireless infrastructure equipments  Security - Firewall, VPN, Encryption, Access control  Storage solutions  Residential Gateways

Software Framework

Scene setting - why specs are not enough  2 NPU vendors want to promote their solution with some ‘numbers’  Both chip architectures comprise –RISC engines –Hardware support engines –Various types of interfaces –Support for internal and external memory  They report the following data –Aggregate MIPS –Max number of lookups per second –... Commonalties in building blocks Commonalties in building blocks Commonalties in specifications Commonalties in specifications Commonalties in Interpretation? Commonalties in Interpretation?

Specifications

Test scenario  What is measured? Performance in packets per second versus a forwarding information base (FIB) that is increased in size.  Start application is IPv4.  Next, counters are added for per flow billing purposes.  Next, load balancing is introduced as an additional feature.  Finally, encryption becomes an additional requirement for 2% of the data that is being forwarded

Performance curves FIB (K entries) FIB (K entries) Performance (Mpps) Performance (Mpps) NPU B NPU A IPv4

Performance curves FIB (K entries) FIB (K entries) Performance (Mpps) Performance (Mpps) NPU B NPU A IPv4 + counters Requires more memory references Requires more memory references

Performance curves FIB (K entries) FIB (K entries) Performance (Mpps) Performance (Mpps) NPU B NPU A IPv4 + counters + Load balancing Requires even more memory references Requires even more memory references

Performance curves FIB (K entries) FIB (K entries) Performance (Mpps) Performance (Mpps) NPU B NPU A IPv4 + counters + Load balancing + encryption No extra references and resources available No extra references and resources available A does not have sufficient resources A does not have sufficient resources

Architecture A Key extract Key extract LU Count Int. mem Int. mem 3 MIPS cores 3 MIPS cores Int. mem Int. mem Int. mem Int. mem External Buffer Mem External Buffer Mem Sched OC-192 POS Hash IPv4 + counters + LB + crypto IPv4 + counters + LB + crypto

Architecture B IPv4 + counters + LB + crypto IPv4 + counters + LB + crypto LB 10 MIPS cores 10 MIPS cores External Buffer Mem External Buffer Mem 10GE Memory interface IMEM

Specifications - revisited

So  No clear value statement could be made in favor of either NPU solutions –NPU A achieves higher throughput but with limited flexibility –NPU B achieves lower throughput but is more flexible  Were the provided specs accurate? –Yes. –The devices performed up to spec. –Although NPU B looks better on paper at first sight, more resources have to be consumed for less per formant results. –There is a cost associated with flexibility  Were the provided specs relevant? –No. They represent granular maximum performances. –For ‘real world’ applications,  some resources could not be maximally consumed  some resources were over consumed

Benchmarking considerations  Processor core metrics are not always relevant for networking applications –It might be relevant for NPU B, since functionality relies almost totally on those cores. –It is definitely not the case for NPU A, since there is extensive additional hardware support for specific functions. GRANULARITY Highly granular specifications, data or benchmarking information can offer a wrongful picture of the actual performance capabilities of the DUT. Since Network Processing Devices are designed with specific applications in mind, benchmarks must exist for those specific applications GRANULARITY Highly granular specifications, data or benchmarking information can offer a wrongful picture of the actual performance capabilities of the DUT. Since Network Processing Devices are designed with specific applications in mind, benchmarks must exist for those specific applications

Benchmarking considerations  External factors affect NPD performance (where you don’t always suspect it) –A forwarding application relies on FIB lookups to determine the destination of a packet –The size of the FIB table can influence performance in many ways  Usage of multiple memory banks  increasing number of hash collisions EXTERNAL FACTORS Benchmarks should include parameters that take into account external factors that are relevant to the particular applications that are being benchmarked. EXTERNAL FACTORS Benchmarks should include parameters that take into account external factors that are relevant to the particular applications that are being benchmarked.

Benchmarking considerations  Interfaces present performance boundary conditions –Ethernet applications require inter frame gaps that result in more relaxed pps numbers INTERFACES Benchmarks should also specify the types of interfaces that are being used since those interfaces have an impact all by themselves on maximum performance figures INTERFACES Benchmarks should also specify the types of interfaces that are being used since those interfaces have an impact all by themselves on maximum performance figures

Benchmarking considerations  Combinations of applications or minor extensions have a completely different impact on both network processing devices –NPU A has a lot of well engineered hardware support that can offer additional services BUT fails almost completely when additional computing resources are required –NPU B is very ‘soft’; performance degrades slowly when additional services are requested and shows no abrupt peaks in the performance curves. HEADROOM Benchmarks should combine applications as they occur in the real world to give a ‘sense’ of headroom that is available to support real world scenarios. It is however very hard to define a metric for headroom HEADROOM Benchmarks should combine applications as they occur in the real world to give a ‘sense’ of headroom that is available to support real world scenarios. It is however very hard to define a metric for headroom

CommBench – A Telecommunication Benchmark For NPs CommBench HPAs PPAs  RTR  FRAG  DRR  TCP  CAST  ZIP  REED  JPEG

Benchmark Characteristics – Code & Computational Kernel Sizes

Benchmark Characteristics – Computational Complexity N a,l – Num Of Instructions/byte required for app a operationg on a packet of length l

Benchmark Characteristics – Instruction Set Characteristics

Benchmark Characteristics – Memory Hierarchy

Example System: Cisco Toaster  Almost all data plane operations execute on the programmable XMC  Pipeline stages are assigned tasks – e.g. classification, routing, firewall, MPLS –Classic SW load balancing problem  External SDRAM shared by common pipe stages

Example System: IXP 2400  XScale core replaces StrongARM  Microengines –Faster –More: 2 clusters of 4 microengines each  Local memory  Next neighbor routes added between microengines  Hardware to accelerate CRC operations and Random number generation  16 entry CAM ME0ME1 ME2ME3 ME4ME5 ME6ME7 Scratch /Hash /CSR MSF Unit DDR DRAM controller XScale Core QDR SRAM controller PCI

References  Network Processor Design – Patrick Crowley etal.  CommBench - A Telecommunications Benchmark for Network Processors, Tilman Wolf and Mark Franklin. Proceedings of IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS),  Network Processing Forum - Benchmarking Network Processing Forum - Benchmarking  