Download presentation
Presentation is loading. Please wait.
1
Network Processors for a 1 MHz Trigger-DAQ System
RT2003, Montreal Artur Barczyk, Jean-Pierre Dufey, Beat Jost and Niko Neufeld CERN-EP & Université de Lausanne
2
Network Processors Developed for high-end routers, on the market since 1999 Dedicated processors optimised for high speed packet processing Large I/O capabilities (up to 10 Gigabit/s), and up to 10 Mp/s Large and fast buffer memories Software programmable Use them in a network based DAQ system, wherever PCs can’t do it (easily) Niko NEUFELD CERN, EP
3
Anatomy of a Network Processor
On-chip Memory + Interfaces for external memories Control and Monitoring Scheduler Packet Buffer Memory General Purpose CPU Routing and Bridging Tables Processor Complex Integrated Network Interfaces Multiple RISC processor cores Several hardware threads Coprocessors for many common networking tasks Search Engine HW Assist MAC/FRAME Processor To and From PHYs Niko NEUFELD CERN, EP
4
NP module as PCI card All infrastructure to operate one IBM PowerNP NP4GS3 3 x 1000 BaseT ports One port converted into PCI, for development purposes 2 NPs can be connected via special cable Build by S3 corp., Ireland Niko NEUFELD CERN, EP
5
Network Processors in a 1 MHz DAQ
Frame Merging Front-end Electronics FE FE FE FE FE FE FE FE FE FE FE FE TRM Links 1.1 MHz GB/s Multi-stages switching Network 349 Links 40 kHz 2.3 GB/s Decision Sorting Multiplexing Layer Event Building Switch Switch 30 Switches NPs NP NP NP 24 NPs Links GB/s 24 Links 1.5 GB/s Readout Network L1-Decision Sorter TFC System Links GB/s Event Builder NP NP 37-70 NPs NP NP Switch Switch Switch Links GB/s SFC SFC SFC SFC 50-100 SFCs SFC SFC Niko NEUFELD CERN, EP
6
Frame Merging Works up to 4 MHz of incoming packets A. Barczyk’s presentation Works for at least 2 x 100 MB streams Niko NEUFELD CERN, EP
7
Frame Merging Helps to optimise link usage
Reduces number of links into readout network Can do re-formatting of data – e.g. protocol adaptation (raw Ethernet IP) Can change Maximum Transmission Unit (MTU) some Ethernet segments provide payload > 1500 bytes Reduce packet rate at output - important for receiving PCs (interrupt rate!) Niko NEUFELD CERN, EP
8
Building your own switching network from NP modules
Using NP modules gives you full freedom in doing the switching Large output buffers Disadvantage: Module has only eight ports (otherwise switch chip is needed) need a large number of modules to build a big network Solution: Use optimised connection topologies to reduce number of elementary modules, while keeping the load on interconnecting links acceptable Niko NEUFELD CERN, EP
9
Network Topologies Banyan Topology “Fully Connected” Topology
64 x 64 port configuration Basic Structure Sources 63 Sources x 72 Destinations Destinations Niko NEUFELD CERN, EP
10
Decision Sorting In the LHCb trigger decisions are generated as small Ethernet packets in one of 1400 PCs 1 MHz of un-ordered decisions in Processing time limited but unknown decisions are taken and sent in arbitrary order Front-end electronics requires decisions to be ordered before sent to the trigger distribution system 1 MHz of ordered decisions out Limited buffer size entails maximum trigger latency Each event entering is made known to central entity (Decision Sorter) 1 MHz of frames Niko NEUFELD CERN, EP
11
Decision Sorting 1 2 3 4 3 4 Readout Network 1 2 3 4 6 5 2 4 1 2 1 3
Front-end Electronics FE FE FE FE FE FE TRM TFC System 3 4 Readout Network L1-Decision 1 2 3 4 6 Links GB/s 5 2 4 SFC SFC SFC 90-153 SFCs SFC SFC 1 2 CPU Farm Switch CPU Switch CPU Switch CPU Switch CPU Switch CPU ~1400 CPUs 1 3 Sorter Niko NEUFELD CERN, EP
12
Measurement Procedure
Test Set-up Measurement Procedure Dual NP connected via back-plane to form 8-port module Download code into NP4GS3 via RISC Watch (JTAG) or PCI of PPC control point Generate traffic either via Gigabit Ethernet NICs (Tigon) or using one NP to feed the other Can use the internal timers of the NP and/or the NICs 4 x Tigon SX 8 x 1000 SX Full Duplex Ports Tigon2 NIC Features Up to 620 kHz fragment rate 1 s resolution timer IBM NP4GS3R2.0 Reference Kit PPC 750 RISC Watch = JTAG via Ethernet Niko NEUFELD CERN, EP
13
Network Processors Processing power optimised for header region of packets Memory model optimised for the hardware (no linear addressing) Programs need to be written in proprietary assembly language Packet processing for several millions of packets per second Fast and big buffer memories Hardware assists for many common tasks, like check-summing, re-framing, tree look-ups Software programmable Niko NEUFELD CERN, EP
14
Conclusions Network Processors are a powerful tool for packet processing They are especially useful, whenever very high rates of packets need to be coped with We have found a lot of useful applications, all could be done with the same standard NP module – the software defines the functionality But what if PCs can do it too…? Niko NEUFELD CERN, EP
15
Backup Slides Niko NEUFELD CERN, EP
16
Data flow in the NP4GS3 Ingress Event Building Egress Event Building
DASL DASL Access to frame data Access to frame data Ingress Event Building Egress Event Building Niko NEUFELD CERN, EP
Similar presentations
