Proposal for a “Switchless” Level-1 Trigger Architecture Jinyuan Wu, Mike Wang June 2004

L1 Block Diagram L1 Switch TSO PP L1B Servers ST L1B BM CPU TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST BM CPU BM CPU BM CPU BM CPU BM CPU BM CPU BM CPU Pixel Data Time Stamp Ordering Cluster Processing Raw Data to L1B Data Sharing Segment Finding Event Building Track & Vertex Processing L1B Servers L1B GL1 Node Triplets to L1B Tracks & Vertices to L1B Trigger primitives to GL1

Use 4 bit arrays. There are 3 constraints total. More constraints help to eliminating fake tracks. It is possible to use bit-wise majority logic (such as 3-out- of-4) to accommodate detector inefficiency issues. Pentlet Finding An Example of Possible Algorithm Plane APlane CPlane EPlane BPlane D

Must Take Care Everything: Event building, an easy part. Rescaling flexibility. To L1B: –Raw data & cluster indices. –Triplets. –Tracks & vertices. Interface with COT farm nodes. To GL1. Types of modules build or buy – the less, the better. Types of links – the less, the better. Anything else?

Gang Things Together & Reroute Cables L1 Switch TSO PP L1B Servers ST L1B BM CPU TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST TSO PP ST BM CPU BM CPU BM CPU BM CPU BM CPU BM CPU BM CPU L1B Servers L1B GL1 Node TSO PP Events are built during TSO & PP processing. ST ST/L1B L1 Switch is gone, its functions are absorbed in TSO & PP stages. Triple links are not needed. L1B for raw data, triplets & etc.

System Interconnection Time Stamp Ordering Module Pixel Pre-processor Module Segment Tracker & L1B Module Buffer Manager Module L1B Server PC Worker Farm Node

TSO & PP: Event Building Time Stamp Ordering Module: 12 fiber in, 16 out (4 pairs/out) Pixel Pre-processor Module: 10 in, 8 out (4 pairs/IO) Each highway contains 10 Time Stamp Ordering Modules. (Receiving 10 cables, 12 fibers/cable, 120 fibers) Each TSO module combines 12 fibers and sends to 16 outputs based on BCO. A Pixel Pre-processor reads 10 inputs from 10 TSO modules, combines them and sends to 8 outputs based on BCO. Each output of PP contains whole detector (120 fibers) information, but 1/1024 BCO (or 1/128 hwy). BCO: HwyTSOPP

Questions: Will switch functions increase cost of TSO & PP? –Data rate at TSO & PP (raw data+cluster indices) is indeed several times higher than after ST (triplets). –Functional blocks are compiled and simulated for Altera Cyclone devices. See: Doc # 2907, –The switch functions can be absorbed into TSO & PP without noticeable cost increase. Is it a problem for the FPGA to receive many independent serial data sources? –It is not a problem. –Functional blocks are compiled and simulated for Altera Cyclone devices.

Merging Switch & User Functions User Func. User Func. User Func. User Func. User Func. User Func.

Segment Tracker, L1B etc. Segment Tracker & L1B Module Buffer Manager Module L1B Server PC Worker Farm Node A ST/L1B module receives two inputs from PP with whole detector information contained. A L1B Sever PC hosts 4 ST/L1B modules. Triplets are sent through BM to Worker Farm Nodes. See next page for detail operations.

ST/L1B and BM Interconnection BM Hash Sorter SDRAM SRAM-ZBT 128K x 32 ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic (4) Triplets are sent to BM and Worker Node (1) Raw data and cluster (x,y) are input. (2) Raw data and cluster (x,y) are stored in SDRAM. (3) Triplets are produced in ST, and stored in SDRAM (5) Worker Node finds tracks, vertices, etc. (6) Tracks, vertices, etc are sent back to L1B. (7) Raw data, cluster (x,y), triplets tracks, vertices, etc are read out from L1B.

A possible GL1 Interconnection BM Hash Sorter SDRAM SRAM-ZBT 128K x 32 ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic GL1 Interface SDRAM SRAM-ZBT 128K x 32 BM Hash Sorter SDRAM SRAM-ZBT 128K x 32 (1) GL1 are sent out by the Worker Nodes. (2) The TSO modules can be used as concentrators. (3) The same BM module is used as GL1 interface. GL1 Node Farm Nodes ST/L1B

Building Blocks: Option 1 Time Stamp Ordering Module Pixel Pre-processor Module Segment Tracker & L1B Module Buffer Manager Module

Building Blocks: Option 2 Time Stamp Ordering Module Pixel Pre-processor Module Segment Tracker & L1B Module Buffer Manager Module

Time Stamp Ordering Module Pixel Pre-processor Module

Rescaling available processing power Original baseline: –Switch located after the segment tracker hardware –Primarily so that event building could be accomplished (routing packets belonging to the same BCO to one CPU). –With this design, it was also easy to rescale the processing power of the track/vertex farm by adding or subtracting nodes: To suit processing requirements To accommodate a fault tolerant design Proposed “switchless” design: –Also allows rescaling of track/vertex farm –In addition, even pixel processors (PP) and segment trackers (ST) can easily be rescaled

Rescaling: Pixel Pre-processor Pixel Pre-processor can be added/removed 1-by-1. Minimum system at the early commissioning stage needs only 1 PP module. Broken PP cards during operation can be tolerated: just reroute data to other outputs on the TSO modules. ST and farm nodes in later stages scale accordingly.

Rescaling: Relative Numbers of Modules In addition to the rescaling of PP modules, the relative numbers of PP, ST and BM modules can be adjusted. Numbers of I/O: –PP module: 8 outputs. –ST module: 2 inputs, up to 3 outputs. –BM module: up to 4 inputs. Different interconnection configurations allow different ratios of number of modules. BM Hash Sorter SDRAM SRAM-ZBT 128K x 32 ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic

Rescaling: Relative # of ST & Nodes The BM/ST ratios: 1/4, 2/4, 3/4, 4/4, 6/4, 8/4, 12/4. BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B ST L1B ST L1B ST L1B ST L1B BM Worker BM Worker BM Worker ST L1B ST L1B ST L1B ST L1B BM Worker BM Worker BM Worker 64 ST, 32 Nodes 64 ST, 48 Nodes

Rescaling: If One Node Breaks: If one farm node breaks, data should be rerouted to other nodes. The loads are to be shared by other nodes. Both “push-only” and “request-read” schemes are possible. BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B If this node is broken. These ST will reroute data. These nodes runs with 25% additional loads. BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B BM Worker ST L1B ST L1B

Conclusion: Event building: support more algorithms. Rescaling flexibilities: support extra flexibilities. To L1B: everything is sent to one location. –Raw data & cluster indices. –Triplets. –Tracks & vertices. Interface with COT nodes: PCI(-express). To GL1: dedicated connections are possible. Types of modules: build 2, buy 2 (types of PC). Types of links: 1, (FiniteBand?). Anything else? Happiness of everyone:.

Rescaling: Using Cable Bundles: Each cable has 4 pairs. Use cable bundles to achieve extra routing flexibility. 4x4: single directional, 2x2: bi-directional

Time Stamp Ordering Module Input from PDCB, 12 Fibers Control & Monitor Port 16 Outputs to Pixel Pre-processor, 4 pairs each. Free Format Euro Card/ VME Format

Pixel Pre-processor Module 10 Inputs from TSO, 4 pairs each 8 Outputs to Segment Tracker & L1B, 4 pairs each. Free Format Euro Card/ VME Format Control & Monitor Port

Segment Tracker/L1B Module ST 2-in 2-out SDRAM SRAM-ZBT 128K x 32 L1B Logic

Buffer Manager Module BM Hash Sorter SDRAM SRAM-ZBT 128K x 32