Evolution of S-LINK to PCI interfaces W. Iwanski (Henryk Niewodniczanski Institute of Nuclear Physics) M. Joos, R. McLaren, J. Petersen, E. van der Bij (CERN)

W.Iwanski LECC, Colmar 9-13 September 2002 System evolution Yesterday Tomorrow Today 4 S-LINKs 4 S-LINKs S-LINK S-LINK S-LINK S-LINK SSP SSP FILAR FILAR S32PCI64 S32PCI64 slow PCI fast PCI fast PCI Eth. MEM MEM slow PCI MEM slow PCI Eth. Eth. CPU CPU CPU PC PC PC W.Iwanski LECC, Colmar 9-13 September 2002

Current use of S-Link to PCI interfaces in test-beams Muon test beam TileCal test beam VMEbus VMEbus VMEbus VMEbus VMEbus SPS SPS SPS SPS SPS S-LINK S-LINK S-LINK S-LINK S-LINK SSPCI SSPCI SSPCI SSPCI SSPCI PC PC Features of SSP / SSPCI Simple hardware design (AMCC S5933 PCI controller) Max. speed = 80 (RIO2) / 132 (PC) MB/s Event overhead = ~8 us One ROL per PMC / PCI slot W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 S32PCI64 32/64 bit PCI 33/66 MHz 3.3 V PCI slots only 3.3 V S-LINK LDC plug-in W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Overview of S32PCI64 S L N K PCI PCI 32 to Map 32 INPUT INPUT BURST L DMA I to BUFFER BUFFER FIFO FIFO I 64 64 FIFO FIFO ENGINE 1024 x 64-bit (1024 x 64 - bit) 128x 64 128x 64 - bit REQUEST REQUEST FIFO FIFO PCI address, length 64 64 - BIT - BIT 33/66 MHz 32/64 - 32/64-bit PCI PCI BACKEND PCI ACKNOWLEDGE CONTROL BACK END FIFO CORE CORE LOGIC ctrl words, length FIFO LOGIC CONTROL, STATUS & CONTROL, INTERRUPT STATUS & REGISTERS FPGA Local logic Commercial IP Core W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Features of S32PCI64 32/64 bit, 33/66 MHz PCI (3.3 V slots only) 32-bit S-LINK (3.3 V LDC plug-in only) 32-bit PCI-bus addressing No Initiator wait states during a burst Max. 64 Mbytes long DMA transfer Protocol overhead per event: 5 PCI single cycles Based on a commercial PCI IP core (PLDA) Highly autonomous data reception Interrupt generation selectable on several conditions Data packets longer than 1kB are segmented W.Iwanski LECC, Colmar 9-13 September 2002

Features and applications of S32PCI64 based systems Less load on the CPU to handle link protocol event overhead =~2.5 us 3.3 V PCI slots required For applications requiring full ATLAS input rate Applications: CosmoDAQ mainly for prototypes Commercially available from NE (Nowoczesna Elektronika) W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 S32PCI64 test set-up Elonex PC SuperMicro 370DLE motherboard memory bandwidth > 528 MB/s SLIDAS data generator VMEtro PBT-515BX PCI Bus Analyser PCI bandwidth exerciser (PCI-Blaster) SLIDAS SLIDAS 160 MB/s 160 MB/s S32PCI64 S32PCI64 PCI 66 MHz/ 64 bit ServerSet III LE CPU MEM PC W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 PCI-Blaster PCI bandwidth exerciser firmware designed and implemented on S32PCI64 board to benchmark PCI bridge and memory in PC Features full speed PCI capability (528 MB/s) read and write capability read and write modes can be set-up simultaneously simple programming model data transfers in specified number of times or in infinite loop 32-bit and 64-bit PCI bus (3.3V PCI bus only) 33 and 66 MHz MHz PCI clock speed 32-bit PCI-bus addressing no interrupt capability W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Overall throughput W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Other measurements Overhead of S32PCI64 hardware in single access from host CPU to S32PCI64 write -> 30 ns (2 wait states) read -> 45 ns (3 wait states) Minimal time interval between 2 consecutive single commands seen on PCI bus (limited by CPU/PCI bridge) write-write -> 75 ns write-read -> 105 ns read-read -> 330 ns read-write -> 345 ns Gap between DMA sub-bursts (packets > 1kB) -> 180 ns W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 FILAR Four Integrated Links for ATLAS Readout LDC protocol FPGA S32PCI64 PCI 64 bit / 66 MHz 3.3 V Ser- des ROL Pluggable f/o transceiver W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Main features of FILAR Four 2.5 Gbit/s HOLA S-LINK Destinations integrated Data channels individually enabled/disabled 32/64-bit 33/66 MHz PCI bus 32-bit S-LINK 32-bit PCI-bus addressing Protocol overhead per event and channel: 2 PCI single cycles Highly autonomous data reception Card temperature readout Module scheduled for 1Q/2003 W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Steps towards FILAR New library written in ROS context FILAR emulator: simplified specification of FILAR implemented in existing hardware (S32PCI64) Data from one S-link connector copied to all four data channels One common flow control signal logical OR of all four XOFF signals W.Iwanski LECC, Colmar 9-13 September 2002

Features of FILAR software Package consists of Linux driver and user library Detection of data packets based on interrupts Support for: multiple PCI cards multiple channels Code optimised for low overhead ~1.5 us / event Current API will also be valid for final FILAR W.Iwanski LECC, Colmar 9-13 September 2002

Performance of FILAR emulator W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Performance (cont’d) Improved performance of FILAR emulator for small packets with respect to S32PCI64 Better performance of FILAR emulator running one data channel than that in S32PCI64 Performance of FILAR emulator running two, three or four data channels is compromised by board limitation flow control signal, working for all channels here, stops new data to whole interface whenever any of data channels is getting full. It prevents other, already empty buffers from being re-filled gaps up to 4.5 us seen between some DMAs If necessary, (still) costly single PCI cycles will be replaced by DMA W.Iwanski LECC, Colmar 9-13 September 2002

W.Iwanski LECC, Colmar 9-13 September 2002 Summary Transition from 32-bit/33 MHz to 64-bit/66 MHz PCI full bandwidth of S-LINK (160 MB/s) Stable design: evolution from S32PCI64 to FILAR great part of software re-used Use of FPGA and IP core in designs easy debugging, testing and upgrading re-usability of existing hardware PCI-Blaster FILAR emulator Powerful and cost optimised FILAR design integration of 4 links in one card reduces costs and overcomes a limit of fast 66 MHz PCI slots in PC pluggable optical transceivers further reduce costs when less than 4 data channels are needed W.Iwanski LECC, Colmar 9-13 September 2002