1. Evolution of S-LINK to PCI interfaces
W. Iwanski
(Henryk Niewodniczanski Institute of Nuclear Physics)
M. Joos, R. McLaren, J. Petersen, E. van der Bij
(CERN)

2. W.Iwanski LECC, Colmar 9-13 September 2002
System evolution
Yesterday
Tomorrow
Today
4 S-LINKs
4 S-LINKs
S-LINK
S-LINK
S-LINK
S-LINK
SSP
SSP
FILAR
FILAR
S32PCI64
S32PCI64
slow PCI
fast PCI
fast PCI
Eth.
MEM
MEM
slow PCI
MEM
slow PCI
Eth.
Eth.
CPU
CPU
CPU PC PC PC
W.Iwanski LECC, Colmar 9-13 September 2002

3. Current use of S-Link to PCI interfaces in test-beams
Muon test beam
TileCal test beam
VMEbus
VMEbus
VMEbus
VMEbus
VMEbus
SPS
SPS
SPS
SPS
SPS
S-LINK
S-LINK
S-LINK
S-LINK
S-LINK
SSPCI
SSPCI
SSPCI
SSPCI
SSPCI PC PC
Features of SSP / SSPCI
Simple hardware design (AMCC S5933 PCI controller)
Max. speed = 80 (RIO2) / 132 (PC) MB/s
Event overhead = ~8 us
One ROL per PMC / PCI slot
W.Iwanski LECC, Colmar 9-13 September 2002

4. W.Iwanski LECC, Colmar 9-13 September 2002
S32PCI64
32/64 bit PCI
33/66 MHz
3.3 V PCI slots only
3.3 V S-LINK LDC plug-in
W.Iwanski LECC, Colmar 9-13 September 2002

5. W.Iwanski LECC, Colmar 9-13 September 2002
Overview of S32PCI64 S L N K
PCI
PCI 32 to
Map 32
INPUT
INPUT
BURST L
DMA I to
BUFFER
BUFFER
FIFO
FIFO I 64 64
FIFO
FIFO
ENGINE
1024 x 64-bit
(1024 x 64 -
bit)
128x 64
128x 64 -
bit
REQUEST
REQUEST
FIFO
FIFO
PCI
address, length 64 64 -
BIT -
BIT
33/66 MHz
32/64 -
32/64-bit
PCI
PCI
BACKEND
PCI
ACKNOWLEDGE
CONTROL
BACK
END
FIFO
CORE
CORE
LOGIC
ctrl words, length
FIFO
LOGIC
CONTROL,
STATUS &
CONTROL,
INTERRUPT
STATUS &
REGISTERS
FPGA
Local logic
Commercial IP Core
W.Iwanski LECC, Colmar 9-13 September 2002

6. W.Iwanski LECC, Colmar 9-13 September 2002
Features of S32PCI64
32/64 bit, 33/66 MHz PCI (3.3 V slots only)
32-bit S-LINK (3.3 V LDC plug-in only)
32-bit PCI-bus addressing
No Initiator wait states during a burst
Max. 64 Mbytes long DMA transfer
Protocol overhead per event: 5 PCI single cycles
Based on a commercial PCI IP core (PLDA)
Highly autonomous data reception
Interrupt generation selectable on several conditions
Data packets longer than 1kB are segmented
W.Iwanski LECC, Colmar 9-13 September 2002

7. Features and applications of S32PCI64 based systems
Less load on the CPU to handle link protocol
event overhead =~2.5 us
3.3 V PCI slots required
For applications requiring full ATLAS input rate
Applications:
CosmoDAQ
mainly for prototypes
Commercially available from NE (Nowoczesna Elektronika)
W.Iwanski LECC, Colmar 9-13 September 2002

8. W.Iwanski LECC, Colmar 9-13 September 2002
S32PCI64 test set-up
Elonex PC
SuperMicro 370DLE motherboard
memory bandwidth > 528 MB/s
SLIDAS data generator
VMEtro PBT-515BX PCI Bus Analyser
PCI bandwidth exerciser
(PCI-Blaster)
SLIDAS
SLIDAS
160 MB/s
160 MB/s
S32PCI64
S32PCI64
PCI 66 MHz/ 64 bit
ServerSet III LE
CPU
MEM PC
W.Iwanski LECC, Colmar 9-13 September 2002

9. W.Iwanski LECC, Colmar 9-13 September 2002
PCI-Blaster
PCI bandwidth exerciser
firmware designed and implemented on S32PCI64 board to benchmark PCI bridge and memory in PC
Features
full speed PCI capability (528 MB/s)
read and write capability
read and write modes can be set-up simultaneously
simple programming model
data transfers in specified number of times or in infinite loop
32-bit and 64-bit PCI bus (3.3V PCI bus only)
33 and 66 MHz MHz PCI clock speed
32-bit PCI-bus addressing
no interrupt capability
W.Iwanski LECC, Colmar 9-13 September 2002

10. W.Iwanski LECC, Colmar 9-13 September 2002
Overall throughput
W.Iwanski LECC, Colmar 9-13 September 2002

11. W.Iwanski LECC, Colmar 9-13 September 2002
Other measurements
Overhead of S32PCI64 hardware in single access from host CPU to S32PCI64
write -> 30 ns (2 wait states)
read -> 45 ns (3 wait states)
Minimal time interval between 2 consecutive single commands seen on PCI bus (limited by CPU/PCI bridge)
write-write -> 75 ns
write-read -> 105 ns
read-read -> 330 ns
read-write -> 345 ns
Gap between DMA sub-bursts (packets > 1kB) -> 180 ns
W.Iwanski LECC, Colmar 9-13 September 2002

12. W.Iwanski LECC, Colmar 9-13 September 2002
FILAR
Four Integrated Links for ATLAS Readout
LDC
protocol
FPGA
S32PCI64
PCI 64 bit / 66 MHz
3.3 V
Ser-
des
ROL
Pluggable f/o
transceiver
W.Iwanski LECC, Colmar 9-13 September 2002

13. W.Iwanski LECC, Colmar 9-13 September 2002
Main features of FILAR
Four 2.5 Gbit/s HOLA S-LINK Destinations integrated
Data channels individually enabled/disabled
32/64-bit 33/66 MHz PCI bus
32-bit S-LINK
32-bit PCI-bus addressing
Protocol overhead per event and channel: 2 PCI single cycles
Highly autonomous data reception
Card temperature readout
Module scheduled for 1Q/2003
W.Iwanski LECC, Colmar 9-13 September 2002

14. W.Iwanski LECC, Colmar 9-13 September 2002
Steps towards FILAR
New library written
in ROS context
FILAR emulator: simplified specification of FILAR implemented in existing hardware (S32PCI64)
Data from one S-link connector copied to all four data channels
One common flow control signal
logical OR of all four XOFF signals
W.Iwanski LECC, Colmar 9-13 September 2002

15. Features of FILAR software
Package consists of Linux driver and user library
Detection of data packets based on interrupts
Support for:
multiple PCI cards
multiple channels
Code optimised for low overhead
~1.5 us / event
Current API will also be valid for final FILAR
W.Iwanski LECC, Colmar 9-13 September 2002

16. Performance of FILAR emulator
W.Iwanski LECC, Colmar 9-13 September 2002

17. W.Iwanski LECC, Colmar 9-13 September 2002
Performance (cont’d)
Improved performance of FILAR emulator for small packets with respect to S32PCI64
Better performance of FILAR emulator running one data channel than that in S32PCI64
Performance of FILAR emulator running two, three or four data channels is compromised by board limitation
flow control signal, working for all channels here, stops new data to whole interface whenever any of data channels is getting full. It prevents other, already empty buffers from being re-filled
gaps up to 4.5 us seen between some DMAs
If necessary, (still) costly single PCI cycles will be replaced by DMA
W.Iwanski LECC, Colmar 9-13 September 2002

18. W.Iwanski LECC, Colmar 9-13 September 2002
Summary
Transition from 32-bit/33 MHz to 64-bit/66 MHz PCI
full bandwidth of S-LINK (160 MB/s)
Stable design: evolution from S32PCI64 to FILAR
great part of software re-used
Use of FPGA and IP core in designs
easy debugging, testing and upgrading
re-usability of existing hardware
PCI-Blaster
FILAR emulator
Powerful and cost optimised FILAR design
integration of 4 links in one card reduces costs and overcomes a limit of fast 66 MHz PCI slots in PC
pluggable optical transceivers further reduce costs when less than 4 data channels are needed
W.Iwanski LECC, Colmar 9-13 September 2002