1. The Read Out Driver for the ATLAS Muon Precision Chambers
The MROD
The Read Out Driver for the ATLAS Muon Precision Chambers
Marcello Barisonzi, Henk Boterenbrood,
Peter Jansweijer, Gerard Kieft, Jos Vermeulen
NIKHEF, Amsterdam
Adriaan König,Thei Wijnen
NIKHEF and Univ. of Nijmegen, Nijmegen
11 September 2002
LECC 2002, Colmar

2. Contents MROD System Overview MROD-0 Feasibility Study
MROD-1 Prototype
First Results
Conclusions & Outlook
11 September 2002
LECC 2002, Colmar

3. ATLAS MDT Muon Detector ~ 300.000 Drift Tubes ~ 1200 MDT Chambers
~ Towers of 6 Chambers
11 September 2002
LECC 2002, Colmar

4. System Overview Chamber Tower 24 ch. TDC CSM CSM-Link (GOL)* MROD
18 x
(GOL)*
24 ch. TDC
6 x
160 MBytes/s
S-Link to ROB
24 ch. TDC
CSM
18 x
24 ch. TDC
CSM-Link
(GOL) *)
11 September 2002
LECC 2002, Colmar

5. CSM Functionality CSM 40 Mbit/s Data/Clock from TDC Serial to Parallel
&
Clock Domain Separator S 1 18
1 Gbit/s
Separator
18 x
40 Mbit/s
Data/Clock
from TDC
(GOL)
Serial to Parallel
&
Clock Domain Separator
1 Start bit
32 Data bits
1 Parity bit
2 Stop bits
36 25 ns = 900 ns
1 Separator word (S)
18 TDC data words
19 words in 900 ns  85 MB/s
11 September 2002
LECC 2002, Colmar

6. MROD Functionality Separator word Check (do not store) TDC0, word 1
Build events in a partitioned memory
from TDC data fragments
(tdc 1) 000…000
Skip (do not store)
time
TDC2, word 4
TDC3, word 2
Separator word
TDC0, word 1
TDC1, word 3
TDC2, word 5
TDC3, word 3
TDC2, word 3
Separator word
TDC1, word 2
TDC2, word 2
(tdc 0) 000…000
TDC1, word 1
TDC2, word 1
TDC3, word 1
TDC0, word 0
TDC1, word 0
TDC2, word 0
TDC3, word 0
11 September 2002
LECC 2002, Colmar

7. MROD Form Factor
• 9 U VME board (single slot), 6 CSM Inputs, 1 S-Link Output
• Optionally 2 extra CSM Inputs with “extension” board to
accommodate some special towers with > 6 chambers
CSM Input Interfaces integrated on main board
• 1 MROD Crate (Sub rack) contains:
12 MRODs (12  Segments)
Up to 4 MROD extension boards
1 Crate Master with Ethernet Interface (ROD Crate DAQ)
1 TIM: TTC-Rx Interface Module (incl. ROD Busy)
192 towers: 192/12 = 16 MROD Crates (1 per  Sector)
11 September 2002
LECC 2002, Colmar

8. MROD Crate DAQ / DCS ROD Busy Network ROB ROB VME-bus “TIM-bus”
… total ...
12 x
MROD
TIM (TTCrx Interface)
6 CSMs
6 CSMs
From TTC system
One MROD Crate services 12 towers (one full  sector).
In total 16 crates will be required for all MDT chambers.
Some MRODs may have 7 or 8 input links via “slave” MROD input cards.
11 September 2002
LECC 2002, Colmar

9. MROD Throughput MROD CSM-Link MROD-in CSM-Link CSM-Link MROD-out
S-Link
CSM-Link
CSM-Link
MROD-in
CSM-Link
Average 5 hits per TDC + header + trailer = 7 words/event
Per tower of 6 chambers max. 88 TDCs * 7  600 words/event (= 2.4 kB/event)
Worst case 100 kHz L1A rate  240 MB/s per MROD
Calculation based on actual tower layout (J.Chapman): max. rate < 60 MB/s per MROD
11 September 2002
LECC 2002, Colmar

10. MROD-0 Feasibility Study
CSM
MROD
MROD-in
CSM
MROD-in
MROD-out
CSM
MROD-in
ROB
CSM
MROD-in
CSM
MROD-in
MROD-in
CSM
11 September 2002
LECC 2002, Colmar

11. (SHARC-I)
11 September 2002
LECC 2002, Colmar

12. MROD-0 Prototype MROD-in MROD-out MCRUSH SHASLINK 11 September 2002
sorted
TDC-data
over
SHARC Link
MROD-in
MROD-out
MCRUSH
SHASLINK
11 September 2002
LECC 2002, Colmar

13. MROD-0 Input Channel MCRUSH 1 MB ZBT Memory Input FIFO Output FIFO
Tetris
Register
FIFO
Control
SHARC
Data FIFO
@ 40 MB/s each
6 Sharc links
Length FIFO
FPGA
Control/Status
Error signaling
11 September 2002
LECC 2002, Colmar

14. MROD-0 Output Channel SHaSLINK PCI bus PCI 9054 SHARC 6 SHARC Links
@ 40 MB/s each
Altera
10K10A
S-Link max.
@ 160 MB/s
11 September 2002
LECC 2002, Colmar

15. MROD-0 Emulation Hardware
sorted +
merged
TDC-data
TDC-data
sorted
TDC-data
MROD-in
(3x)
MCRUSH
CSMSIM
SHASLINK
MROD-out
SHASLINK
ROBSIM
CRUSH +
SHASLINK 2 3 1 1 3
fragment lengths
xxxxx
Module type
S-Link
optionally double/triple
MROD-in output thus
simulating 2 or 3 MROD-ins
SHARC-links
event fragment lengths
via SHARC-link simulates
future MROD-1 functionality
11 September 2002
LECC 2002, Colmar

16. MROD
CSMSIM
MRODIN
MRODOUT
ROBSIM
11 September 2002
LECC 2002, Colmar

17. MROD-0 Performance Study Results 11 September 2002 LECC 2002, Colmar
CSMSIM
MRODIN
MRODOUT
ROBSIM
11 September 2002
LECC 2002, Colmar

18. MROD-0 Performance Study Results 11 September 2002 LECC 2002, Colmar
CSMSIM
MRODIN
MRODOUT
ROBSIM
11 September 2002
LECC 2002, Colmar

19. MROD-0 Performance Study Results 11 September 2002 LECC 2002, Colmar
CSMSIM
MRODIN
MRODOUT
ROBSIM
11 September 2002
LECC 2002, Colmar

20. MROD-0 Performance Study Results 11 September 2002 LECC 2002, Colmar
CSMSIM
MRODIN
MRODOUT
ROBSIM
11 September 2002
LECC 2002, Colmar

21. MROD-0 Performance Analysis
Measured event rate for single output 40 MHz with 2 and 3 emulated CSM inputs: maximum event rates of 70 and 50 kHz respectively are measured.
MROD-1 will use the 80 MHz: both the processing speed and the bandwidth increase proportionately  event rate  100 kHz ?
 use 2 SHARC-II processors for MROD-out.
11 September 2002
LECC 2002, Colmar

22. MROD-1 Prototype 3x (in total) Memory FPGA VME64x SHARC FPGA Memory
Sharc Links
3x (in total)
Memory
FPGA
TTC
Interface
SHARC
FPGA
Memory
11 September 2002
LECC 2002, Colmar

23. SHARC-II
11 September 2002
LECC 2002, Colmar

24. The ADSP-21060 and the ADSP-21160 SHARCs
• 40 MHz / 80  100 MHz CPU (SIMD mode)
512 KB / 512 KB internal memory
6 x 40 / 80  100 MB/s links. Throughput of all links simultaneously
is 160 / 480  600 (?) MB/s, without disturbing the CPU.
No handshaking on links, but hardware XON-XOFF protocol,
10 / 14 DMA channels
Support for bus arbitration: at max. 6 SHARCs can be connected to a common
bus without glue logic. Each SHARC can access the internal memories of each other
SHARC. The SHARCs also provide support for a so-called host interface, which can
act as an additional master on the common bus.
Fast interrupt servicing due to the presence of shadow registers
Two 40 Mbit/s / 40  50 Mbit/s (at max.) synchronous serial ports
Can be booted via link 4
11 September 2002
LECC 2002, Colmar

25. MROD-1 Form Factor • 9 U VME boards, 2 units wide
1 S-Link output on daughter board
6 GOL inputs on daughter boards
• SHARC II (ADSP21160) DSPs:
(3 for input, 2 for output processing)
Altera APEX FPGAs, 200k gates
TIM bus over special P3 back plane
VME64x interface
GOL
daughter
boards
Input
Input
S-Link
daughter
board
Input
Output
Motherboard
11 September 2002
LECC 2002, Colmar

26. MROD-out Board
11 September 2002
LECC 2002, Colmar

27. MROD-in Board
11 September 2002
LECC 2002, Colmar

28. MROD-1 Prototype Status
Fully functional MROD-1 modules exist:
7 MROD-in and 3 MROD-out boards have all been extensively tested, also in conjunction with simulated data via the input link.
Software to boot the SHARCs via the VME bus and a run-time environment providing file and terminal I/O via a server program under both the LynxOS and Linux operating systems on the VME processor is available.
11 September 2002
LECC 2002, Colmar

29. MROD-1 Prototype Status (Contd.)
Current MROD-1 SHARC software is largely ported from the MROD-0 prototype. Further development of this code is required.
This process is hampered by a compiler which is not fully reliable, in particular when it comes to optimization which in turn is very important to obtain good performance.
Transition to SHARC-II DSPs has not always been completely straightforward: work-arounds have been implemented. See example below ….
11 September 2002
LECC 2002, Colmar

30. MROD power-up modification
11 September 2002
LECC 2002, Colmar

31. First MROD-1 Results
Preliminary single channel tests with a CSM simulator show a sustainable 125 kHz event rate with 18 simulated TDCs with five 32-bit words per event.
Equivalent of 45 MBytes/sec.
Rate of 125 kHz appears to be limited by the input (i.e. the CSM simulator) rather than by the MROD-1 itself. This is being investigated.
11 September 2002
LECC 2002, Colmar

32. Conclusions & Outlook
The first MROD-1 test results are in line with the MROD-0 results and the SHARC-I to SHARC-II extrapolation.
Results are encouraging. Improvements still possible. More tests need and will be done.
Development of a GOL receiver card to connect to the CSM is well underway.
System integration tests with both the CSM and the DAQ-1 and ROD CRATE DAQ follow soon.
Application in NIKHEF local Cosmic Ray Test Stand.
11 September 2002
LECC 2002, Colmar