DDL1 ALICE Detector Data Link (DDL) and it’s use in STAR TOF J. Schambach

DDL2 Presentation Outline Hardware Overview DDL Protocol / Transactions RORC DDL in TOF Software Demo

DDL3 Readout system Front-end electronics Detector Data Link DDL SIU DDL DIU RORC Source Interface Unit Destination Interface Unit Read Out Receiver Card PC Data Acquisition PC Optical Fibre ~200 meters

DDL4 DDL architecture Source Interface Unit (SIU) (1) –Interface to the Front-end Electronics (2) Destination Interface Unit (DIU) (3) –Interface to the Readout Receiver Card (4) Full duplex optical link (5) –Multimode optical cable of up to 200 m

DDL5 DDL hardware

DDL6 SIU Physical Layout

DDL7 DDL interfaces SIU-FEE interface –3.3V (LVTTL) interface –32-bit wide half-duplex data bus (bi-directional bus) –Bi-directional flow control –User defined clock (synchronous interface) –JTAG interface DIU-RORC interface –3.3V (LVTTL) interface –32-bit wide full-duplex data bus –Bi-directional flow control –User defined clock (synchronous interface)

DDL8 DDL Interface signals

DDL9 SIU Connector Pinout

DDL10 SIU-FEE interface fbD(31..0)- data lines(bi-directional) fbTEN_N- transfer enable(bi-directional) fbCTRL_N- CONTROL qualifier(bi-directional) fiDIR- bus direction(FEE input) fiBEN_N- bus enable(FEE input) fiLF_N- link full(FEE input) foBSY_N- front-end busy(SIU input) foCLK- interface clock(SIU input) TAP_TCK- JTAG clock(FEE input) TAP_TDI- JTAG data in(FEE input) TAP_TDO- JTAG data out(SIU input) TAP_TMS- JTAG mode select(FEE input) TAP_TRST- JTAG reset(FEE input)

DDL11 Link management

DDL12 DDL Configurations

DDL13 Front-end Commands

DDL14 Front-end Status Words Data Transmission Status Word (DTSTW), produced by SIU:

DDL15 Front-end control on-line FECTRL CTSTW on-line DIUSIURORC Online FEE command Report FEE FEE control idle foCLK fiBEN_N fiDIR FECTRL fbD fbTEN_N fbCTRL_N

DDL16 Front-end status read on-line FESTRD FESTW on-line DIUSIURORC Online FEE command Status and report FEE FEE status read CTSTW foCLK fiBEN_N fiDIR FESTRD fbD fbTEN_N fbCTRL_N HiZ foCLK fiBEN_N fiDIR FESTW fbD fbTEN_N fbCTRL_N HiZ

DDL17 Event read foCLK fiBEN_N fiDIR RDYRX fbD fbTEN_N fbCTRL_N HiZ foCLK fiBEN_N fiDIR EOBTR fbD fbTEN_N fbCTRL_N HiZ

DDL18 Block write on-line STBWR CTSTW DIUSIURORC Online FEE command Report FEE STBWR CTSTW FEE data EOBTR FEE command Report EOBTR Flow control Block data data block foCLK fiBEN_N fiDIR fbD fbTEN_N fbCTRL_N foBSY_N EOBTRDn-1Dn

DDL19 Block read on-line STBRD CTSTW DIUSIURORC Online FEE command Report FEE STBRD CTSTW FEE data EOBTR data block Block data FEE command Report EOBTR Flow control foCLK fiBEN_N fiDIR STBRD fbD fbTEN_N fbCTRL_N HiZ foCLK fiBEN_N fiDIR EOBTR fbD fbTEN_N fbCTRL_N HiZ

DDL20 RORC features Interface between the DIU and PCI local bus –pRORC: 32 bit/33 MHz PCI version, max. throughput 132 MB/s –D-RORC: 64 bit/66 MHz PCI version, max. throughput 528 MB/s PCI master capability, data push architecture –Autonomous operation with little software assistance –Supports multi-paged memory management Direct data transfer to the PC memory –No local memory on the board –Small elasticity buffers between different clock domains Built-in test capability –Internal pattern generator can produce formatted data

DDL21 D-RORC Hardware D-RORC with integrated DIU ports to read out two DDL channels to support integration with the HLT system D-RORC with plug-in DIU to read out single DDL channel to support the tests of FEE readout sytems

DDL22 RORC Roadmap pRORC:32-bit, 33 MHz (PCI I/F by ASIC) –well adapted to the prototype version of the DDL –can be used for the new version of the DDL (adapters) –already used by several test beams (SDD, HMPID) D-RORC I:64-bit, 66 MHz (PCI I/F by IP core) –well adapted to the new version of the DDL –will integrate two DIU functions on-board –will support the DAQ/HLT interface D-RORC II:64-bit, 66/133 MHz (PCI-X I/F by IP core) –to avoid compatibility issues –to follow the evolution of the PCs

DDL23 Hardware Architecture APEX FPGA 64-bit/66 MHz, PCI/PCI-X Media I/F 1 Media I/F 2 Busy I/F Conf. Flash JTAG P11P12 P13P14 CMC I/F Optical I/F LVDS I/F Configuring 250 MB/s 528 MB/s

DDL24 Firmware Architecture PCI core (64-bit master, memory mapped) Control registers addresslength Transmit DMA addresslength Receive DMA RX FIFOTX FIFO DIU I/F or DIU core Slave I/FMaster I/F RAF TAF DMA control TAF – Transmit Address FIFO RAF – Receive Address FIFO DIU or Media I/F PCI bus

DDL25 The Free FIFO PRORC PC memory bank Firmware readout page address Free FIFO PC CPU Allocation of free pages

DDL26 Direct Memory Access PRORC Firmware PC memory bank DDL No involvement PC CPU

DDL27 The Ready FIFO PRORC PC memory bank readout DDL Ready FIFO Firmware lengthpage status lengthpage status lengthpage status Delivery of filled pages PC CPU

DDL28 Software Event ready test: 1) SOFTWARE start = curr; do { do { while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr); while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr); } while (Ready FIFO [curr].status == 0xffffffff || } while (Ready FIFO [curr].status == 0xffffffff || Ready FIFO [curr].status == 0) Ready FIFO [curr].status == 0) end = curr; end = curr; curr = NEXT( curr ); curr = NEXT( curr ); startend Blocks from start to end are now available 2) HARDWARE delivery of signal, enabling of semaphore or setting of flag Release event Firmware Firmware data length transfer status Ready FIFO Event ready? N Y Pop descriptor from Free FIFO Move data into buffer until DTSTW or up to buffer size Update size size = 0 DTSTW? Y N size > max size? N Y Throw away rest of data do not increment block length until DTSTW received Mark DTSTW “OVERRUN” bit Push data length in Ready FIFO Push transfer status = Optional: update Ready FIFO in memory Push data length in Ready FIFO Push transfer status = DTSTW Update Ready FIFO in memory Deliver “done” interrupt Optional: set stop flag on error Free FIFO empty? Y N  block: Reset entries of Ready FIFO Push address & size on Free FIFO (if space available in Free FIFO) Pull data length and transfer status from Ready FIFO Push pending free blocks on Free FIFO (if any) Load Free FIFO Initialize Ready FIFO Load configuration registers Clear stop flag Initialize internal data structures stop flag = ON acknowledge reset done Transfer status possible values: ffffffff ffffffff unloaded (set by sw) loaded, no DTSTW (set by fw) else else loaded, DTSTW (set by fw) Free FIFO block size Index of Ready FIFO start address stop flag ON? N Y RORC

DDL29 Test equipments Front-end Emulator Interface Card (FEIC) –Fully functional hardware to emulate the detector front-ends –Formatted data block generation –Internal (free running) or external (pulse) triggering capabilities –Adjustable parameters (using front-end control) –Operates at the nominal speed of the DDL Source Interface Unit Simulator (SIMU) –Simulates the behavior of the DDL without any additional hardware –Eases the development and the hardware debugging –Size is similar to the real SIU

DDL30 Bandwidth: D-RORC to D-RORC Testing the transfer between two D-RORC cards rorc_send –g... (pattern generator) rorc_send... (DMA from)

DDL31 TOF Essential Model Level 1

DDL32 TOF Essential Model Level 2

DDL33 Run 5 TCPU

DDL34 DDL Interface Implementation

DDL35 TCPU Firmware Data Path

DDL36 DDL Software PCI driver API routines for DATE Executable utility programs for test and stand-alone (= without FEE or DATE) use of the DDL, such as –Reset the RORC, DIU or SIU –Display the status of the RORC, DIU or SIU –Test the functionality and measure the performance of the whole DDL and RORC system

DDL37 Installation of the utilities Utilities are in the directory./rorc/Linux/ Linux kernel version: 2.4 Driver module must be inserted. As root type: /sbin/insmod./Linux/rorc_driver.o or insert a similar line into /etc/rc.d/rc.local Checked if physmem and RORC driver are loaded: cat /proc/modules Check if RORC card is plugged: rorc_find

DDL38 $ /sbin/lspci 00:00.0 Host bridge: Intel Corp (Carmel) Chipset Host Bridge (Hub A) (rev 01) 00:01.0 PCI bridge: Intel Corp (Carmel) Chipset AGP Bridge (rev 01) 00:02.0 PCI bridge: Intel Corp (Carmel) Chipset PCI Bridge (Hub B) (rev 01) 00:1e.0 PCI bridge: Intel Corp AA PCI Bridge (rev 02) 00:1f.0 ISA bridge: Intel Corp AA ISA Bridge (LPC) (rev 02) 00:1f.1 IDE interface: Intel Corp AA IDE (rev 02) 00:1f.2 USB Controller: Intel Corp AA USB (rev 02) 00:1f.3 SMBus: Intel Corp AA SMBus (rev 02) 01:05.0 Multimedia audio controller: Cirrus Logic CS 4614/22/24 (rev 01) 01:06.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 01) 01:08.0 Ethernet controller: Accton Technology Corporation SMC2-1211TX (rev 10) 02:1f.0 PCI bridge: Intel Corp AA PCI64 Hub PCI Bridge (rev 02) 03:00.0 PIC: Intel Corp AA PCI64 Hub Advanced Programmable Interrupt Controller (rev 01) 03:04.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 02) 03:09.0 SCSI storage controller: Adaptec AIC-7892P U160/m (rev 02) 04:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200 AGP (rev 03) Identifying the RORC card

DDL39 Program ‘rorc_find’ List all plugged and presently not used RORC devices together with their version and serial numbers. rorc_find The following device(s) found: Minor Channel Device type and HW identification (RORC’s FW version) D-RORC (FW: 1v35 of July ) new DIU DDL card 2v0 LD: 20K60E SP: 2125 Mbps S/N: D-RORC no DIU 1 0 pRORC pRORC 1v1 S/N: (FW: 1v75 of June ) new DIU DDL card 2v0 LD: 20K60E SP: 2125 Mbps S/N: RORC channel(s) not in use was found. RORC driver reported 2 RORC device(s).

DDL40 rorc_reset Initialize the RORC card and the DDL link rorc_reset [-{M|m} | 0] [-{D|d|B|b|S|s|F|f|O|o|E|e|C|c}] Where -D or –d reset the DIU -B or –b reset both RORC and DIU - S or -s reset SIU -F or –f clear Free FIFO -O or –o clear RORC’s other FIFOs -E or –e clear RORC error bits -C or – c clear RORC’s byte counter No optionreset RORC

DDL41 rorc_id Display RORC’s hardware and software identification words rorc_id [-{M|m} | 0] [-{D|d}{S|s}] [-{T|t} | ] Where -D or –d display DIU’s firmware id as well -S or –s display SIU’s firmware id ad well time-out time-out value for DIU or SIU reply e.g.: rorc_id RORC driver version: 4.2 RORC revision id: 1 Hardware identity word of the RORC: pRORC 1v1 S/N:00103`, i.e. Version: 1.1, S/N: Firmware identity word of the RORC: 0x , i.e. Version: 1.72 Release date : October Free FIFO size: 128 entries

DDL42 rorc_send_command Send a DDL command and receive the reply rorc_send_command [-{M|m} | 0] -{C|c} [-{T|t} | ] [-{V|v} | 2] Where command a hexadecimal number starting with “0x”, or an ASCII mnemonic of a DLL or pRORC command, e.g.: LBON RORC loop-back on RDYRX ready to receive message to the SIU EOBTR end of block transfer message to the SIU (see the sw manual or the program help for command codes and format) time-out time-out value for RORC, DIU, SIU or Front-End reply diu_version 1 for prototype, 2 for final version.

DDL43 rorc_receive Send and receive data to the Front-End. The most important options: rorc_receive [-{M|m} |0] [-{G|g}] [-{Y|y}] [-{Z|z}] [-{X|x} ] [-{K|k} ] [-{E|e} ] [-{P|p} |0] [-{I|i} |0] [-Q |1] Where (from version 4.2: rorc_receive ) -G send data using RORC’s data generator -Y do not loop-back generated data but send it via the link -Z do not send RDYRX and EOBTR commands output file dump the receives the data into this file without checking it check_level0 : do not check the received data, 1 : check only the first word, 2 : not the first word, 3 : check the whole event events # events to send pattern event pattern to send or receive, it could be: c, a, 0, 1, I, d init word the first word of each event’s payload (after the event serial number) GBytes display the number of received bytes after each received Gbytes data (see the sw manual for further options)

DDL44 feic.menu Check and set the Front-End Emulator Interface Card feic.menu [-{M|m} |0] This script calls the rorc_send_command program several times to check the FEIC setting or to set new parameters. The following features of the FEIC’s data generator can be modified and displayed: data pattern alternating, flying 0 or 1, incrementing, decrementing data event length 16, 32, 64, …., 256 Kwords trigger mode push button, ext. trig., 16 or 128 clocks after each event, every 10or 100 ms seed the seed value for random data length See the description of rorcArmFeic routine in the sw manual for further details.

DDL45 Other utilities program name program function diu_id DIU hw and fw identification siu_id SIU hw and fw identification rorc_status show RORC status rorc_reg show RORC registers diu_status ask and display DIU status siu_status ask and display SIU satatus rorc_send download data from the PC to the FEE rorc_send_jtag download JTAG data