Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt The CBM-MVD read-out system Christoph Schrader for the CBM-MVD Collaboration 1

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 2 Demo (MI20) data synchronization … Prototype v2 (MI26)  optical readout  sensor ladder  compatible to a dedicated MVD sensor Prototype SIS-100 (MISTRAL ) DAQ project phases Prototype v1 (MI26)  conservative approach (based on LVDS signals)  full readout concept  full bandwidth

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 3 Prototype Version 1

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt MIMOSA-26 pixel array: 1152 columns x 576 rows (18.4 µm pitch and µs readout time) pixel array: 1152 columns x 576 rows (18.4 µm pitch and µs readout time) JTAG Digital input Digital input Digital output DACs 4 x memory banks PLL Temp probe Temp probe Discri test Discri test Power 1152 discriminators zero suppression logic CDS 2 channels D0,D1+clock80 MHz 160 Mbit/s 9 hits/ row ∑ 570 hits multiplexer 4  ~ 80 Mbit/s (570 words of 16 bit  ~ 9120 bit/frame) available

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt Prototype Version1 FPC based on MIMOSA-26 FEB... vacuum clock start reset JTAG converter board converter board converter board... readout controller board driver board FEB sensors... readout controller board FEB LVDS, 1m 4x 80 Mbit/s (MIMOSA-26) LVDS 4 x 80 Mbit/s FPC 2 Gbit/s optical fiber to the MVD network FPC Slow control board

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop materialmaterial budget [% x/Xo] Polyimide (12.5 µm) Adhesive (15 µm) Copper (18 µm)0.063 (Alu 0.011) Adhesive (20 µm) Polyimide (25 µm) ∑ Cu ~ % x/Xo* sensor copper/ aluminum Adhesive (20 µm) polyimide wire bonds µm 10 µm polyimide connector pads Flex-Print Cable # of wires50 wire length500 mm pitch200 µm resistance2.47 Ω (measured 2.75Ω) capacity12 pF inductivity12 µH max. frequency400 MHz (LVDS) ∑ Alu ~0.032 % x/Xo * ** * 100% fill factor ** same dimension as copper

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop 7 Flex-Print Cable 10 MHz - 22 dB Single-ended crosstalk B. Neuman

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop dB Flex-Print Cable could be better to have differential signal for JTAG B. Neuman

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 9... clock start reset JTAG FPC The FEB: transfer D0,D1,clock to the converter boards (0.5 m away) to loop clock, reset, start, JTAG passive filters for power supply FPC D0,D1 clock TDO Temp current FEB FPC full passive board (more radiation tolerant) FEB

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt signal conditioning for data transfer power supply for LVDS drivers diode based temperature measurement (internal sensor diode) 6 channel ADCs to transfer in serial the current and temperature current monitor and protection to disable the power of the sensors (in case of latch-up) switch for JTAG chain shutdown <2 µs power on 200 µs 10 Converter Board FEB converter board converter board converter board readout controller FEB readout controller see talk:  M.Koziel

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 11 Sensor slow control via JTAG Requirements for the slow control system:  Initializing of each sensor via JTAG  Scalability: > 100 MAPS  Sequential reading of the sensor ID  Triggered monitoring/reprogramming of the sensor parameters (during a spill break)  Error detection and error logging  Automatic hardware bypassing/unbypassing by hardware switch for faulty sensors  New arrangement and initialization of prober sensors left (40 sensors < 10ms)  Compatible to the control software “EPICS” Switch should be integrated in the sensor Slow control board Slow control board Converter board Converter board

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop TrbNet converter boards 4 optical links up to 3.8 Gbit/s each readout controller board readout controller board LVDS 4 x 80 Mbit/s Readout Controller Board HADES optical Hub Pexor PCI-Express card 20x up to Gbit/s LVDS 4 x 80 Mbit/s MVD Network see talk: J.Michel

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Data checking -Is the MAPS turned on? -HDR/TRA has arrived -Is the frame number OK? -Is the data length OK? -Is the entire frame OK? -Is the time between frames 115,2 us? -Are the MAPS out-of-sync? -Is the Arbiter SYNC pulse OK? -Buffer overflow Data is consistent! consecutive frame numbers is 0x2D00 cycles, or 11520! = 115,2 us Number of data packets is always 0x023a, or 570! 0x84a8 – 0x57a x2D00 B. Milanovic

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 14 Prototype Version 2  compatible to dedicated MVD SIS-100 sensor  sensor ladder  optical readout

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt mm pixel array ~10 mm digital data sparsification ~2.5 mm readout  Available in 2014  double side readout  based on MIMOSA-26 readout protocol  2 output pairs with 200 MHz MISTRAL

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 16 see talk: T.Tischler MISTRAL sensors 2 st 10 cm FEB

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt FPC Flex-Print Cable FEB pads sensors bonds FPC pads (FEB side) for MIMOSA-26 bus (signal chain) Individual signals/sensor  7 lines/sensor 17 ∑ Cu ~0.075 % x/Xo ∑ Alu ~0.036 % x/Xo Copper (18 µm) Adhesive (20 µm) Polyimide (12.5 µm) Adhesive (15 µm) Copper (18 µm) Adhesive (20 µm) Polyimide (12.5 µm) % x/Xo The half sensors has to be reversed left to right

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt Flex-Print Cable Sensor (50 µm Si)0, FPC (copper/alu)0.08/0.04 CVD (150 µm)0.11 Adhesive (20 µm) Polyimide (25 µm) CVD Σ cu ~0.40 % x/X o 10 mm < 8 mm active Σ Alu ~0.30 % x/X o see talk:  T.Tischler FPC % x/Xo ~ 170 µm 150 µm

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt CBMNet vacuum clock start reset JTAG converter board converter board converter board... readout controller board driver board 10x 200 Mbit/s LVDS, 1m 1 Gbit/s optical fibers... readout controller board Prototype Version2 LVDS 10 x 200 Mbit/s

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt Thank you 20

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop 21 Flex-Print Cable 20 MHz 1.58 dB Single-ended power attenuation

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop 22

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop 23

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 24 Devices: FPC: FCI opuserie JTAG switch: SN65LVDCP22PW 6 channel ADC: LTC1408 optical links: normal SFPs ROC FPGA: ECP2M Lattice 8b/10b encoding: TLK2501IRCP

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 25 Synchronization and time management

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 26 Hub Arbiter Synchronization Slow control board MAPS ROC There is one common clock for all devices! Arbiter provides clock (LVDS, optical) clock equally long LVDS

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 27 Synchronization Hub Arbiter MAPS ROC Sync equally long LVDS During each SYNC pulse from the Arbiter, the ROCs detect if the MAPS are running out-of-sync (10 ns resolution) Errors are reported to the Arbiter, which can schedule a reset and reprogramming during next spill-break. The OOS resolution depends on FPGA frequency. Time between SYNCs is the readout time of one frame (115,2 us). If using one common clock the SYNCs can be issued with 100% accuracy (no time delays!). Slow control board start

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 28 Hub Arbiter MAPS ROC buf Frame Req. The Arbiter sends to the ROCs a frame request to save the next frame data in the buffer. The latency for that message can be several microseconds.(<<30 µs) The frame request contains the frame number and further processing information. Data readout

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 29 Data readout Hub Arbiter MAPS ROC Frame data are buffered in the ROCs. ROCs report about their status to the arbiter (error handling). buf In case of buffer overflow the arbiter decide to throw the data away for the given frame of all boards. acknowledge Req.

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 30 Hub Arbiter MAPS ROC buf If JTAG detects a JTAG error in the sensor, it will remove it from the chain and turn it off. If any sensor is showing errors in its data or is turned-off the data are marked as false. Error handling

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 31 Data readout Hub Arbiter MAPS ROC Readout for one frame is requested ROCs send the corresponding data to the hubs. buf data Readout Req.

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt high data-rate digital interface connector (15 Gbit/s) FPGA configuration high data transfer with optical link (2 Gbit/s) application process interface (API)‏ power supply +5V, 10A clock distribution slow control via ethernet  optical link (TrbNet) A standard platform the TRBv2 (HADES) provides a flexible add-on board concept 32

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt sensor controlling handle up to 4 digital channels (2 sensors) platform to study online data sparsifications  for data reduction  fake hit suppression compatibility with HADES DAQ (TrbNet) for data transfer developed by IKF electronic workshop the add-on board with a FPGA is mounted on the TRBv2 back side The controller board future upgrade: to handle several prototype modules future upgrade: to handle several prototype modules 33

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt Hub-Add-on (HADES) 20x up to Gbit/s capable of Gigabit-Ethernet to send data to standard PCs (TCP) implements basic data processing features de-multiplexer for I/O’s buffered readout 34

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt PC-Interface: PEXOR 3 PCI-Express card fast Lattice SCM40 FPGA TrbNet (HADES) 4 optical links up to 3.8 Gbit/s each 35

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt The first set-up hub Controller board Controller board Trbv2 readout of several prototypes is possible 36 to test: data protocol data reduction 160Mbit/s  80Mbit/s network

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 37 Number of ROCs # sensors40120 raw data rate [Gbit/s] compressed data rate [Gbit/s] for 10A.GeV 128 compressed data rate [Gbit/s] for 30GeV 1420 # of input channels # number of ROCs 815 max. output data rate [Gbit/s] 4890 load/station [%] 10A.GeV) 3211 load/station [%] 10A.GeV) 3628 ROC with one FPGA: 8x input (1Gbit/s) 2x output (3 Gbit/s) 1x slow control ~ factor 3 for safety data rateshardware required

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 38 Front side number of sensors to cover the CBM acceptance at: ~ 5cm ~ 1cm distance from target 5 cm10 cm # sensors MIMOSA MIMOSIS The MVD detector design FPC back side inactive area first station based on MIMOSA-26 form factor

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt FPC 39 FEB... driver board FEB FPC The FEB: full passive board transfer D0,D1,clock to the converter boards (<1m away) to loop clock, reset, start, JTAG passive filters for power supply The driver board: - feed through for the vacuum vessel - to loop clock, reset, start, JTAG - converts JTAG, start, reset (LVDS)  LVTTL - <2m away from the FEB FPC as FEB first version for prototype (MIMOSA-26) replaced by FPC with passive filter FPC material budget: ∑ Cu ~0.086 % x/Xo ∑ Alu ~0.032% x/Xo

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt de- MUX frame HDR data 0data 1 data 2 … channel pattern word pattern word pattern time stamps slow control … 8 bit in MHz FPGA 8 bit/channel in MHz 8 bit/channel in MHz 16 bit/channel in MHz 800 Mbit/s 16 bit/channel in MHz < 300 Mbit/s Word/channel pattern buffer RAM FIFO select out the place holder and words which are send twice like HDR and trailer 10b/8b data reduction infos. flags FEE ?? FEE processing D0 D1

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 41 FEB buffer? clock reset start JTAG FPC pads for sensors converter LVDS  LVTTL clock (LVDS) reset startJTAG LVTTL clock (LVDS) D0,D1 (LVDS) buffer? TDO (LVTTL) converter LVTTL  LVDS + amplification Temp (LVTTL) TDI (LVDS) FPC to converter board buffers/converters are critical in case of radiation damage and need cooling  first studies with a passive board (prototype) pass. filter power next FEB ? ?

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt Prototype Version1 FEB... vacuum clock start reset JTAG converter board converter board converter board... readout controller board driver board FEB sensors... readout controller board FPC Hardware is under test

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 43 sensor1 TMS TCK TDI TDO sensor2 TDO last TDO is used select (FPGA) TDI select (FPGA) JTAG chain

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 44 MIMOSA-26 protocol and readout protocol

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt The data path sensor line D0 8 bit word multiplexer line D1 8 bit word data reduction sensor readout protocol sensor readout protocol sparsification network protocol TrbNet (HADES network protocol) data reduction and fake hit selection scalable sensor protocol with unique sensor ID and time stamp architecture sensor readout: line merging and first level data reduction 45...

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt line D0 HDR 16 bit with serial output Sensor Output Protocol HEADER: fix bit pattern Frame counter: 32 bit Data Length: < 570 # words/frame Colm. Address: Row Address: < 9 hits/row + neighbor pixel Trailer: same as header line D1 F 1 D L CA RA 1 T … HDR F 2 D L RA 0 RA 2 T … 46 continuous readout: zeros as place holder  first data reduction HDR words zeros HDR

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt channel pattern line D0 line D1 output enable HDR … … F 1D LL A … HDR F 1DLL A HDR F 2DL 14 clock cycles HDR F 2D LRA T T TT create 16 bit words form serial channel input Data handover 47 one 16 bit word/clock cycle 15 clock cycles

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt enable … HDR D L L A T enable … HDR D LF 1 L A HDR D LF 2 TT RA F 1 F 2 First level data reduction output removed RA … 48

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt Sensor readout protocol header/trailer: shows start and end of a frame scalable system unique ID for sensor unique ID for readout board time stamp: 32 bit frame counter data length: gives the number of words/frame address of the hit pixel + neighbor pixels data flow 49

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt data length sensor-ID: 64 sensors/boardboard-ID: 64 boards/node pari ty frame counter frame counter frame counter HDR data length 32 bit frame counter or time stamp Sensor readout protocol 50

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt ADDR line ovf NNADDR column NNADDR column trigger counter trigger counter trigger counter bit trigger counter or time stamp Sensor readout protocol

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt max. 570 words Sensor readout protocol sensor-ID: 64 sensors/boardboard-ID: 64 boards/node pari ty 11 1

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt backslides 53

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 54 simulation results:  MIMOSIS needs an output buffer of ~ 24 Kbit/frame  MIMOSIS needs a readout frequency of ~ 800 MHz to transfer (serial) 24 kbit in 30µs  Max. data rate: 800 Mbit/s  Mean data rate: < 300 Mbit/s [Byte] Byte simulations of S. Seddiki - MIMOSA-26 protocol (16 bit) - 30 µs integration time - max. pileup ~5 coll/frame (Au-Au) - HADES beam fluctuations Data Volume per Sensor maxmean 10 7

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt MIMOSIS 55 first data reduction (<300Mbit/s) pixel array: ± 1152 columns x ~256 rows (18,4 µm pitch and 115,2 µs readout time) pixel array: ± 1152 columns x ~256 rows (18,4 µm pitch and 115,2 µs readout time) JTAG Digital input Digital input Digital output DACs 4 x memory banks PLL Temp probe Temp probe Discri test Discri test Power 1152 discriminators zero suppression logic CDS 2 channels D0,D1+clock 80 MHz 9 hits/ row ∑ 570 hits multiplexer 2014 for SIS-100 Requirements for R/O  2 output channels/sensor with 800 Mbit/s  readout frequency 400 MHz  8 bit word length  for 3 stations (400 sensors) > 800 LVDS cables 30 µs 400 MHz 800 Mbit/s

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 56  max 24 Gbit/station/s for 30GeV  max. 12 Gbit/station/s for 10A.GeV  Higher data flow in p-Au since the δ electron hot spot in Au-Au limits the beam intensity simulations of S. Seddiki based on MIMOSA-26 protocol (with data reduction) Data flow per station

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 57

Institute for Nuclear Physics, University of Frankfurt C. Schrader; Sept. 2011, Mont Sainte Odile, Workshop Institute for Nuclear Physics, University of Frankfurt 58