1. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 1 Vertex Processor Board Optical Tx Board PRR Pile-Up System

2. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 2 Outline Pile Up System overview –Dataflow –Data reordering –Output Board Optical Tx board –Functionality –Board specs –Clock tree –Radiation hardness –Board tests –PCB changes needed VEPROB –Functionality –Board specs –Board tests –Concluding remarks Interconnection tests Conclusions / Outlook

3. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 3 Pile-Up System overview Small system, but still: 4 different types of active boards (some highly complex) 6 different types of passive boards

4. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 4 Optical Tx Boards Dataflow All data of one event (2048 bits) goes to a single processor –I/O limitation -> multiple (4) processors in round robin –Each event transmitted in 4 clock cycles Getting the bits at the right place at the right moment –Simultaneous transmission of ¼ silicon data –Algorithm per ¼ sensor (left / right in parallel) event4 3 2 1 Optical Tx Boards Veprob 1 Veprob 2 Veprob 3 Veprob 4

5. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 5 Data reordering Data reordered via: –Passive repeater cards –Cat6 cabling to balcony –Passive transition boards –Optical TX boards –Optical patch panel Seems simple, but:

6. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 6 Complications: First/second Si wheel mirrored Complex layout of sensor –Mix of stripno - beetle channo and stripno - beetle channo –Gives a twist when multiplexed in time, hence 2 flavors of Optical TX boards (jumper selectable) Layout of hybrid is not compromised –Unavoidable routing irregularities –Signals on 3 different repeater cards –2 types of transition boards Data reordering, some details Repeater -> optical board connections (1 of 2)

7. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 7 PRR Active boards –Hybrid, reviewed Jan 2006 –Optical Tx Board : today –Processor Board : today –Output Board : no proto yet

8. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 8 Output Board Merges Pile Up decision from 4 Processor Boards Large number of monitoring histograms Board items –9Ux400mm 2.4 mm 8 layer pcb –Xilinx Virtex 4 FPGA 10% of logic and 85% of memory used –4 serial inputs 1.6 Gb/s copper –Latency Virtex 4 might be too long Fallback -> 4 extra serial inputs with TLK2501 –2 optical serial outputs -> L0DU –CCPC / Glue Card –TTCrq FPGA code almost done & partially simulated Schematic entry in progress, finished before the end of 2006 Layout finished in January 2007 Board production finished March 2007

9. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 9 Optical Tx Board

10. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 10 Optical Tx Board Data input –Clock synchronization –Time alignment Event time multiplexing BCID labeling VELO control board interface –ECS I2C –TFC (TTCrx) LHC clk BCID reset Power on reset Clock de-jitter and distribution Power-Up –Gol(crt4t) POR via ECS –Flash based FPGA 128 LVDS Data 80 Mb/s LVDS receivers GOL Data sync Event multiplexing BCID labeling LHC clock Qpll-DeJitter Clocktree Parallel Optical transmitter JTAG (lab only) I2C, POR, BCIDrst Optical Tx Board GOL Actel APA300 Power VELO ctrl brd 12VEPROB

11. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 11 Optical Tx Board PCB details 9U X 400mm double height 10 Layer 1.6mm –4 power layers (GND, 2v5, 3v3, GND) –6 routing layers Power via standard LHC backplane (TELL1) Custom backplane P2 P3 (cPCI) LVDS inputs via transition boards in backside of crate Controlled impedance differential traces Signal integrity verified with ICX (Mentor Graphics)

12. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 12 Data to clock synchronization Time align all inputs Compensate timing differences in cables & PCB traces Input sampling –Four FF on different clocks edges 3.25 ns resolution –Max 12.5 ns compensation Sampling phase of each input bit selectable via ECS Global clock phase, via delay selection (delay25) on VELO control board (0.5 ns steps) DFF D clka clkb clka clkb clkselect(0) clkselect(1) clka clkb Beetle data ff1ff2ff3 Fine tune clk delay (control board)

13. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 13 Optical Tx Board Event multiplexing 8 Optical Tx Boards in optical station multiplex events to four processor boards –Time multiplexing four quarters, each in 25ns Each processor board processes one complete (out of four) event –large number of strip inputs to vertex finder algorithm event 4 3 2 1 Optical Tx Boards Veprob 1 Veprob 2 Veprob 3 Veprob 4

14. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 14 Clock distribution LHCb clock (40.08 MHz) received from VELO control board –Optical Tx Board is synchronous system Clock must be de-jittered for GOLs –QPLL on board Board uses both 40 and 80 MHz clock Clock distribution based on MC100LVEP family –LVPECL standard –Similar to MUON ODE Board QPLL 1 to 10 clock Driver (MC100LVEP111) LVDS 40MHz LVPECL 1 to 5 clock Driver (MC100LVEP14) LVPECL GOl 3X Diff-> single ended (DS90LV048) FPGA (Actel ProAsic) 1 to 10 clock Driver (MC100LVEP111) LVDS 80MHz Diff-> single ended (DS90LV048) Diff-> single ended (DS90LV048) Passive Delay Line (3 ns) LVPECL LVCMOS 3X LVDS 40MHz Optical Tx Board

15. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 15 Radiation Hardness Optical station –Located on balcony, 10m from beam –Radiation dose 640 Rad in 10 years (including safety factor 2) –Additional safety factors: 2x2 => dose 2.5 kRad in 10 years –All static registers triple redundant Single event upset counters –All active devices qualified for radiation hardness Rest of trigger logic –Behind wall

16. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 16 Radiation Hardness(2) ComponentTested byTechnologyTIDreference GOLCERN MICRad-Tolhttp://proj-gol.web.cern.ch/proj- gol/publications/paperLEB2001.pdf CRT4TCERN MICRad-Tolhttp://mic-asics.web.cern.ch/mic- asics/documents/CRT4T.pdf QpllCERN MICRad-Tolhttp://proj-qpll.web.cern.ch/proj- qpll/images/qpll2IrradiationResults.pdf LHC4913CERN MICRad-Tolhttp://rd49.web.cern.ch/RD49/ Agilent transmitter HFBR772 LHCb Muon Marseille COTS15kradLHCb note 2004-013 Actel ProAsicPlusLHCb Muon INFN – LNF COTS68 kradODE PRR presentation Clock driver MC100LVEP111 LHCb Muon INFN – LNF COTS68kradODE PRR presentation Clock driver MC100LVEP14 LHCb Muon INFN – LNF COTS68kradODE PRR presentation LVDS receiver DS90LV048ATMTC Atlas Muon RPC COTS70kradhttp://sunset.roma1.infn.it:16080/muonl1/reviews/ 12march2002/RPC_RHA_status.pdf LVDS driver DS90LV047ATMTC Atlas Muon RPC COTS70kradhttp://sunset.roma1.infn.it:16080/muonl1/reviews/ 12march2002/RPC_RHA_status.pdf Open drain Buffer MC74LCX07DTG Atlas Muon RPC COTS780kradLHCb note 2006-01

17. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 17 Optical Tx Board tests ECS control –I2C via VELO control board OK –Control via PVSS panels read/write OK Minor changes in SDA/SCL termination TFC –Fast controls via TTCrq on VELO control board –LHC LVDS clock 40 MHz via delay line (delay25) OK –(BCID reset from TTCrx) Power up GOL –2v5 power via CRT4T for each GOL –On/Off via power on reset (POR VELO control board) OK Clock jitter measurement –De-jitter by Qpll –Low jitter clocktree –GOL always locked OK Link test results later in presentaion Input timing –Verified OK Board temperature –Temperature cycling, to be done –Crate doesn’t fit in climate chamber –All devices T < 45 C Edge to edge σ=18 ps 10u period σ=26 ps 100u period σ=72 ps

18. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 18 PCB changes needed Few bugs in layout Center pad shorts pads(CRT4T QPLL) –Solution on proto, center pads isolated with kapton tape –Not wanted for series production –Geometry changed: CRT4T done QQPLL done AC coupling in gigabit traces between GOL’s and HFBR772 missing –Capacitors added on proto –To be changed in layout Mirrored geometry JTAG connector (Actel programmer) I2C SCL termination changed All modifications “local” -> very low risk Center pad

19. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 19 VErtex PROcessor Board VEPROB

20. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 20 Optical Ribbon Orc Card VEPROB 24 input fibers Xilinx Virtex2-pro100 FPGA 1704 pins –24 TLK input busses –CCPC Glue-Card interface to ECS –1.6 Gb/s outputs (Xilinx Rocket I/O, MGT) TTCrq timing fast control signals –L0 trigger –LHC clock –BCID reset ECS via CCPC and Glue card Optical Ribbon Orc Card Channel Sync Ethernet CCPC Glue Card Optical transmitters JTAG Xv2pro100 Vertex Finder Opt in. TTCRq Monitor & control VEPROB MGT L0 buffer MGT TFC ECS Output Board TELL1

21. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 21 VEPROB vertex finder Vertex finder algorithm in single FPGA Each event 8 groups of 256 input strips Correlation matrix number of combinations 4082 (left) + 3594 (right) 1 st, 2 nd peak position & height –128 bins (1,2,3,5 mm width) Multiplicity (# active strips) counters & histogramming: 70% FPGA resources Latency: 875ns (35LHCclks) L0 buffer 1 st peak finder Peak masking Vertex finder Beam -1500 150 Z-position Top view detector AlBl BrAr right left Sensor: Decision logic Correlation Matrix + histogram 2 nd peak finder Correlation Matrix + histogram pipeline Strip data Multiplicity

22. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 22 VEPROB PCB details 9U X 400mm single height 12 Layer 1.6 mm PCB –4 power layers (1v8, 2v5, 3v3, GND) –8 routing layers –Stiffener across board Main functionality in Xilinx Virtex2Pro100 FPGA 89% slices used –LUT4 82%, RAM 20%, registers 28% Gigabit LVDS traces & data path ORC to FPGA 50 Ohm controlled impedance Front side I/O –2 optical ribbon, 2 ORC Mezzanines Back side I/O –Power via standard LHC backplane (TELL1) –TTCrq Mezzanine fiber –CCPC & Glue-Card, ethernet –Output to TELL1 buffer –Output to Output Board

23. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 23 VEPROB tests ECS control –Slow controls and monitoring via CCPC and Glue card TTCrq via I2C OK FPGA access registers and memories through local bus OK TTCrq –40 & 80 MHz QPLL LVDS clocks to FPGA OK FPGA / algorithm –Data path OK –Synchronization input links OK –Debug functionalities OK Board temperature cycling test (to be done) –All devices T < 55 C OK Link tests later in presentation

24. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 24 Some remarks Difficulties in assembly of large BGA –Due to board thickness –1704 Ball grid array 1 mm pitch, 45 x 45 mm –Extensive test during solder process with mechanical sample –Solder process under control OK Design flaw in PCB –Missing pull up at FPGA configuration signal (mounted at solder side connector) OK No re-design needed

25. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 25 Link tests

26. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 26 Pile-Up test setup dataflow All boards (proto-types) present for testing @Nikhef only tests done Hybrids kapton cables repeater boards transition boards optical boards VELO ctrl brd specs master ECS pvss, test software C UTB patterngen. VEPROB boards output board L0DU @IP8 default dataflow TTCvx VME cpu Analog Tell1 digital tell1 UTB databuffer VME cpu TTCvi Odin 1 2 3

27. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 27 Interconnection tests Gigabit Optical connections –Optical Tx Board to VEPROB 2 optical ribbons (GOL to ORC TLK2501) –2 fibers out to TELL1 (ORC receivers) Xilinx rocket output -> HFBR5720 -> TLK2501 Gigabit LVDS coaxial output –VEPROB to Output Board Xilinx Rocket output v2pro -> Xilinx Rocket input Xilinx Virtex4 For test: Rocket-out to Rocket-in v2pro MPO-SC cassete SC-MPO cable MPO-MPO 0,6,9dB MPO-MPO MPO-LC cable SMA coax 0,6,9dB MPO-MPO 1 2 3

28. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 28 Gigabit interconnection tests BERR test on all links –Link 1 : 12 output links Optical Tx Board -> Orc card(Veprob) 0 dB 48 h -> 0 bit errors 6 dB att. 60h -> 0 bit errors 9 dB att. 60h -> 0 bit errors 15 dB att. many errors –Link 2: 2 output links, VEPROB -> ORC on Tell1 Xilinx Rocket I/O -> HFBR5720 -> Orc 0 dB 60h -> 0 bit errors 6 dB att. 48h -> 0 bit errors 9 dB att. 60h -> 0 bit errors 15 dB att. many errors –Link 3: 1 Output link VEPROB -> Output Board (Cu link) Tested: Xilinx rocket Output -> Coax VEPROB Xilinx Rocket Input 0 dB 48h -> 0 bit errors 3 dB att. 48h -> 0 bit errors 6 dB att. 60h -> 0 bit errors All links conform specs Link 3 σ=28 ps (electrical) Link 1: σ=57 ps (electrical)

29. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 29 Conclusions Optical Tx Board –All functionality as expected, all tests (excl. thermal) OK –Optical Tx Board PCB needs minor changes, but low risk Ongoing –PCB production time schedule 2 weeks to new layout PCB production 4 weeks Assembly 4 weeks, after all components in house VEPROB –All functionality as expected, all tests (excl. thermal) OK –PCB layout OK, no re-design needed Pull up, FPGA configuration missing, mount on connector –PCB production time schedule 5 boards in house All components in house except $$ FPGA Assembly 4 weeks VEPROB series assembly can be started when FPGA’s are delivered

30. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 30 Outlook Board production ready: 3 months Test programme –Individual board tests 1 month –System tests 1 month (maybe 2)

31. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 31 Spare Slides

32. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 32 Passive boards Kapton cables –In house Repeater boards: 3 different types –1 type produced & tested –2 other types layout done Transition boards: 2 types –inner silicon strips, proto board used in test setup –outer silicon strips, layout done 6U vme 30x30 cm 20x75 cm

33. 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 33 Latency LHC Clocks spec / sim measured Hybrid 50ns2 Cabling to optical station18 m (5ns/m)108 ns5 VETO Optical Station Synchronisation and demultiplexing 4 GOL 64 ns max3 Optical Ribbon 52m (old:60m) (4.5ns/m) 260 ns11 (4m ribbon) Vertex processor board Orx-card(tlk2501) 4 43 channel sync and vertex finder algoritm(xilinx FPGA) 875ns35 (was 42) MGT Rocket i/o transmitter + cable 4 Output board MGT Rocket i/o receiver 4 Mux 1 MGT Rocket i/o transmitter 3 Cable to L0DU 90ns4 Total: 84(87)