CERN TFC status review, January 21, TFC status review l Agenda  Overview  Common solutions  TFC simulation framework  TFC Switch ‘THOR’  Throttle Switch and Throttle OR ‘Munin’  Test and monitoring board ‘FREJA’  LVDS splitter  ECS interface  Readout Supervisor ‘ODIN’  Board production Andrea Borga Zbig Guzik Richard Jacobsson

CERN TFC status review, January 21, Overview VELO L1 FE TTCrx VELO L0 FE TTCrx VELO L0 FE TTCrx Readout Supervisor Readout Supervisor Local trigger L0 L1 Readout Supervisor TFC SwitchL1 Throttle SwitchL0 Throttle Switch TTCtx TTCoc TTC system Clock receiver and fanout LHC clock L1(GbE)L0(LVDS) Trigger splitter VELO L0 FE TTCrx VELO L0 FE TTCrx VELO L1 FE TTCrx TTCoc VELO L1 FE TTCrx VELO L0 FE TTCrx VELO L0 FE TTCrx VELO L0 FE TTCrx ECAL L0 FE TTCrx ECAL L1 FE TTCrx L0 / L1 Throttle OR L1(GbE) TTC monitoring (FREJA) L0 / L1 Throttle OR BPIM

CERN TFC status review, January 21, Common solutions l 9U VME crate and TELL1 backplane  One connector  Use only +5V  -5V, 3.3V and 1.8V made on board  PC power connector  When/how can we get such a crate? l ~2.4mm PCB  1.6mm rails  milling edges  Add holes for rigidity bar l Reset logic to CCPC and Glue card  VME reset from backplane ( Row 17,18,19 (A=B=C) nSysReset)  Front panel push button  Voltage supervisor l Power monitoring circuit  Range 4.7 V – 5.3 V  LED and status bit in FPGA Bottom layer Top layer Front view 3mm 1.6mm 2.4÷2.7mm

CERN TFC status review, January 21, Common solutions l FPGAs  Altera APEX 20K60E, 20K100E, 20K200E, QFP packages  MAX for a few simple tasks: debouncer, JTAG hub, LED driver l ECS interface debugging  RS-232 header on board l FPGA programming  At startup via Altera native programming using reprogrammable configuration device  Directly via JTAG bus from glue card (one or all FPGAs using hub)  Configuration device programmable Via JTAG from glue card (GPIO or JTAG hub) Via onboard JTAG header l Boundary scan FPGA EPC2 Glue JTAG bus GPIO JTAG Altera native programming FPGA 0 EPC2 JTAG hub JTAG 0 JTAG 1 JTAG Altera native programming FPGA 1 EPC2 JTAG Altera native programming JTAG 2 JTAG 3 … Glue JTAG bus JTAG GPIO SELECT HEADER

CERN TFC status review, January 21, Common solutions l Board ID  I 2 C bus 0, 0xA0(write)/0xA1(read) l PLX local bus  20 MHz synchronous mode Separate oscillator Board clock on TFC Switch, Throttle Switch (OR)  16 or 32-bit multiplexed address/data l Boards use same components to the largest extent possible l Hardware design tools  Protel 99SE (all boards except test board ‘FREJA’)  Protel DXP (FREJA) – very easy to upgrade to Protel DXP

CERN TFC status review, January 21, Common solutions l Front panel  Front panel – PCB space 2.5 mm  Front panel thickness 2.7mm  Centre support 15mm x 10x10mm 2 Front panel support Keep out layer (reference for milling machine) board extractor and front panel support Front panel (mechanical 2) Board edge (mechanical 1)

CERN TFC status review, January 21, TFC simulation framework l Based on Visual Elite  Includes LHC bunch generator, L0DU, L1DU, TTCrx, THOR, ODIN, FREJA, Front-End, Glue l Heavily used and maintained up to date  functional and timing simulation CCPC LCLK Q_L0 AFIFO FEB Q_L1 Q_MP Q_FE TTCrs

CERN TFC status review, January 21, TFC Switch ‘THOR’ … … … 16:1 MUX … DELAY … 16 INPUTS 16 OUTPUTS ENABLES … FPGA EPC2 Glue JTAG CCPC LBUS MUX SELECT 16x4 DELAY DATA(6..0) 16 LOAD 16 ENABLE l 16 AC-coupled ECL input l 16 AC-coupled ECL output l Delays: 2.2ns range, 17.5ns step l LEDs:  Power monitor  FPGA status  Output enable (16)  Output activity

CERN TFC status review, January 21, TFC Switch ‘THOR’

CERN TFC status review, January 21, TFC Switch ‘Thor” l Status  Second prototype tested in full chain: RS  TFC Switch  TTCtx  TTCoc  TTCrx/TTCrq Delay characteristics

CERN TFC status review, January 21, TFC Switch ‘THOR’ Skew variation (jitter w.r.t clock) at the output of TTCrx without/with TFC Switch and channel A+B idle

CERN TFC status review, January 21, TFC Switch ‘THOR’ Cycle to cycle jitter and skew variation with respect to TTCrs 40 MHz clock with channel A+B idle using THOR at the output of TTCrm Cycle to cycle jitter and skew variation with respect to TTCrs 40 MHz clock with 100% PRD on channel A+B using THOR at the output of TTCrm Cycle to cycle jitter and skew variation with respect to TTCrs 40 MHz clock with 100% PRD on channel A+B using THOR at the output of TTCrm+QPLL Measurement of the influence at the output when switching input channel, trigger by address line (blue) (without QPLL)

CERN TFC status review, January 21, TFC Switch ‘THOR’ TTC signal at output of Readout Supervisor TTC signal at output of TFC Switch and 40 MHz clock at the output of TTCrx TTC signal at output of TFC Switch after “3 loops” and 40 MHz clock at the output of TTCrx l Summary:  Jitter fine  Timing alignment fine  Stability fine  Signal shapes on long multi-drop chain, distorsion  AC-coupling  Effect of cascading (x3), seems to be OK

CERN TFC status review, January 21, TFC Switch ‘THOR’ Current status CHF - Design of preproduction version ready, being checked - Final VHDL code ready - First version of PVSS control interface ready

CERN TFC status review, January 21, Alt. TFC Switch “Backup solution”: l Switch logic in FPGA  No long external multi-drop chain  > 16x16 switch easy  Fully combinational or resynchronize using PLL  Adjusting timing by manual placing and routing in FPGA l Tested with test board ‘FREJA’  TTC signal through FPGA OK  Jitter OK  Timing alignment OK 16 INPUTS 16 OUTPUTS … … … 16:1 MUX … DELAY … … Glue CCPC EPC2 JTAG LBUS DELAY FPGA

CERN TFC status review, January 21, Throttle network l TELL1  Throttle OR: RJ9 connector with both L0 and L1 throttle (LVDS) l Throttle OR  Throttle OR : LVDS or optical ?  “Isolate crates”  Need optical receivers on throttle OR l Throttle OR  Throttle Switch: Optical optionally LVDS  Throttle OR will have both optical and LVDS outputs l Throttle Switch  Readout Supervisor: LVDS L0 / L1 Throttle OR 1/crate 2 L0 & L1 throttles (LVDS) 2 2 … 2 20 TELL1 boards L0 / L1 Throttle OR … 2 20 TELL1 boards L0 / L1 Throttle OR 2 2 L0 & L1 throttles (LVDS or optical) L1 Trottle Switch L0 Throttle Switch Detector 1 … … 16 detectors L1 throttles (optical or opt. LVDS) L0 throttles (optical or opt. LVDS) Readout Supervisor … L0 & L1 throttles (LVDS) … …

CERN TFC status review, January 21, Throttle network l Optical transmitter and receiver  Siemens SFH-756 and SFH-551 l Optical fibre  660 nm, 2.2/1.0 mm plastic fibre  Looking for fibre satisfying CERN safety regulations l Tests:  1 MHz over 50 m OK  Fibre cutting easy, works even with a pair of scissors… l Why not common Throttle Switch for L0 and L1?  Limited front panel space if optical, OK for LVDS

CERN TFC status review, January 21, Throttle Switch l Inputs  16 optical receivers on mezzanine  alternatively 16 inputs LVDS on pair 0 of RJ9 (back)  additional 16 inputs LVDS on pair 1 of RJ9 (back) l Outputs  16 LVDS on pair 0 of RJ9 (front) (L0)  16 LVDS on pair 1 of RJ9 (front) (L1) l History buffer  16 bits current input states  16 bits current output states  48 bits time counter  16 bits frame counter  64/3 k events deep l LEDs  Power monitor  FPGA status  2 FIFO status  16 input enables  16 input activity 16 x RJ9 (FRONT) 16 INPUT (LVDS) 16 INPUT (OPTICAL) Glue CCPC EPC2 JTAG LBUS FPGA History buffer LBUS 16x MUX 2:1 16 INPUT (LVDS) SELECT { 16 x RJ9 (BACK) 16 OPTICAL (MEZZANINE) { 16 OUTPUT (LVDS) 16 OUTPUT (LVDS) } LATCH ENABLE DERANDOMIZER

CERN TFC status review, January 21, Throttle Switch * Current status - PCB production launched end of November, only mezzanine received - First board with edge milling and TELL1 back plane - Code for self test in preparation - First version of FPGA code ready (AHDL)

CERN TFC status review, January 21, Throttle Switch ‘MUNIN’

CERN TFC status review, January 21, Throttle OR l Throttle OR only a variation of Throttle Switch  Design to be finalized when Throttle Switch debugged  Inputs 20 L0 throttle LVDS on pair 0 of RJ9 (front) 20 L1 throttle LVDS on pair 1 of RJ9 (front) L0 and L1 throttle optical input for cascading via optical (2-3x)?  Outputs L0 throttle optical output L1 throttle optical output 2 L0/L1 throttle LVDS output  Same history buffer as Throttle Switch and same LEDs

CERN TFC status review, January 21, Trigger splitter l L0DU  RS : 1:3 splitter of the 16-bit L0 LVDS signal  Based on six Fairchild FIN1108, 8-port high-speed LVDS repeaters  Input designed as multi-drop configuration  Take care of equalizing nets  9U or 6U?  Using the back we could put two 1:3 splitters on a 9U L0 decisions BPIM information or detector status …. 8 channel LVDS repeater …. Terminations ….…. OUTPUT 3 (16) OUTPUT 1 (16) INPUT (16)

CERN TFC status review, January 21, Trigger splitter

CERN TFC status review, January 21, Test board ‘FREJA’ TTC FIBRE EXT CLOCK (ECL) INT CLOCK Q_FREJA TTCrx FPGA MEZZANINE TTC DATA BUFFER FIFO I²C GPIO (LVDS) GPIO (ECL) CLOCK SELECT TTCrs TTC Glue CCPC EPC2 JTAG LBUS L0DU L1DU L1FE L0FE FREJA Readout Supervisor TTC L1 decision unit L0 decision unit L0 trigger generator BID counter FIFO L1 latency L1 generator L1 interval L0 decision BID [11..0] L1 decision BID [1..0] L0 event ID[12..0] Front End unit BID counter Event ID counter L0 derandomizer Readout Pipeline [12x160] External buffer L0 Accept Channel B decoder Test setup Board block diagram FPGA code

CERN TFC status review, January 21, Test board ‘FREJA’ l * Current status  CHF  - Fully debugged and working  In use for tests  - Ready to produce one more  - PVSS interface and automatic  checking to be prepared

CERN TFC status review, January 21, ECS interface l CCPC works very well Used alternative Glue card (“Glue light ”)  FPGA emulating PLX, full code in VHDL, no metastability  43 MHz  40mm x 85mm  Pin compatible with glue card specs  Busses: full PLX local bus, 1 JTAG, 1 I2C, GPIO  Cover all needs for TFC system l Programming, configuration, monitoring working very well  STAPL player using PCI library  Local bus register read/write program using PCI library  “I2C read/write driver”  DIM server for local bus and JTAG access written

CERN TFC status review, January 21, Glue card * Status - Backup solution - Fully debugged and working - “Production ready” FPGA Regulator 2.5V Regulator 3.3V EPC2 ByteBlaster PCI PARALLEL PORT R2-232 driver LOCAL BUS GPIO(8..0) JTAG I2CI2C RS V SERIAL PORT

CERN TFC status review, January 21, Readout Supervisor ‘ODIN’ BUNCH_CURRENT TTCrs Q_MP Q_L0 Q_L1 TTCrx AFIFO FE_BUFFER DETECTOR_STATUS DI (EGRESS) Q_FE GbE L1BROADCAST & COMMANDS TTC_DATA L0_TRIGGER L0_DATA L0_TRG L0_DATA L1_DATA L1_TRG LHC_TTC TTC GPS DI (INGRESS) GbE Q_L0 (67 %): LBUS ODIN_bid ODIN_eid ODIN_L0_pipeline ODIN_L0_resynchronise ODIN_L0_synch_chk ODIN_handling.vhd ODIN_l0d_emulator ODIN_gap_generator ODIN_per_trg ODIN_rnd_generator ODIN_L0_counters ODIN_bx_pipeline ODIN_bx_type Q_L1(13 %): LBUS ODIN_ingress_pll ODIN_L1_trigger_GBE ODIN_L1_trigger_LVDS ODIN_L1_internal ODIN_AFIFO_read ODIN_L1_synch_chk ODIN_L1_request ODIN_l1b_emulator ODIN_L1_counters Q_MP(26 %): LBUS ODIN_orbit ODIN_bcr_ecr ODIN_ecr_internal ODIN_cal_trg ODIN_l0e_reset ODIN_l01e_reset ODIN_per_cmd ODIN_IP_broadcast ODIN_broadcaster ODIN_MP_counters ODIN_TTCrx Q_FE: LBUS ODIN_egress_pll Read_feb ODIN_read_feb ODIN_read_L1data ODIN_data_format ODIN_IP_handling ODIN_GBE_ctrl ODIN_DAQ_link ODIN_FE_counters FPGA FIFO MEZZANINE FPGA MEZZANINE Other TTCrs I2C register (SRES etc) JTAG hub and star

CERN TFC status review, January 21, Readout Supervisor ‘ODIN’ * 9000 CHF (includes TTCrs)

CERN TFC status review, January 21, Readout Supervisor ‘ODIN’ l TTCrs mezzanine  Implement on motherboard  Problem of MHz or MHz (QPLL acceptance – MHz) ? l GbE card power  Add regulator for 3.3V l Programming  Add configuration device for each FPGA l Backplane  TELL1 backplane l Front panel l Improve JTAG bus l Detector status  LVDS l Remove optical throttle inputs l Remake PVSS interface for current version

CERN TFC status review, January 21, Readout Supervisor upgrade l L0 accept buffer  IDT 70V659S, 128K x 36 bits (SRAM DPM) l L1 trigger sorter  L1 decision buffer K7M163625A (2x) 512K X 36 bits (SRAM SPM)  First version of VHDL code written and simulated l Links to TELL1  TLK2501 design block from Marseille BUNCH_CURRENT TTCrs Q_MP Q_L0 Q_L1 TTCrx L0 ABUF DETECTOR STATUS L0 LINK Q_L0FE L1BROADCAST & COMMANDS TTC_DATA L0 TRIGGER L0 DATA L0_TRIGGER L0_DATA L1 DATA L1_TRG L1_TRIGGER LHC TTC(BST) TTC GPS GbE RX GBE L1 LINK FPGA RAM MEZZANINE FPGA MEZZANINE L1 DBUF RAM L0 DATA L1 DATA

CERN TFC status review, January 21, Readout Supervisor plans l More testing! l Production:  Mount one more of current version (PCB in cupboard) for Marseille?  Produce more of current version for other people?  Correct current version for final production?  Upgrade current version for final production?

CERN TFC status review, January 21, Production l PCB rules l Common schematics and PCB libraries for all boards  Footprints optimized l Component database l Standard Gerber files l Files for “dotmaster” l Choosing outside company to produce PCB l How much spare components to order?

CERN TFC status review, January 21, Installation Patch panels for TTC fibres