1. FTK AM boards Design Review P. Luciano, N. Biesuz, W. Billereau, J.M. Combe, S. Citraro, D. Dimas, S. Donati, C. Gentsos, P. Giannetti, K. Kordas, A. Lanza, D. Magalotti, P. Neroutsos, S. Nikolaidis, M. Piendibene, E. Rossi, A. Sakellariou, C.-L. Sotiropoulou, P. Vulliez 1

2. The Goal of this review We need a new motherboard version, AMBSLP_V3 for different reasons:  Increase and optimize the power distribution  Improve the air flow for cooling (remove obstacles to air)  Provide capability to program FPGAs from VME  Add a flash ram to make shorter the Amchip configuration time ……. Other details This is the right moment to get your comments to include solutions in the next board version. The design we present is not final 2

3. Outline Core Crate power Architecture of AMBSLP: Associative Memory Board Serial Link Processor Detailed description of power distribution on Board & PCB design FPGA design LAMBSLP: PCB design Measurments & Tests Serial links AMchip configuration 3

4. FTK IAPP TASKS: CLUSTERING PM BOARD

5. Detector Front End PU AUX CARD AMBSLP LAMBSLP To DAQ memories GLOSSARY IN OUT AMBSlim AMBSLP generations of AM boards For Cooling tests FTK ~final prototype Processing Unit Serial Link Processor Data Formatter SLP2

6. Plan for the power & resource sharing quotations custom CAEN (~12k€) and std WIENER (~14 k €) 5 V: 120 A by CAEN PS (33Amargin, 73% usage), 115 A by Wiener – For: SBC, each board VME transeivers and SSBs: 7 A for the SBC (7A at boot) 16 A for the 16 AMboards (1 A/board ) 64 A for the 4 SSBs (16 A/board) 12 V: 330 A by CAEN PS (85 A margin, 74% usage), 184 A by Wiener – For AM core: 2,4 W*64=192W  13,8 A/AMB  15,3 A (1.28 A/pin) for eff=0,85  245A/crate – For FPGA power ~12 W  ~ 1 A 48 V: 84 A provided by CAEN PS (~31 A margin, 63% usage), 67,5 A by Wiener – For AM I/O (2.5 V): ~52,5 W/AMB  840 W/crate  988 W/crate(eff 0,85)  20,6 A – For AM 1.2 V: ~6,5 W/AMB  122 W/crate (eff. 0,85)  2,54 A – For AUX: 75 W/AUX  1200 W/crate  1412 W/crate (eff 0,85)  29 A – Super TOT = 53 A/crate 6

7. 7 WIENER 3,3V, 12 V, V1,V2pins @ 70 o : 1,2 A/pin

8. 8

9. How power is distributed on the Backplane J0 1,5 A/pin desired 10+2 12V pins (blue/green) in J1  18 A*12V/Slot  216 W 10 48 V pins (orange/green) in J1, J0  15A*48V= 720 W/slot16 5V pins (red) in J1, J2 and J0 connectors  24 A*5V/slot  120W

10. II OPTION: use 48 V instead of 12 V for Amchip core Use a 350 W DC-DC converter from 48 V to 12 V – Option A http://www.digikey.it/product-detail/it/NQB-420DWA-AN/102-2674-ND/3910892 Option A http://www.digikey.it/product-detail/it/NQB-420DWA-AN/102-2674-ND/3910892 – Option B http://www.digikey.it/product-detail/it/IQG48033A120V-1D9-R/285-2227-ND/3908550 Option B http://www.digikey.it/product-detail/it/IQG48033A120V-1D9-R/285-2227-ND/3908550 – Option C http://www.digikey.it/product-detail/it/QBDW033A0B41-HZ/555-1271-ND/2770696http://www.digikey.it/product-detail/it/QBDW033A0B41-HZ/555-1271-ND/2770696 from 12 V to 2,5 V – http://www.digikey.it/product-detail/it/SIL40C2-00SADJ-HJ/454-1344-ND/2347886 http://www.digikey.it/product-detail/it/SIL40C2-00SADJ-HJ/454-1344-ND/2347886 from 12 V to 1 V – http://www.digikey.it/product-detail/it/UVT020A0X3-SRZ/555-1257-1-ND/2665528 http://www.digikey.it/product-detail/it/UVT020A0X3-SRZ/555-1257-1-ND/2665528 from 12 V to 1.2 V 1.8 http://www.digikey.it/product-detail/it/UDXS1212A0X3-SRZ/555-1336-1-ND/4696382 10

11. 11 Airflow from pin 3 to pin 1 Airflow from pin 1 to pin 3 Option B Option A Critical issue: output current derating with temperature and more critical power dissipation Option C

12. Instead: GE DC-DC 12V  1V 40 A (GE in general starting from 12 V) 12 Part number: MVT040A0X3-‐SRPHZ They are fine with 0 air flow full power up to ~65 deg ambient temperature

13. Temperature Measurements in LAB 13 3 DC-DC converters working in parallel to generate 100 A on a appropriate LOAD, working for hours without fan! ALL TEMPERATURES BELOW 70 degrees without air flow!

14. AMBSLP 14 12-> 1,2 -1,8 48->12 12->2,5 12->1 V OPTION: AMCHIP core from 12 V OPTION: AMCHIP core from 48 V

15. Architecture of the AMBSLP 15

16. AMBSLP Serial Links @ 2Gb/s VME 9 U Clock @100MHz Supply Voltages: 2,5V 1,8 V 1,2V 1V Power consumption: ~ 250 Watt 16

17. AMBSLP Serial Links @ 2Gb/s VME 9 U Clock @100MHz Supply Voltages: 2,5V 1,8 V 1,2V 1V Power consumption: ~ 250 Watt 17 AMB distributes SSs to 64 AMchips collect roads and send them to the AUX 12 Slinks 8 Slinks 4 Slinks

18. AMBSLP Serial Links @ 2Gb/s VME 9 U Clock @100MHz Supply Voltages: 2,5V 1,8 V 1,2V 1V Power consumption: ~ 250 Watt 18

19. Mircea Bogdan, November 11, 2014 19 FTK Rear Transition Module -VME Block Diagram for a typical VME Slave Module Interface. The Rear Transition Module (RTM) receives power from the crate, but is not part of the VME data transfer bus. Generally, the RTMs can not be accessed via the Crate CPU, and are used only to bring IO to the front processing module. Addr[26:0] Data[31:0] Addr[26:0] Ctrl lines To Lambs Data[31:0]

20. Block Transfer inplemented 20 ADDED a flash ram where we will save the AM BANK inside the AMBSLP Time to configure an AMBSLP will be ~= time due to configure AUX (AM bank dominates) once JTAG will be driven by VME chip firmware  A flash RAM is added to AMBSLP to be able to write at the same time in the AMBSLP & AUX Re-program firmware through VME not implemented yet, but the JTAG chain of FPGAs will be connected to the VME chip on the next board version Note: full-board configuration rarely needed (eg: power cycle / change patterns) (few/year ) FLASH RAM S34MS01G2_04G2: 4 Gb for 1 Gb bank (144 bits * 128kpatterns*64 chips)

21. AMBSLP VME 9 U Serial Links @ 2Gb/s Clock @100MHz Supply Voltages: 2,5V 1,8 V 1,2V 1V Power consumption: ~ 250 Watt 21

22. AMBSLP VME 9 U Serial Links @ 2Gb/s Clock @100MHz Supply Voltages: 2,5V from 48V 1,2V from 48 V 1V from 12 V 1,8 from 12V Power consumption: ~ 250 Watt Astec Industry Standard ALD25K48 ALD20G48 22

23. 23 Improving the AMBSLP layout and placing the 4 SSBs in the 4 slots with lower air flow (see next talk) NEW T-SIMULATION Instead of LDOs

24. AM power summary 2.5 V typ 24A (measured) max 40A (2 DC-DC ALD20G48 2x20 A) – 2.5 V power dominated (80%) by AMchips and ser-des fanouts – “Typical” values are measured to be very stable From Amchip miniasic and miniLAMB – extrapolated to AM06 5  11 ser or des From LAMBSLP with AM05 Real fanout chips on LAMBSLP 1.2 V typ 12-13 A max 18A (1 DC-DC ALD25K48 1x25 A) 1.0 V for FPGA’s core (non Amchips) estimated~10 W – Taken from 12 V (1 DC-DC UDXS1212A0X3-SRZ for 12 A) 1.0 V Amchips 3 DC-DC (MVT040A0X3-‐SRPHZ) in parallel for 2 LAMBs – ongoing tests before PCB submission ( next review on AMchip, 28 November, will have full core consumption discussion, see Backup slides for a summary) 24

25. MVT040A0X3-‐SRPHZ 25

26. DC-DC converter’s ripple 1.0 V power supply: Amchip requires max ripple ±40mV (max 80mV pp) – 3 DC-DC (MVT040A0X3-‐SRPHZ) for 2 LAMBs – Assuming 2 max Capacitance is 2*7mF (-10%) for 32 Amchips – 390 uF per Amchip (6x100nF, 4x4.7uF, 3x100uF) 1.2 V power supply (serdes core): Amchip requires max ripple ±40mV (max 80mV pp) – 1 DC-DC (ALD25K48) for 4 LAMBs + AMB – Max Capacitance is 10mF for 64 Amchips + 8 FPGAs – 130 uF per Amchip or FPGA (3x100nF, 4x4.7uF, 1x100uF) 2.5 V power supply (analog IO): Amchip requires max ripple ±50mV (max 100mV pp) – 1 DC-DC (ALD20G48) for 2 LAMBs + ½ AMB – Max Capacitance is 10mF for 32 Amchips + 4 FPGAs – 260 uF per Amchip or FPGA (5x100nF, 4x4.7uF, 2x100uF) 26

27. 48 V distribution in input to 4 DC-DC converters FUSE 27 GROUND large plane on layers that have the Serial Links

28. 12 V distribution in input to 4 DC-DC converters now 7 DC-DC converters in the next version 28

29. FPGAs 29

30. HIT & ROAD Logic FIFO Dual Clock GTP RX Module Data Processing & Monitoring GTP TX Module FIFO Dual Clock HIT_BUS SYSTEM_CLOCK RX_GTP_CLOCKTX_GTP_CLOCK Parallelized Data FIFO VME HITs from AUX HITs to LAMBs SpyB FIFO Dual Clock GTP RX Module Data Processing & Monitoring GTP TX Module FIFO Dual Clock ROAD_BUS SYSTEM_CLOCK RX_GTP_CLOCKTX_GTP_CLOCK Parallelized Data FIFO VME LAMBLAMB ROADs from LAMBs ROAD FPGA SpyB ROADs to AUX 30

31. DATA FORMAT Type of wordValueK character IDLE WORDBCBC1C1C1111 HITS DATA - 2 words (XXXX & YYYY 16 bits each) XXXXYYYY 0000 End Event WORD RR= error bits NNNN=L1ID F7RRNNNN1000 From AUX to HIT Type of wordValueK character IDLE WORDBCBC1C1C1111 HITS DATA XXXXYYYY 0000 End Event WORDF7RRNNNN1000 From HIT to AMchip Type of wordValueK character IDLE WORDBCBC1C1C1111 ROAD DATA -BB=bitmap AAAAAA= road address BBAAAAAA0000 From Amchip to ROAD Type of wordValueK character IDLE WORD0000BC500010 ROAD DATABBAAAAAA0000 END EVENT WORDF7RRMMMM1000 From ROAD to AUX 31

32. CONTROL CHIP 32

33. 33 AUX specs: –The expected number of hits for a 70 pile-up WH event is < 650 per layer, or 1 hit per 15 ns. (The 8 layers are handled in parallel.) –AUX specification is 1 hit per 5 ns. AMB spec is: 2 reduced resolution HITs each 10 ns –The expected rate of roads from the AMB into a processor chip is < 800/event, or 1 road per 13 ns. –The specification is 1 road per 5 ns/AUX processor. AMB spec is: 1 road each 5 ns/AUX processor Processing Rate Requirements From AUX review https://indico.cern.ch/event/349945/ November 11, 2014AUX design review

34. November 11, 2014AUX design review34 Serial link data rates: WH 70 pile-up MC (80 pile-up with full wildcard usage to be produced) Specification Tested to BER<10 -15 < 2 Gbps 6 Gbps 6.4 Gbps < 0.6 Gbps 2 Gbps < 2 Gbps 6 Gbps 6.4 Gbps < 0.85 Gbps 2 Gbps 1.2 Gbps 6 Gbps 6.4 Gbps 2 Gbps 6 Gbps 6.4 Gbps

35. FPGA used resources 35 SPY Buffers: 8 k 32 bits words / input or output serial link Input Fifos: 1 k 32+4 bits words / input serial link VME Fifos: 8k 32+4 bits words / input or output serial link HIT, ROAD = XC7A200T-3FFG1156E – CTR=XC6SLX16-3CSG324C -- VME =XC6SLX45T-3CSG484C

36. PCB details Technical data: (1) Base material FR4 tg180°; (2) Surface finishing ENIG; 14 layers (8 GND&VCC, 6 signal layers); board size: 416 m 2 X 367 m 2 X 2.3 m 2 ; 50 Ω lines for the serial link distribution; 100 Ω differential impedence  Cadence Trasmission Line Calculator Vias: (1) 0.25 mm under BGAs (0.8 mm pitch); (2) 0.6 mm for high current; (3) 0,4 mm for generic vias 36 22.5 o rotation of the substrate  expensive Eventually the company suggests: FR408 or FR408HR Er = 3,7

37. LAMBSLP 37

38. LAMBSLP W 16 AM05 1V core full custom & stdcells 2,5 V I/O 1,2 V stdcells for SerDes 38

39. Lamb architetcure VMEDATA[31:24]LAMB++ 3 VMEDATA[23:16]LAMB++ 2 VMEDATA[15:8]LAMB++ 1 VMEDATA[7:0]LAMB++ 0 VMEDATA[31] AM_chain 7 VMEDATA[30] AM_chain 6 VMEDATA[29] AM_chain 5 VMEDATA[28] AM_chain 4 VMEDATA[27] AM_chain 3 VMEDATA[26] AM_chain 2 VMEDATA[25] AM_chain 1 VMEDATA[24] AM_chain 0 e.g. LAMB3 39 HOLD IMPLEMENTED FPGA for VME to AMchip JTAG interface:1 VME operation will access in parallel 32 AMchip chains

40. The High Frequency connector 40 SAMTEC ASP-134486-01 ASP-134488-01 1V 1,2V 2,5V Signals

41. Amchip BGA 41 GROUND large plane on TOP layer Capacitances on bottom layer 1V – YELLOW – WHITE 2,5 V – RED GND - BLUE SIGNALS - GREEN

42. Power Layers – PCB details 1 V 1 V (core), 1,2 V full plane (serial link core), 2,5 full plane V (I/O) FR4 Tg150°; 12 layers; board thickness 1,80 mm; Base material FR4 tg180°; Surface finishing ENIG 42 Next version will be a full plane

43. LAMB cross section 43

44. WHAT WE HAVE today: 44 1 AMBSLP_v1 (Melbourne) 3 AMBSLP_v2 (Pisa, AUTH, CERN) 2 MiniLAmb 3 LAMBs: 2 w 4 AM05s (Melbourne, AUTH), 1 w 16 AM05s (Pisa) MOTHERBOARD MEZZANINE

45. TESTS 45

46. Test STANS First with MiniLAMB and mini@sic Second with LAMB and AM05 46

47. miniASIC Configuration Test VME Config Procedure: CPU VME Conversion JTAG 47

48. Measures in Frascati: Jitter Analysis BER Eye diagram Good results input path No Good results output Input Measure Output Measure AMBSLP_v1: Measures 48

49. AMBSLP_v1: Measures Result 49

50. AMBSLP_v2: Second step Test Serial Link @ 2 Gb/s Input path Output path Send PRBS data PRBS checker BER < 10 -14 50

51. AMBSLP_v2: Third step Test All serial links are working, a particular reset procedure was implemented PRBS checker  BER BER < 10 -14 Patterns correctly downloaded and fired, but firmware has to be adjusted for some layers VME working, also on AUX and LAMB, Block Transfer implemented. SPY BUFFERs working and Lost Synchronization CHECK implemented. 51

52. Conclusions: NEXT STEPS The system is working, but no extensive tests done up to now We are near to submit a new PCB for both LAMBSLP and AMBSLP: – LAMBSLP: all serial links moved on internal layers, number of capacitances reduced to DC-DC converters allowed maximum … + other details…. – AMBSLP: (1) make it compatible with T-sim suggestions, (2) improve power for AM core, (3) improve pattern downloading speed (flash ram added on board), (4) add capability to dowmload firmware from VME, + other details …. We ask permission to buy the FPGAs before building the last version of AMBSLP 52

53. BACKUP 53

54. 1/3-AMchip info (AMchip review Friday 28) AM chip power – 2.5V typ 90mA max 200mA – 1.2V typ 72mA max 120mA – Typical power from 2.5V and 1.2V ~0.32W Typical values measured at 2.0Gb/s “normal conditions” Max values are worst combination of datasheet typical values at 2.4Gb/s – 1.0V highly data dependent: core power can go from very low to full power when data arrives (and then back at stop) 54

55. 2/3 - Amchip core 1.0 V power possible peaks – highly data dependent – Nominal: 2.3A (extrap. from AMchip05 measurement) Assume input data at full speed 100MHz Assume 50% bit swapping 100% bit swapping would double this value! – Add consumption of glue logic (top level) + xx mA – Add worst case (temperature, voltage, process)*1.2 – Expected Total ~3 A – 100% bit swap would be huge (double) – Can we cover 50% bit swap in worst case cond.? Max available current 3.3A/chip (105A from 3 DC-DC for 32 Amchips) but current per connector pin > 1,6 A  too high  3A/chip limit  3.3 A is feasible if the PS voltage goes from 12 V  14 V. 55

56. 3/3 - More realistic Amchip power We cover 50% bit swap at 100MHz in worst case conditions for power distribution What is a realistic upper limit for power dissipation? – 2W typ (2.6W realist max) – See next slide 56

57. realistic upper limit for power dissipation AMchip limit ~ 960 FTK used Layers: Pix0,1,2, SCT0, 2, 4, 5,6 Less than 50% usage Worst case usage channels 0-7: IBL 800 words - Pix0 700 words Pix1 ~500 words Pix2 ~400 words Lay3-67 4*< 400 words Total 4300/8000 ~ only 54 MHz instead of 100 MHz available  ~half of the clock cycles have data If needed interlock on data rate can be implemented in the AM board. ttbar mu=80 IBL Expected power ~< 2W 57 Nhits Towers Layers Pixel 0

58. About AM06 core power Power proportional to the number of hits – Also proportional to the number of switching bits – Assume 50% switch probability for all 16 input bits – We will probably use only about 12-14 bits depending on layer Average hits per event – 2900 FTK TDR table 5 page 77 endcap (extrapolated to mu=80) – 3200 Last slide here rounded up – 4300 Last slide here worst case with IBL rounded up AMchip estimated “worst case” core power ~3W [full input rate (8000 hits/ev), 50% probability swap for all 16 bits, Typical consumption (2.5 W)+20% to cover worst case] Expected power at the simulated input rate (80 PU) → 3W*3200/8000= 1.2W Goal: 80% maximum speed 3W*2*3200/8000= 2.4W core – FTK system requirement 70% Design power distribution for 2.4W core max usage (80% worst case) Design cooling for 2.4W = core (80% typ) + IO: Power/chip = 2.5W typ core * 0.8 + 0.4 IO = 2.4W total 58