1. Optical TMB Mezzanine Production Readiness Review J. Gilmore N. Amin, V. Khotilovich, V. Krutelyov, A. Safonov, A. Schneider, I. Suarez (Texas A&M University) 24 May 2013 1

2. Outline Design overview & development of the OTMB Baseline Testing – Communications, timing, radiation testing Performance Testing – Firmware, performance of the TMB in the full system, monitoring and validation Operational aspects: – OTMB base board, cooling and power Production & commissioning test plans 2

3. ME1/1 DAQ and Trigger Electronics 3

4. Trigger Motherboard Mezzanine Upgrade The new TMB, a.k.a. OTMB, features – Receives fiber optic data from 7 DCFEBs using 3.2 Gb/s links on ME1/1 CSCs  Optical Trigger Motherboard interface is required for the new DCFEBs – Provides a programmable logic platform that allows for improvements to the CSC trigger algorithm  Necessary for efficient HL-LHC trigger operation – Full backwards compatibility with the copper cable data links used on other CSCs 4

5. Optical Trigger Motherboard R&D OTMB Features: – 12-layer PCB, with impedance control layers Fabrication & Assembly by Sierra Circuits – Virtex-6 FPGA – Snap12 optical receiver – TI voltage translators Board design stable for over two years – 2 early prototype boards Extra features and components specifically for rad testing – 4 prototypes shortly thereafter Extensively used for testing and commissioning Very successful version, only a few minor modifications in the design since – 4 more final prototype boards Rad-tolerant regulators & other minor improvements – Thousands of hours of reliable operation under various conditions Test stands at CERN (B904 & SX5), TAMU, OSU, Rice, UCLA 5 Final OTMB Mez

6. 6 6 Snap12 Fiber Receiver - fibers from 7 CFEBs Snap12 Fiber Transmitter -only installed for special tests I/O Voltage-level shifters, 3.3 V to 2.5 V Virtex-6 FPGA + PROM QPLL PCB Dimensions: 7.5” long by 5.25” wide 11 mm clearance from TMB base board Final Mezzanine Prototype Board This is exactly how the production OTMB mezzanine will look

7. OTMB: BASELINE TESTS AND RADIATION TESTS 7

8. OTMB: Communication Tests Baseline tests using custom firmware & software: – Fiber transmission tests with DCFEBs  Fixed patterns and PRBS patterns  No errors detected – Measurements of the fiber transmission latency  Serialize/deserialize logic requires 5 BX  This is 2.5 BX increase, but still 1.5 BX from the critical limit – Tests of communications with DMB, CCB & RAT  Carried over the crate backplane Other baseline tests use standard CSC firmware & software: – Pattern tests on copper cable inputs with CFEBs  No errors detected – Data readout via DMB  No errors detected – Trigger setup & control via VME and CCB  No errors detected – Timing window/phase delay tests to MPC and ALCT  Success, timing windows identified Full performance achieved with the prototypes – Over 2 years of operational experience with these boards 8 DCFEB Board

9. ALCT – OTMB Communication Timing ACLT-TMB timing using standard CSC tools in the test stand at Bat. 904: – Scan the adjustable phase delays for transmit clock vs receive clock  As expected, reproduces the performance of existing CSC electronics  Successful error-free communication for a broad (7x6) window  Easily surpasses the 3ns x 3ns minimum window requirement 9 7 x 6 error-free TMB-ALCT timing window 7 x 6 error-free TMB-ALCT timing window

10. Voltage Regulator Radiation Tests Testing performed at the Texas A&M Nuclear Science Center – 1 megawatt reactor operating at 6 kW, provides 9.9 *10 8 n/cm 2 s Multiple samples of several COTS regulators, two exposures – First exposure represents ~10 HL-LHC year dose (10 krad) – Second exposure adds ~20 HL-LHC years, total of 30 year dose (30 krad) – Regulator performance tested before and after each exposure  Regulators were unpowered during exposure Several parts showed perfect results – These are used in new electronics  National Semi LP38501 and LP38853  Micrel 49500 and 69502  TI TPS74901 Other parts showed signs of failure – We don’t use these in our final design  Maxim 8557  Sharp PQ035ZN1, PQ05VY053, PQ070XZ  TI TPS75601, TPS75901 10

11. SEU Test Results A special “beam-test” version of firmware was used TI Bus-Exchange Level-Shifter: sn74cb3t16212 – No SEU observed. Combined 2011+2012 result:   < 4.0 *10 -12 cm 2 Reflex Photonics 3.5 Gbps Snap12 Receiver: r12-c00501 – Random PRBG data patterns @3.2 Gbps on each of eight links – These SEUs only caused transient bit errors in the data – Combined 2011+2012 test result:  = 9.5 *10 -10 cm 2 per link  HL-LHC: expect ~1 SEU/week/link 11

12. FPGA SEU Test Results Tests with intentionally heavy use of FPGA resources have been designed to ensure that the estimates are conservative GTX Transceiver (55% are used in the FPGA) – Random PRBS data patterns @3.2 Gbps on each of eight links – These SEUs only caused transient bit errors in the data – 2012 GTX SEU cross section result:  = (10 ± 0.8) *10 -10 cm 2  HL-LHC: expect ~3 SEU/year/link Block RAM (74% are used) – Use built-in Xilinx ECC feature to protect data integrity – Software controlled the writes and reads for BRAM memory tests – No errors were detected in the BRAM contents: mitigation at work – 2012 BRAM SEU cross section result:   < 8.2 *10 -10 cm 2 CLB (43% are used) – With only “partial” mitigation, the rate reduced to an acceptable level  Most of the logic is a shift register system with voting  Some of the control and monitoring logic remained un-voted – 2012 CLB SEU cross section result:  = (6.0 ± 0.5) *10 -9 cm 2  HL-LHC: expect ~1 CLB SEU per FPGA per day at HL-LHC  Factor of 6 lower rate compared to 2011 SEU tests (no mitigation attempt at all in 2011) 12

13. OTMB OPERATIONAL PERFORMANCE TESTING: THE FULLY INTEGRATED SYSTEM 13

14. B904: ME1/1 System Testing Since Summer 2012 ME1/1 Test Chamber Crate DCFEBs TMB with new Mezzanine board 14

15. OTMB Firmware A year of testing and validation using an early version of the firmware – A fully functional version including optical features, but supporting 5-DCFEB operations mode instead of 7 The complete 7-DCFEB trigger firmware released and extensively tested for over two months – A completely realistic full-chamber trigger & DAQ operating regime allowing exhaustive testing  In the absence of the final ODMB board, a clever workaround: use two DMBs in parallel for 7 DCFEB control & DAQ – The focus turned to performance testing the system  All the analysis & DQM software required expansion to “7”  The CSC trigger emulator was also updated for 7 DCFEBs 15 2xDMB = ODMB solution 2xDMB = ODMB solution

16. TMB Functional Diagram TMB Inputs: – ALCT wire stubs – DCFEB hits in layers TMB Output: – 3D LCT stubs ALCT wire “stubs” DCFEBs comparator “hits” TMB: Receive wire “stubs” Receive strip hits & make stubs Match wire and strip stubs TMB: Receive wire “stubs” Receive strip hits & make stubs Match wire and strip stubs comparator “hits” 3D LCT “stubs” 16

17. OTMB Trigger Performance Collect cosmic data at B904 using an ME1/1 chamber with new electronics – Full 7 DCFEB electronics configuration  Use OTMB to trigger – The data saved contains low level trigger primitives (the raw “inputs”) Run the TMB Trigger Emulator on raw data – Software based trigger emulation – Takes the same “inputs” as the ones used by the TMB Comparison of the data and emulation is an ultimate test of performance – Enables detailed comparisons of the “primitives” including reconstructed positions, stub parameters, quality, etc.  Much deeper than just verifying the OTMB “trigger decision” ME1/b ME1/a ME1/1  =1.55  =2.09  =2.43 17

18. Emulator vs OTMB half-strip positions LCT Occupancy with the OTMB The peculiarities in the shape are expected and follow the unusual geometry of ME-1/1 – The high half-strip numbers correspond to the bottom “a” part of the chamber – Slanted wire orientation causes some wire groups to cross both ends of the chamber No errors with the statistics of ~60k cosmic events – An exact match of the OTMB calculations and the prediction Emulator-predicted LCT stub positions LCT stub positions reported by OTMB 18

19. Finer Look Into the Details Deeper level investigations are illustrated by comparing the reconstructed stub quality ranking – Essentially determination of the CLCT hit patterns (straight tracks are high pT and receive higher quality rank) The agreement is exact Conclusion: tests on a chamber with new electronics in Bat. 904 show that OTMB and its firmware work exactly as expected – The error rate is estimated to be less than 1 in 60,000 19

20. Detailed Testing and Monitoring Data Quality Monitoring (DQM) System – The production level system used by CMS to monitor the quality of the data has been fully adapted for the new ME1/1 system  Another major milestone now turned into a regular routine 20

21. ME-1/1 Event Display: A Typical Cosmic Event 21

22. A Not So Typical Event: a Di-Muon! 22

23. OPERATIONAL ASPECTS: OTMB BASE BOARD, TEMPERATURE, POWER ETC. 23

24. OTMB Base Board The old TMB Base Boards are migrating to ME4/2, along with the old TMB mezzanine cards – A batch of 87 OTMB Base Boards will be built to serve ME1/1 Only minor modifications in the 2013 version of the OTMB Base Board, fully backward compatible 24

25. OTMB Base Board v2013 A few minor changes for OTMB Base Boards in 2013: – Directly based on original 2005 design – A new front panel to make room for optical links  Need to reshuffle diagnostic LEDs – All original copper connectors stay for full backward compatibility Modification for the 2013 version specifically targets improvement in power distribution and increasing the operational safety margin for the 3.3V rail – Wider tracks to carry increased OTMB current & reduce voltage drop – Micrel regulator to supply OTMB base board circuits with power from the underutilized 5.0V rail  A thoroughly tested regulator, also used on OTMB and DCFEB – Full backwards compatibility is maintained The 2013 version of the OTMB Base Board remains fully backward compatible with old TMB Base Boards 25

26. Power Distribution Optimization Bring in the underutilized 5.0V primary power rail to reduce the load on the 3.3V line 26 TMB v2013 TMB v2005 – Only small changes are required at the base board level – No changes at mezzanine level

27. OTMB Temperature 27 The OTMB with new Mez has been monitored for several months in various operating conditions – FPGA junction temperature range: 56 C – 60 C  Well under the 85 C normal operating limit from Xilinx  DCS is used at Pt.5 to monitor the temperature regularly  FPGA has automatic shutoff at 150 C, before critical damage point is reached One of the test crate locations is in Bat. 904: – Similar to actual crates, but very open in front  Forced air flow is not contained  Poor cooling  These temperature measurements are conservative

28. OTMB MEZZANINE PRODUCTION & COMMISSIONING TESTS 28

29. 29 TAMU Production Test Stand – Standard P. Crate – TMB w/new Mez – Snap12 fiber linked to transmitter board on the bench – Custom loopback boards for DMB and ALCT/RPC tests  Special firmware – Tests controlled via VME and CCB w/customized test suite  Big software effort  nearly complete

30. OTMB Mezzanine Production Testing Four main test groups: – Crate communication tests with loopback boards:  CCB, DMB, ALCT and RPC path tests  Based on customized firmware – Fiber link tests for Snap12 inputs  A bench-mounted Mez prototype board transmits PRBS data on all fiber links to a production test board in the crate  Based on customized firmware – CFEB emulator board (5-cable support)  A dedicated board developed for testing OTMB copper cable inputs  Based on customized firmware – VME, MPC and base board chip tests  Custom software and a GUI utilizing the EmuLib online software tools & normal TMB firmware 30 DMB Loopback Board DMB Loopback Board RAT Loopback Board RAT Loopback Board

31. Software Testing Suite Automatic software testing procedure for the VME tests 31 Extended individual pin test report (raw format) Extended individual pin test report (raw format) Main VME test GUI Main VME test GUI – A lot of effort over last year – Still a lot to do, but plan to be ready by mid-July

32. OTMB Production Testing Planning 32 We are ready and in good shape: – Have been preparing for this day for two years  Software, firmware, custom boards, loopback adapters  A major effort by TAMU students, postdocs and engineers – Focus on automation, formalizing procedures and streamlining logging – Resources and manpower are aligned, timing is critical  Testers are students, it is of utmost importance that a bulk of the OTMB testing happens over the summer

33. Production & Commissioning Plans We will build 87 mezzanine boards – Includes spare boards & test stand needs – Will go for quotes & place orders after PRR Break mezzanine production into 2 stages – Initial delivery of 6-8 production boards for testing  Should take a few days to give the OK to continue – Followed by the remainder of the production Production testing plans & timeline – We can test 15 boards per week  First 20 boards (including initial 6-8) ready in ~1 week  All testing probably completed within ~5 weeks  Plus a few extra weeks to handle any serious problem boards Will ship boxes of tested boards about every 2 weeks – Approximately 10 mezzanine boards per box 33

34. In Conclusion OTMB mezzanine prototypes have undergone two years of intensive testing at several test sites Full functionality has been proven on the ME1/1 chamber at Bat. 904 and elsewhere We have a production test plan and test stand equipment in place The OTMB mezzanine is ready for production 34

35. Extras **** EXTRA SLIDES **** 35

36. Emulator vs.TMB: More Details The comparison procedure included the identity check in data and emulator – of the number of reconstructed ALCTs, CLCTs and LCTs – and of all of their properties Plots here illustrate comparison of the reconstructed stub quality – CLCT quality is # of layers with hits in a pattern – LCT quality values >= 11 reflect the straightness of a pattern (straight tracks with high pT receive higher quality rank) The agreement is perfect in all the numbers 36

37. Cosmic Muon Crossing ME1/1 A & B Boundary 37

38. Cosmic Muon Crossing ME1/1 A & B Boundary 38

39. Other Views of ME1/1 at B904 39

40. Latest TMB Mezzanine: Towards Production Very similar to the last prototype – Removed unneeded test components – Relocated a few parts  Allows the board to be ~0.6” smaller – Other minor changes  Improved clock routing  Made larger mounting holes Trying a larger FPGA this time – Same package, but 20% more logic available – Draws more power Four boards were made – All boards pass all tests – One in use at B.904 Few minor things to fix for final board – Program address signal conflict: solution identified  Also settled on programming with slave-mode clock – One LED on top-side: move to bottom side for visibility – Adding more test points for diagnostic capability 40

41. OTMB Mezzanine Production Testing Equipment & requirements for production tests – TMB Mezzanine test stand with full capability at TAMU  Fiber link tester for Snap12 link inputs  Uses a Mez prototype board to transmit PRBS data to production boards  CFEB emulator board with support for 5 cables  Crate tests with loopback boards (CCB, DMB, ALCT and RPC path tests)  VME, MPC and base board chip tests performed with standard EmuLib tools – Requires both custom and standard firmware to test full functionality  A warm-up period is allotted after downloading to ensure realistic conditions – Every board will require some mechanical assembly at TAMU  Screw-on components (stiffener frame & snap12) and FPGA heat sink Testing software and automation – Developing a custom GUI to control tests and log all results  Vadim and students are doing this work – Good progress on GUI development so far  Still working on logging functions and error reporting & analysis 41

42. 42 OTMB+BB-v2013OTMB+BB-v2005 Primary Power Rails3.3V5.0V3.3V5.0V FPGA logic core2.6A FPGA I/O + MGTs, Snap12 & PROM 3.1A RAT1.2A TMB1.3A1.0A0.3A TOTAL:5.72.57.90.3