1 ATLAS Lehman Review, Silicon ROD Douglas Ferguson, Richard Jared, John Joseph and Lukas Tomasek Wisconsin May 21 to May 23, 2003

2 ATLAS Lehman Review, Silicon ROD Material Covered ROD Schedule and High Level Summary Scope Major Events ROD Overview Current Status

3 ATLAS Lehman Review, Silicon ROD ROD Schedule and High Level Summary

4 ATLAS Lehman Review, Silicon ROD ROD Schedule and High Level Summary

5 ATLAS Lehman Review, Silicon ROD High Level Summary 9% of SCT production RODs complete Authorized Funds Paid to Date/Committed WTotals($)M&S($)Labor($)Travel($)Totals($)M&S($)Labor($)Travel($ 1.1.3RODs Milestones ETC02 Actual Projected Pixel ROD PRR 6/03 9/ SCT ROD PRR 4/03 6/ SCT Prod Complete 10/03 11/ Pixel Prod Complete 3/04 3/ Pur ROD SCT Crates 3/03 9/02 Two crates have been ordered Installation complete 2/05 2/05

6 ATLAS Lehman Review, Silicon ROD ROD Comparison of Costs Small call on contingency The ETC03 had a +48K difference from ETC02 (pixel included). This is in the noise. The pixel ROD have been added to the baseline.

7 ATLAS Lehman Review, Silicon ROD Scope Baseline Scope ATLAS ROD Pixel test beam support hardware Design to production models SCT and pixel RODs Development of ROD test stand Fabrication of SCT RODs Fabrication of pixel RODs Purchase of SCT and pixel ROD crates Management Contingency allocated

8 ATLAS Lehman Review, Silicon ROD Major Events 1. Fabrication and debugging of 9 RevC production model RODs has been performed. 2. Successful testing of data flow through the RevC production model ROD at Cambridge. 3. The test stand software has been upgraded to support pixel user evaluation. 4. DSP software has been coded to support user evaluation of SCT and pixel user evaluation. 5. The ROD passed the US ATLAS FDR. 6. SCT user evaluation has had much effort in the UK. They have configured modules, readout events and histogrammed the the data. 7. The ATLAS PRR was held on May 13, The chairman Philippe Farthouat has said that we passed (report on May 23). 8. Pixel user evaluation has started. Modules have been configured. Testing of events readout is currently being done. 9. The pixels will require a revised ROD (no changes to VHDL of C code. FPGAs will be changed to provide more derandomizing space and the DSPs will be changed to provide more memory space.

9 ATLAS Lehman Review, Silicon ROD ROD Overview

10 ATLAS Lehman Review, Silicon ROD

11 ROD Implementation: FPGA Front End / DSP Back End 96 Data RCVR ROD Controller: Operation, Command, Trigger 43 FPGA Formatter and EFB 96 Back of Crate Card / Front End Electronics Input Memory & XCVR Control VME Slave Interface Slink on BOC VME Bus 32 Cont / Addr Data BusControl, Status, Mask Trig Data Timing Interface Module BOC Setup Bus Serial Output Link Serial Input Link Xon/Xoff ROD Busy Dec. Trig Count Control & Status for Block Xfer/ DMA / S-Link DSP Event Trapping, Histograms (Four DSP Chips) DMA 1 Router FPGA S-Link & DSP DMA Host Port Interface Data Valid S-link Data DMA 4 Router Halt Output DMA Cont. Event Data/Tirg Type R/W Bus R/W Bus R/W Bus... Busy Program Reset Manager Token and Dynamic Mask S-link Data Boot FPGAs

12 ATLAS Lehman Review, Silicon ROD controller FPGA boot FPGA program FLASHpower suppliesslave DSPs & memories router FPGAs event builder FPGA debug & derandomizing memories ROD bus buffers data formatter FPGAs real time data path master DSP & memory

13 ATLAS Lehman Review, Silicon ROD Current Status (ROD hardware) The data and control path in the ROD have been coded and tested. The SCT VHDL is in maintenance mode (changes implemented as needs are found). C code is mostly written for module calibration. EFB FPGA VHDL in maintenance mode Router FPGA VHDL in maintenance mode Formatter FPGA SCT VHDL in maintenance mode The formatter for the pixel 40MHz links has been written. The formatter VHDL for the pixel 80Mhz links has to be written. Controller FPGA VHDL in maintenance mode Program Reset Manager VHDL in maintenance mode

14 ATLAS Lehman Review, Silicon ROD Current Status (ROD software ) The infrastructure software is written and tested. Many primitives have been written and tested. The code needed for module calibration has been written. The code is being upgraded as the user evaluation needs are defined from use. The histogramming code will need (for speed) to be executed in machine language once the full functionality is tested by the users. Queuing of events in the FIFOs will also be needed to improve the speed. Future needs: histogram fitting, occupancy measurement, error detection, counting & processing.

15 ATLAS Lehman Review, Silicon ROD Current Status (Test Stand Software ) The test stand software is complete and is in maintenance mode. Updates are made as needed. Concerns: The upgrade to the pixel ROD has some risk because the FPGA and DSP pins will change and the printed circuit card will be rerouted. Three boards will be initially fabricated to reduce risk. The time to calibrate the SCT modules is too long. The C code will have to be changed to improve speed.

16 ATLAS Lehman Review, Silicon ROD US ATLAS FDR (short version) Review Committee Report for the SCT & Pixel ROD US-ATLAS FDR of 20-Aug-2002 Introduction: A Final Design Review of the Read Out Driver (ROD) intended for use by the ATLAS SCT and Pixel subsystems was held at LBNL on 20-Aug The primary intent of this review was to evaluate the present status of the ROD design for use by the SCT and Pixels with a goal of approving the build of 9 pre-production RODs so that further user evaluation can be obtained prior to a Production Readiness Review (PRR) as early as possible. While the goal is for the ROD design to be compatible for both SCT and Pixels, it is recognized that there has been as yet no user evaluation of the design with Pixel readout. Given the schedule constraints of the SCT and the overall cost constraints of the project, the plan is to evaluate the appropriateness of the present ROD design for both SCT and Pixels and decide on the pre-production build knowing that further design modifications may be needed for Pixels after appropriate user evaluation is completed with that readout. The review committee included: Christopher Bebek – LBNL Kevin Einsweiler – LBNL Alex Grillo – UCSC Henrik von der Lippe – LBNL Abe Seiden - UCSC

17 ATLAS Lehman Review, Silicon ROD US ATLAS FDR (continued) Recommendations: The committee was impressed with the amount of work accomplished since the last review. Basic functionality of the SCT ROD has been shown along with the first examples of on-board histogramming. While more user evaluation is still needed to be sure that the ROD meets all requirements for SCT and Pixels, this is at least a good start. The major obstacle to such evaluation is software for a DAQ system and focused attention of users running the ROD through all its necessary operations. User evaluation of the Pixel version is awaiting a Pixel module to readout. The committee makes the following recommendations to the ROD design group: 1.It is clear that the ROD development is ready to proceed to implement all identified design changes into a new PCB layout. Also, more ROD units will enhance the chances to obtain more, much needed, user evaluation. Therefore, the committee strongly recommends that the design group proceed as quickly as possible to complete layout of the next version of the ROD PCB and fabricate sufficient boards to eventually load 9 boards. Given the changes to be made to the layout, we recommend that initially only two boards be loaded and tested. The committee at first thought that only one should initially be loaded for testing but we have accepted the advice of the ROD design team that the initial debug of two boards facilitates diagnosis of problems by having two samples to compare. Once basic functionality is demonstrated, the remaining 7 boards can be loaded. Note: 3 then 7 boards have been fabricated. These boards have been used in user evaluation. 2.The alternate footprint (BGA456 in addition to BGA676) for the Formatter on pixel boards should not be executed. We believe that this creates too big a risk in the layout for an option that may not be needed or used. If the planned Formatter FPGA proves to have insufficient performance for the Pixel readout, one of the higher performing FPGAs with the same footprint can be utilized (405/EM or 600/E). The added cost in that case is a worthwhile risk instead of the risk to the layout now. Note: The change was not executed.

18 ATLAS Lehman Review, Silicon ROD US ATLAS FDR (continued) 3.The allowable power supply margins of the FPGAs and DSPs should be analyzed vs. the regulation spec of the DC-DC converter over the temperature and supply range expected for the board. If it is not convincing that a single supply will work reliably for long-term use, the power plane should be split and two DC-DC converters should be employed. While splitting the power plane does imply some risk, care at this time to avoid errors may be a better risk than possible reliability problems over the lifetime of the ROD. Note: An analysis was performed and no change was made. 4.Attempt to optimize the FPGA VHDL code as planned to reduce the utilization. At the time of the PRR, the utilization must be reviewed again. If it is still uncomfortably high to accommodate possible needed enhancements during commissioning, a decision will have to be made then to upgrade to the high cost FPGA. Note: FPGA utilization is now Formater 70%, EFB 56% Router 67%, Controller 70% and PRM 50%. 5.A discussion should be held with the person now re-designing the S-link board for ATLAS reviewing the ROD’s S-link interface vs. those of the S-link to confirm that there are no compatibility issues. Note: The S-link was reviewed with CERN and has been tested as part of the user evaluation. 6.There seem to be legitimate concerns about how the detailed phase of the data streams from the on- detector electronics will be monitored and timed relative to the TIM-generated crossing clock. Since the BOC is the module responsible for this synchronization, it is suggested that this is the correct place to implement monitoring of the timing of the “raw” data streams. The SCT/Pixel off-detector design group should consider how this may best be implemented in the final system. However, we suggest that the pre- production RODs should simply provide a reasonable number of header-based test point, close to the FPGAs, for logic-analyzer analysis, but should not attempt to provide multiplexed front-panel access to the full set of input data streams. Note: The BOC supports passing the raw data to the ROD. A multiplexer has been added to the Formater FPGA to allow selection of one stream at a time for monitoring on a front panel connector. Module timing is performed by varying the BOC timing and observing the results with the ROD.

19 ATLAS Lehman Review, Silicon ROD US ATLAS FDR (continued) 7.Since the current ROD design does not meet the requirement to readout events via VME at 1kHz with a full 16 RODs/crate and it would require a major redesign to try to accomplish this, but it appears that one ROD/crate would be sufficient for any planned ROD use, the committee recommends that this requirement be changed to “VME readout at 1kHz for 1 ROD/crate.” Note: The requirement was changed to 100 Hz. Current test indicate that the ROD can be read out at full luminosity at 300 Hz. 8.The SCT community is strongly urged to devote more resources to write the necessary DAQ software so that more detailed user evaluation can be performed. To expedite this work, it is recommended that one of the existing RODs be sent immediately to Oxford to allow that group to begin learning to use the ROD. Note: The SCT community now has a large effort devoted to user evaluation and preparation for the ROD use at the macro assembly sites. The committee wishes to thank the ROD design group for their time preparing for this review and presenting a clear status of the project. Those members of committee who are members of the ATLAS collaboration would also like to thank Christopher Bebek and Henrik von der Lippe for their contribution to ATLAS by spending the time to review this complex development.