1. US ATLAS SCT ROD FDR 20 August 2002 The University of Wisconsin, Madison ATLAS Silicon ROD Testing

2. The ROD and all of its functionality has been tested against the Requirements Document and all system considerations. Introduction

3. The ROD VME Bus Interface circuit operates as required by the ROD Specification and with respect to the Crate Interface Document. List of ROD VME functions that have been tested successfully: Geographical addressing R/W access to: Program Reset Manager FPGA, FPGA Configuration Flash, Master DSP HPI, Master DSP Boot Flash, All devices on the ROD Bus, BOC Bus D32 Block Transfer, Read and Write access (2.2MBytes/S Reads) ROD Interfaces: VME Bus

4. The ROD VME Bus Interface circuit operates correctly when used with the National Instrument (NI) VXI master board. The ROD interface to the Concurrent Technologies Single Board Computer (SBC) is not as stable (from John Hill, Cambridge). Errors can happen during block writes to the Master DSP SDRAM. The SBC handles write access to the VME Slave differently than the NI master and changes are required to the PRM FPGA to correct for the differences. No other problems have been reported The ROD is an A32/D32 device, and supports D32 block transfers from the host ROD Interfaces: VME Bus The oscilloscope plot to the right shows the timing of LACK_n during a D32 block transfer read from a slave DSP.

5. Each ROD must support the BOC bus as defined in ROD-BOC interface document for configuration of its associated optical-card. ROD Interfaces: BOC Electronics Board Test Results: 1.Using a partially loaded BOC Electronics Board, the ROD performed read and write operations to the BOC Control Registers. All access to these registers was successful. 2.The pre-production ROD used at Cambridge has been controlling BOC0 with out failures since it was delivered in January 2002. Each ROD must receive the ATLAS BC clock from its associated optical-card. Test Results : 1.Using an oscilloscope and a frequency counter, the clock path from the BOC to the ROD was monitored. 2.The ROD used at Cambridge has been running from the BOC Clock input since the first system integration test in June 2001.

6. TTC Bus Distribution Each ROD must receive the following TTC related signals from the backplane on the TTC Bus: ROD Interfaces: TIM Electronics Board Test Results : L1A => TTC(0) In This test was set up using a pulse generator connected to the TTC(0) input on the back plane. The pulse width was set to about 25ns, and the trigger was manual. The command was generated by the ROD Controller FPGA and transmitted to the FE with a latency of 2 clocks. 1. L1A 2. BCR 3. ECR 4. Serial ID Data (24 bits of L1ID and 12 bits of BCID) 5. Trigger Type (8 bits of L1 trigger type) 6. Cal Strobe

7. ROD Interfaces: TIM Electronics Board Test Results : This test was set up using a pulse generator connected to the TTC(1) and the TTC(2) inputs on the back plane. The pulse width was set to about 25ns, and the trigger was manual. The commands were generated by the ROD Controller FPGA and transmitted to the FE with a latency of 2 clocks. ECR => TTC(1) InBCR => TTC(2) In

8. ROD Interfaces: TIM Electronics Board Test Results : Cal Strobe => TTC(3) In Serial ID Data => TTC(5) In The Calibration Strobe Circuit in the Rod Controller has a bug. The first Cal Strobe from the TIM will produce the correct result. Following the first strobe, the output will alternate between the correct pattern and an incorrect pattern. The problem can be fixed with minor changes to the VHDL. No hardware changes are required. The Serial ID Data circuit in the ROD Controller is able to deserialize the Event ID data from the TIM within the specification defined by the TTC. The data can be seen in the Event Header, or viewed in special diagnostic registers in the EFB when an Event is sent through the ROD. Trigger Type => TTC(4) In The Trigger Type Data circuit in the ROD Controller is able to deserialize the Trigger Type data from the TIM within the specification defined by the TTC. The data can be seen in the Event Header when an Event is sent through the ROD. The results to support that the ROD processes the Trigger Type and Serial ID Data correctly is demonstrated in the User Evaluation Tests, and all of the DSP Trap Data. The ROD must receive and process this data correctly or errors will be seen in the output events.

9. The ROD must be capable of writing properly formatted event fragments to an ATLAS standard S-Link ROD Interfaces: Read Out Links Event Fragment Format 1 0xB0F00000 2 0xEE1234EE 3 0x00000009 (# Header Words) 4 Format Version Number 5 Source Identifier 6 L1 ID 7 BC ID 8 L1 TT 9 DET 10 First Event Data Word 11 *** 12 *** 13 *** 14 *** 15 *** 16 *** 17 *** 18 *** 19 *** 20 *** 21 *** 22 Last Event Data Word 23 Error count 24 Error flags 25 0x00000002 (# status words) 26 nData (# of data words) 27 0x00000001 28 0xE0F00000 S-Link Data 1 B0F00000 2 EE1234EE 3 00000009 4 00000001 5 00000000 6 00000001 7 00000000 8 00000000 9 00000000 10 21008E80 11 *** 12 213B8DC0 13 97D0A0C0 14 21408770 15 *** 16 21798461 17 85108D00 18 8D708FA0 19 96709DF0 20 A3B1A420 21 AD80217A 22 A5204000 23 00000000 24 00000000 25 00000002 26 000000FD 27 00000001 28 E0F00000 Sim Data 1 B0F00000 2 EEEEEEEE 3 00000009 4 00000001 5 00000000 6 00000001 7 00000000 8 00000000 9 00000000 10 21008E80 11 *** 12 213B8DC0 13 97D0A0C0 14 21408770 15 *** 16 21798461 17 85108D00 18 8D708FA0 19 96709DF0 20 A3B1A420 21 AD80217A 22 A5204000 23 00000000 24 00000000 25 00000002 26 000000FD 27 00000001 28 E0F00000 The discrepancies in these results are as follows: 1.The Start of Event Word needs to be changed in the simulation data. The S-Link value matches the value defined by the DAQ group.

10. ROD Interfaces: Read Out Links The order in which the FE links appear in the output event record must not change from event to event The ROD does not change the order in which the FE links appear in the event record from event to event. The Token-Passing architecture is designed to start with Link 0 and end with Link 95. The data is played from the Formatter Link Output FIFOs in low to high order, and remains in this order through all of the functional blocks of the ROD. The link order can be seen in a search for Link Headers.

11. ROD Interfaces: Read Out Links The ROD must transmit output event records in the order in which they occurred The ROD transmits Events in the order in which they occur. The Event order can be seen in a search for the L1ID field in the Event Headers.

12. The ROD must recognize protocol errors in the input data streams and flag them in the output streams. When the inmems are loaded with out masking the Formatters, Header Errors are detected in the Formatters, and the Link Formatters go into Raw Data Mode Receipt of Data

13. The current reading were taken from the front panel current meter of the Weiner Crate. The values are accurate to about 0.5 Amps. ROD Voltage Requirements: The ROD receives all of its DC power from the VME Crate Power Supply. +5 VDC to power the VME Interface circuit and the DC to DC Converter used to supply the FPGA and DSP core voltage. The typical current required for 1 ROD that is operating in 100kHz data taking mode and has all DSPs booted and running is 4.5 +/-.5 amps. The core voltage DC to DC Converter efficiency (PT7705 - 1.8V@18A) => 79% +3.3VDC to power all of the FPGA and DSP I/O circuits, and all of the remaining logic and memory parts. The typical current required for 1 ROD that is operating in 100kHz data taking mode and has all DSPs booted and running is 2.5 +/-.5 amps. +12VDC is routed to the BOC. The estimated current requirement is less than 100mA per BOC. ROD Power Requirements: The typical power required by 1 ROD that is set up for data taking and has all DSPs booted and running is 30.8 +/- 4.2 Watts. The typical power required by 16 RODs that are set up for data taking and have all DSPs booted and running is 493 +/- 66 Watts ROD Power Requirements

14. ROD User Evaluation: 100kHz Event Rate The ROD was set into the internal test mode that uses the Input FIFOs and the internal TIM FIFO to retransmit a 10 event simulation file through the ROD data path every 100us, 65536 times (655360 events per test run). A HP 16500C Logic Analyzer was connected to the S-Link, sampling state values on the rising edge of S-Link CLK, and set to trigger on the first falling edge of S-Link WEN. The Slave DSPs were set to trap a specific event (Atlas Specific) every 256 times it was detected in the Routers. DSP0 Trap: Atlas Trigger #4, prescale = 256 DSP2 Trap: Atlas Trigger #9, prescale = 256 DSP3 Trap: Atlas Trigger #5, prescale = 256 Back pressure was not allowed. (Mentioned because it is a variable in the eventTrapSetup primitive.) The average event size for this simulation file is 200 words. Test Setup:

15. ROD User Evaluation: 100kHz Event Rate: Test Results At the end of each test run, the FE Trigger Counter and the FE Occupancy Counters were always equal to 0. This means that the ROD did not miss any triggers, and detected a link trailer for every event. During the test run, the FE Occupancy Counters were read at a 550kHz rate. The graph below shows the Formatter Event Occupancy for the test run.

16. ROD User Evaluation: 100kHz Event Rate - Results S-Link Data was trapped on the Logic Analyzer, and in the Slave DSPs that were set up for trapping. Data from DSP 0 Event Trap The discrepancies in these results are as follows: 1.The 15 in the beginning of fragment word is a code placed into the event header by the router and used by the DSP during data processing. 2.The Start of Event Word needs to be changed in the simulation data. The S-Link value matches the value defined by the DAQ group. DSP Data SIM Data

17. ROD User Evaluation: 100kHz Event Rate - Results Internal IDRAM Data from DSP 0 Event Trap This data represents the processing of the trapped data in the DSP. Burst Buffer Occ Start of Status

18. ROD User Evaluation: 100kHz Event Rate - Results Data from S-Link: Logic Analyzer Trap The discrepancies in these results are as follows: 1.The Start of Event Word needs to be changed in the simulation data. The S-Link value matches the value defined by the DAQ group. 2.Occasionally, a bit error can be found in the S-Link data when compared to the Simulation Data. When the data from the logic analyzer is examined, the results show that where these bit errors occur, the data signal starts at the correct value, changes state in the middle of a cycle, then returns to the correct state towards the end of a cycle. The duration of the bit error is always a multiple of the sampling time of the Logic analyzer when sampled with the internal clock. I think that this requires more examination, but I don’t think that it is anything serious. SIM Data SLink Data

19. ROD User Evaluation: 100kHz Event Rate - Results Data from S-Link: The remainder of the Event Data from 1 event of the 100kHz test SIM Data SLink Data

20. ROD User Evaluation: 3 Point Gain Curve This test shows that the ROD can configure and read data from a module, issue L1 triggers, receive and process the event data, and perform histograms on the data trapped in the DSPs. The ROD was setup to operate with L1 triggers generated on SP0, with internal register Event ID data sent to the EFB for header generation. The 2 chip module was used to perform the 3 point gain curve. Charge was injected for 1.5fc, 2fc, and 2.5fc Every 4th strip was pulsed in the chips. There were 50 pulses at each charge. Four chips histograms have been plotted: 1. Chip 0, Channel 0 2. Chip 0, Channel 4 3. Chip 1, Channel 4 4. Chip 1, Channel 12 Test Setup:

21. ROD User Evaluation: Chip 0 Chan 0 Histograms

22. ROD User Evaluation: Chip 0 Chan 4 Histograms

23. ROD User Evaluation: Chip 1 Chan 4 Histograms

24. ROD User Evaluation: Chip 1 Chan 12 Histograms

25. ROD User Evaluation: Chip 1 Ch 12 : 3 Point Gain

26. Excerpts from a Message from Peter Philips: In general the S curves look OK. Calibration bus 0 is strobed. Sometimes there are small dips from 1.0 occupancy whilst still nominally at the top plateau: these seem to occur at a threshold in the vicinity of the noise occupancy offset, not an unknown phenomenon. There are also occasionally a few hits in the tail, but with statistics of only 50 events one might expect this. Since these effects do not appear across a large number of channels I think we can assume that the data is OK, and that the histogramming has been performed correctly. Three Point Gain analysis http://hepwww.rl.ac.uk/Atlas-SCT/ROD/RCPlot.ps Gain of strobed channels around 53mV/fC Noise of strobed channels around 650 ENC (These are values read from the plots.) Probably about right for a hybrid with this chipset. A lot of channels return higher noise values, but I think this is being driven by the dips from the plateau (causing the fit to fail). I don't have time to rewrite the analysis to accommodate this - not right now, anyway. Although I would have liked to have seen this data for a fully populated module - and ideally with a loop over the four calibration lines - I guess this will have to do. I'm not aware of any show stoppers, so let's have some more RODs. It's taken a lot of hard work to get this far: maybe things will be easier from now on.... ROD User Evaluation

27. Slave DSP 1: Clock instability and SDRAM errors The SDRAM and SBSRAM clocks used by SDSP1 are 180 degrees out of phase with the same clocks used by SDSP0, SDSP2, and SDSP3. The reason for this, I believe, is is two fold. The first issue is that about 22 of the 36 system clocks used on the ROD route directly beneath DSP1, with at least 10 different clock traces in the location of the DSP PLL circuit. The second issue is a layer stack up error cause jointly by LBL and the Fabrication vendor that cause signal layer to be placed adjacent to one another with out a power or ground plane for separation. We specified a layer stack up on the Fabrication drawing, but had the layer numbers different on the artwork. The result of this error is that noise from the clock traces is coupling into the PLL circuit. The clock instability is most likely responsible for all SDRAM errors. The next revision of the ROD will have the correct layer stack, and I believe that all of the problems with SDSP1 will disappear. ROD Existing Issues: SDSP1 The rising edge of the 80MHz clock should line up in phase with the rising edge of the 40MHz clock.

28. Router: Trapping Algorithm Instability The Router underwent a major reconstruction to facilitate the functions required for the user evaluation. The added complexity of the algorithm has increased the utilization of slices in the Router to about 80%, and the VHDL to FPGA synthesis is very erratic. As if the synthesizer (Synopsys - FPGA Express) is not building what we think it’s building. In some modes, the DMA to the DSP will result in the end of event marker getting shifted by 16 words. None of these problems have been explored exhaustively, and some are easy to work around for the present. We feel that we need to return to the DSP trapping algorithm, simplify it, and write and synthesize new VHDL to solve these issues ROD Existing Issues: Router

29. Formatter: Module Readout on a single link After the start of user evaluation, it was discovered that for certain calibration data sets, the Link FIFOs in the Formatter would not be able to hold all of the data from the module if only 1 link were active. We are proposing to re-design the Formatter to allow dynamic allocation of Block RAMs if the Static Mask Bit for a link is set. In this design, adjacent links combine FIFOs to create a 512W x 32 FIFO. The work for this is on hold as it is a lower priority then other issues. ROD Existing Issues: Others Access using the SBC Because the SBC controls the VME bus differently than the NI system, we will need to spend some time characterizing any issues. The problems we have encountered recently with the Pixel Turbo PLL has made it necessary to make mention of any potential issues we may have.

30. We believe that the ROD is working very well, and that we need to build a new version to fix the issues that remain. As the number of users increases, we will need more RODs assembled to support the requests. Summary