1. Level-1 Calorimeter Trigger Readout Driver FDR/PRR 15 th August 2006 Introduction to the ROD Norman Gee 15-Aug-2006 Norman Gee

2. 15 Aug 2006Norman Gee - L1Calo ROD Introduction2 Contents Introduction –Overview; Readout Scheme; Crate Structure; Physical Implementation Input and Output Data –Characteristics of the data; Zero suppression / compression; Example data; Requirements summary Input data handling –Readout process; Timeout; Rod Busy; TrigType Timeout; ECReset Output Data handling –RoI Data; Data routing to S-Link; Data Reduction; Replay mode; Monitoring; generation of S-Link packets VME Interface, Canbus

3. 15 Aug 2006Norman Gee - L1Calo ROD Introduction3 DAQ RODs To ROS e/ ,  /had Clusters (CP) 0.2 x 0.2 Jet /  E T (JEP) 0.1 x 0.1 Pre- Processor (PPr) Analogue tower sums 0.1 x 0.1 (~7200) RoI RODs To RoIB To CTP 8 PPr crates 4 CP crates 2 JEP crates 2 ROD crates L1Calo Overview DAQ RoIs

4. 15 Aug 2006Norman Gee - L1Calo ROD Introduction4 Processing Module Readout Scheme Write Read Scrolling Memory

5. 15 Aug 2006Norman Gee - L1Calo ROD Introduction5

6. 15 Aug 2006Norman Gee - L1Calo ROD Introduction6 Physical Implementation 9U * 400 mm Main module 9U * 160mm Rear module

7. 15 Aug 2006Norman Gee - L1Calo ROD Introduction7 Internal Structure 18 G-Links –handled in groups of 4 Five Input FPGAs –data input, compression, formatting Switch FPGA –Data routing to 4 S-Links Monitoring FPGA –Simple event monitoring Overall module functions –VME FPGA, ACE, CPLD, TTC, –CANbus, power (details in Viraj’s talk)

8. 15 Aug 2006Norman Gee - L1Calo ROD Introduction8 Data Characteristics Preprocessor data has two components per channel (64 ch per module): –Raw FADC output – typically ~five 11-bit values – to cover a calo pulse… –…and a calibrated ET value in GeV/c (“LUT data”). –Most of the time, there is no calo pulse - so FADC output is pedestal + noise Can reduce data volume by bit-field compression, zero-suppress LUT data. Cannot discard FADC data (at least at low L1A rate) – needed to check BCID. Most other modules also produce two components, but very different from above: –Integer values in GeV entering or leaving boards; –Threshold hits – single bits to show if a digital value exceeds a threshold. Zero suppression expected to be effective - calo cell value rarely exceeds 1 GeV All information relating to one bunch crossing is read out in a single clock cycle. All the data reduction is designed to be lossless.

9. 15 Aug 2006Norman Gee - L1Calo ROD Introduction9 Example G-Link Data format (JEM)

10. 15 Aug 2006Norman Gee - L1Calo ROD Introduction10 Example S-Link Data format Sub-Block Header Payload created from G-Link data Sub-block Trailer (suppressed if there were no errors) Note: there is also a “neutral” format. It copies raw data direct from G-Link to S-Link without interpretation. Very important for debugging.

11. 15 Aug 2006Norman Gee - L1Calo ROD Introduction11 Summary of Requirements Receive 16- or 20-bit data from up to 18 G-Links. Check BCNs & parity. Optionally discard some timeslices from different sections of source modules. Compress or zero suppress data from each channel, as required. Write formatted data to the FIFO buffers (internal to the FPGAs) Receive CLOCK, BCR, ECR, event number and event type from TTC. Select data to be transferred to the ROS based on trigger type. Build and write ATLAS-standard event fragments to up to four S-Links. Spy on events (FPGA or VME), optionally select according to event type. Generate BUSY signal to the Central Trigger Processor. Process complete events at up to 100 KHz sustained rate. For RoI data only, limit the number of RoIs to a programmable maximum. Support CANBus board temperature and voltage monitoring.

12. 15 Aug 2006Norman Gee - L1Calo ROD Introduction12 Description of readout process At Source Module –L1A arrives, n (e.g. 5) timeslices copied into readout FIFOS in G-Link data format. –DAV asserted. Data for n slices sent, with parity after each. DAV removed At the ROD –L1A arrives, BCN stored in FIFOs in Input & Switch FPGAs –Input FPGA: G-Link data received, reformatted, zero-suppressed, stored in data FIFO. Checks on G-Link status, Longitudinal parity, BCN -> error bits in substatus word –Switch FPGA: receives event type. –Switch FPGA: waits for all Input FPGAs to complete; creates S-Link header; copies payload to S-Link; creates S-Link trailer;

13. 15 Aug 2006Norman Gee - L1Calo ROD Introduction13 Input data timeout Happens if a source module fails to provide expected data. Input timeout is detected by Switch FPGA: –Start a timer when the first Input FPGA has processed data available. –Declare Input timeout if timer expires before all expected Inpout FPGA channels have processed data available. When building the S-Link packet, timed-out Input channels are instructed to provide a sub-header and a sub-status word with timeout flag set. –Late data is not used even if available by then. If the missing data does eventually arrive (unlikely), it will be put into the following event –but will have wrong Bunch Number. Any surplus data is erased by the ECR mechanism.

14. 15 Aug 2006Norman Gee - L1Calo ROD Introduction14 Generation of ROD- Busy Four S-Links - each can assert LFF (“Xoff”) if destination is too slow. When LFF, Switch stops working. Input FPGA Data FIFOs start to fill up. When depth limit is reached, ROD-BUSY is generated. –Programmable depth limits on three FIFOs – event data, event management, BCN FIFOs. –Must be set to handle worst case – 8 events separated by 5 bc each. We have two sets of counters: –one set to monitor the fraction of time each S-Link is asserting LFF. –one set as an 8-bin histogram of Input FPGA data FIFO depth.

15. 15 Aug 2006Norman Gee - L1Calo ROD Introduction15 Trigger-type timeout Trigger type is sent by TTC broadcast so has no guaranteed arrival time –depends on other TTC channel B traffic. Trigger type is needed to populate ROD header. If ROD waits indefinitely, Input FPGA FIFOs will fill up and ROD BUSY will be generated. –If trigger type doesn’t ever arrive, ROD will block. Trigger-type timeout starts once processed data for an event is ready in the Input FPGAs. After the timeout, a default value will be used. If trigger type arrives very late, it is cleared by ECR

16. 15 Aug 2006Norman Gee - L1Calo ROD Introduction16 Event Counter Reset Sent by CTP at about 1 Hz. Ignored by TTCrx ASIC, but used in ROD to clear ROD-internal 24-bit L1_ID and increment 8-bit ECR Counter. Both together form 32-bit ROD_L1ID field in S-Link header. Also usedin L1Calo RODs to clear any partial events.

17. 15 Aug 2006Norman Gee - L1Calo ROD Introduction17 RoI Data Collected from source modules just like DAQ data –One timeslice only. Sent to separate “RoI-RODS” on separate links. Zero-suppressed as DAQ data. Collected by Switch FPGA and counted as sent to RoIB over S-link Programmable limit to number of RoIs sent. –When this limit is reached, no more RoI payload is sent. Status bit is set. –This guarantees that RoIB is not overwhelmed by wild events in ATLAS Normally only 1-2 RoIs expected per event. –Calibration runs will produce many more – not used in Level-2.

18. 15 Aug 2006Norman Gee - L1Calo ROD Introduction18 Data Routing to S-links – DAQ Data Data from each Input FPGA (4 G-Links) is concatenated into a separate ROD fragment. Data from the last two input FPGAs (nos 3 & 4) are combined into one fragment –G-links 0 - 3 -> S-link 0 –G-Links 4 – 7 -> S-link 1 –G-Links 8-11 -> S-Link 2 –G-Links 12-17 -> S-Link 3 All data from Input FPGAs 3 & 4 (4 or 6 G-Links) is concatenated into S-Link 3.

19. 15 Aug 2006Norman Gee - L1Calo ROD Introduction19 Data Routing to S-links – RoI Data All data from 18 G-Links is concatenated into a single ROD fragment The fragment is duplicated to Slinks 0 and 2. –S-Link 2 has a logic analyser header.

20. 15 Aug 2006Norman Gee - L1Calo ROD Introduction20 Data Reduction Most data (apart from PPM) is subject to zero-suppression. S-Link words are not written if the data values and error bits are all zero. –Typically expect only a few percent of non-zero trigger towers. –For 5% occupancy of trigger towers, achieve factor 5-10 reduction Addressing information is added to identify remaining data: – in ROD header (identifies DAQ and RoI, plus crate family) –in Sub-block header (identifies source crate, type of module, slice number) –in S-Link words (identify data source within module) For the PPM, zero-suppression is not appropriate, so a variable-length bit field approach is used. –Based on subtracting FADC samples to give small numbers, which can be packed in short bit-fields. Expect a factor 2 data reduction. –Prototype scheme is documented and checked. Will be tuned.

21. 15 Aug 2006Norman Gee - L1Calo ROD Introduction21 Replay Mode Replay mode runs from the Input FPGAs. –Disable all G-Links; –Preload FIFOs with required data and put lengths in event management FIFOs. –Generate L1As internally (software) or using TTC system. –Transmitted data is recycled back into the FIFOs so one or several events can be played continuously BCN, Event number and Trigger Type are obtained from fixed registers, not TTC.

22. 15 Aug 2006Norman Gee - L1Calo ROD Introduction22 Event Sampling and Monitoring Event sampling buffers are provided in the Switch FPGA for each S-Link. Sampled events may be accessed from VME or copied to the Monitoring FPGA. Monitoring FPGA code is not yet written. Data will be available for monitoring by code in FPGA-internal PC cores or to PCI daughterboard. Events are eligible for sampling if they satisfy an event-type mask and a prescale factor. –All S-Links are sampled (or not) on the same event. Two sampling modes: –Random: events sampled whenever a buffer is available; –100%: All events sampled. Software LFF is asserted if all buffers are full. This mode will assert ROD-BUSY if the trigger rate is too high. A mechanism is provided to sample the same events in all RODs

23. 15 Aug 2006Norman Gee - L1Calo ROD Introduction23 Generation of S-Link fragments 5 components: –BOF and EOF control words Fixed in Firmware –ATLAS ROD Header Firmware and registers –User Header Registers –Data payload Input FPGAs –Status words Calculated in Switch from Input FPGA status bits.

24. 15 Aug 2006Norman Gee - L1Calo ROD Introduction24 ROD format – User Header Sadly not allowed for in ROD header so must be part of payload Up to 4 words in current firmware. Number of words software controlled. First word contains no of header words including itself. Currently only 1 word is used –Contains slice number corresponding to L1A.

25. 15 Aug 2006Norman Gee - L1Calo ROD Introduction25 VME Interface Characteristics ROD conforms to VME64x. Legal VME addresses will always respond to cycles. –All software uses the same programming model, which knows RW or RO, register length (16 or 32 bits), bitfield positions, etc. Programming model contains registers for all functions of all firmware - some not used in some implementations. Default firmware is loaded by ACE based on module Geographical address –no jumpers needed to configure spare modules Numbering (of links, FPGAs, etc) starts from 0 in documentation like C++. All modules have a read-only module type at their base address. Hardware and firmware revisions are all readable in registers.

26. 15 Aug 2006Norman Gee - L1Calo ROD Introduction26 VME Code fragment ModuleRegister16 { controlPulseReg 1030 PartDependencies { 500 } Attributes { 0x3 CMM.controlPulseReg "Write only“ } State { "" } } CMM.controlPulseReg { 1 {"Reset Module"} 1 {"Reset TTC"} 1 {"Reset RoI G-Link" } 1 {"Reset DAQ G-Link" } 1 {"Reset CAN Controller" } 1 {"Reset I2C Controller" } 1 {"Reset Crate Flash Controller" } 1 {"Reset System Flash Controller" } 1 {"Reset DLL" } 1 {"Clear Parity Errors" } 2 {"Reset FPGAs" Value { 0 {"Nop"} 1 {"Crate FPGA"} 2 {"System FPGA"} 3 {"CrateAndSystem FPGAs"} }} 4 {"Unused" } }

27. 15 Aug 2006Norman Gee - L1Calo ROD Introduction27 Handling Errors xxx sources of error: Errors detected in the upstream module: –e.g. incoming LVDS or backplane parity errors, saturation, overflow. –The module must always produce the correct length data, with flags set to indicate errors. –The ROD will generally report the errors with the affected data. Some common errors are copied to the ROD fragment status words. Errors detected in the ROD –Related to G-Link transmission or protocol failure, or to mismatching events. –Identified in Input FPGA and reported in substatus word. For RoIs, errors are reported only in the ROD fragment status words. No data error or transmission error should cause the ROD to hang or even slow down.