1. Trigger matched firmware Top Level block diagram 4ch 3.2gbps deserializer “Test Assembly 1” Module Latency buffer 1GB DDR2 DRAM a.c.r. 2015-09-01 The GTKRO firmware allows the FPGA on board the GTK-RO board to configure the TDCpix controlled by it and to collects its output data, storing it into the DRAM buffer and retrieving it when required by the trigger received through the TTCrq “Test Assembly 2” Module 1 two way 320MHz ser/des TDCpix configuration link controller Latency buffer 1GB DDR2 DRAM Stefano’s ethernet MAC TM_Data TX module Stefano’s ethernet MAC TM_Data TX module Slow control module main PLL PLL phase controller for TDCpix config. house keeping uC TTCrq interface module Angelo Cotta Ramusino INFN-FE, for GTK_WG mtg, CERN 2015-10-13 GTK readout firmware: trigger matched

2. “Test assembly” module (one of two) in detail: 1) from input links to the latency buffer ½ of 4ch 3.2gbps deserializer module Data Receiver Even frame buffer FIFO Odd frame buffer FIFO Ch 0 Rx Wrapper Module Data Receiver Even frame buffer FIFO Odd frame buffer FIFO Ch1Rx Wrapper Module Even frame buffer FIFO Odd frame buffer FIFO Merger Module Even DPRAM buffer Bucket Sorter Module Even DPRAM buffer DDR2 Interface module Latency buffer 1GB DDR2 DRAM Data from two input serial data channels is processed in pipelined steps (each coarse pipeline corresponding to one 6.4us time frame). At the end of each step data is stored into an intermediate buffer until it reaches the DRAM memory, which is the latency buffer  step1   step2   step3  a.c.r. 2015-09-01 Angelo Cotta Ramusino INFN-FE, for GTK_WG mtg, CERN 2015-10-13

3. “Test assembly” module (one of two) in detail: 2) from the latency buffer to the subdetector PC Trigger matching unit 0 Trigger processor module Data from the DRAM based latency buffer is fetched and processed in parallel by 4 “trigger matching units”. The results are merged into one output packet. When 8 trigger matched packets are present in the “TM Data TX controller” an UDP packet is sent to the subdetector PC via the ethernet mac. a.c.r. 2015-09-01 DDR2 Interface module Latency buffer 1GB DDR2 DRAM Angelo Cotta Ramusino INFN-FE, for GTK_WG mtg, CERN 2015-10-13 TTCrq interface module Stefano’s ethernet MAC TM_Data TX controller Trigger matching unit 1 Trigger matching unit 2 Trigger matching unit 3 Trigger matched packet builder

4. Status: The version (wip20150915) of Trigger matched firmware currently in use has been installed on Sept.15 th. It allows data collection but its operation is still unstable. On the other end: it seem to work reliably on boards (more then one tested) coupled to the single chip assembly in the PC Farm room it has worked reliably when coupled to GTK stations for which the pixel array was disabled (the TDCpix where sending only framing words and no timestamps) it works more reliably when the TDCpix ASICs in the Gigatracker assemblies are configured with configuration files with higher thresholds The three facts above seem to correlate the failure rate to the input data rate, but the precise reason for the correlation is not so straightforward to determine because the receiver modules in the current TM firmware have a filter on input multiplicity and so the maximum amount of accepted timestamp within a frame period is limited to a safe (according to simulations) value. Thus, the cause of the erratic behavior doesn’t seem to be the high input multiplicity alone Designed features of version “wip20150915” (incremental w.r.t. the version described at the Praga meeting): The finite state machines in the modules “Ch X Rx Wrapper Module” which control the reception of data from the TDCpix ASICs have been modified to solve the issue reported at the Praga meeting: “… From the consistency checks performed on the reconstructed time stamps it results that a variable fraction of hit records extracted from the DRAM buffer belong to the adjacent time frame as shown in Mathieu’s presentation...” For this firmware version both the address of the external DRAM memory at which to store the TDCpix data collected within a time frame of 6.4us and the time at which this storage occurs are determined by the framing words issued by the TDCpix every at the end of each 6.4us time frame. “out of frame” timestamps should now be just those (few %) sent after the framing word. Undesired features (issues) : The most disturbing issue presented by version “wip20150915” is the fact that once every ~20 burst ( sooner at higher intensity) one of the 2 “Test assembly” modules stops sending trigger matched events; its trigger requests FIFO fills up and the EOB trigger request is lost. The response of the GTKRO board to the SOB trigger of the next burst is then piled on top of residual trigger matched events from the previous burst. This situation causes then the loss of the burst for which the “test_assembly” stopped sending data and of the burst for which the SOB is not properly aligned. Angelo Cotta Ramusino INFN-FE, for GTK_WG mtg, CERN 2015-10-13 GTK readout firmware: trigger matched

5. CHOKE / ERROR readyness : Answers to Marco’s questions: Question 1: (the CHOKE system): in which conditions your system generates a choke signal, how is this tunable to allow for the reaction delay of the L0TP, what happens if triggers are not stopped within a given time (how long), how is the occurrence of this recorded Answer: a GTK readout (GTKRO) unit generates the CHOKE signal whenever the number of entries in the trigger request buffers in either one of the 2 “Test assembly” modules exceeds the “L0_CHOKE_GENERATION_THRESHOLD”; this is currently fixed at the level of 32 but it can be made programmable to adjust to the reaction delay of the L0TP. Currently the trigger request FIFOs have a depth of 2 ** FIFO_WIDTHU = 64 locations, in the assumption the the L0TP would react to the CHOKE request with a few tens of us (no latency is applied to the CHOKE trigger). The FPGA has enough resources to allow the increase in depth of the trigger request FIFOs acording to the actual reaction time of the L0TP. If triggers are not stopped then the trigger FIFOs will fill and this will in turn generate an ERROR. An ERROR condition is also generated, currently, if the DRAM based latency buffers for GTK data cannot be accessed, this condition being flagged by the READY line from the DRAM controller become unactive. The CHOKE and ERROR flags are cleared when the corresponding triggers are received by the GTKRO unit. The above functions are implemented but not yet in-field tested Question 2: which mechanism does your system have to interrupt data input if instantaneous rate is too high, at which level is this tunable and how is this recorded Answer : the receiver modules in the current GTKRO trigger matched firmware feature a filter on input multiplicity and so the maximum amount of accepted timestamp within a frame period is limited to a safe (according to simulations) value. At the moment this represent only a debug feature and the filter level is currently set to about 50% of the nominal bandwidth. It could be made programmable if necessary. The condition of intervention of the input multiplicity filter is not currently flagged into the trigger matched data stream since the multiplicity filter should not be necessary under nominal operating conditions Angelo Cotta Ramusino INFN-FE, for GTK_WG mtg, CERN 2015-10-13 GTK readout firmware: trigger matched

6. DEBUGGING: Hardware issues: One issues has been recently discovered with one of the TTC-rq boards installed on the GTK-RO units; susbtituting the daughter cards solved the issue Firmware issues: Status of version under development: A new version (wip20151006) of the trigger matched firmware is being tested with boards connected in turn to the single chip assembly and to TDCpix ASICs in the gigatracker assemblies. Designed features of version “wip20151006” (incremental w.r.t. version “wip20150915”): the “TM_Data TX controller” modules which prepare the trigger matched data packets and the Ethernet MAC modules which send the trigger matched data packets to the subdetector PC are reset by a pulse scheduled 500ms after the EOB trigger. This clears all the FIFO buffers in which trigger matched data is stored and it should guarantee each SOB trigger to be aligned even after a burst in which one “test assembly” stopped more diagnostic features are being added to try to identify the condition which causes the “test assembly” module to stop processing triggers Angelo Cotta Ramusino INFN-FE, for GTK_WG mtg, CERN 2015-10-13 GTK readout firmware: trigger matched