Introduction to TEL62 (common) firmware M. Sozzi Pisa - January 30/31, 2014

Firmware development for the TEL62 must be a collaborative effort: Pisa will drive it, but contribution by at least 1 person per sub-detector is essential (need not be a firmware expert, but somebody willing to devote time to this) Mostly centered on TDC-daughtercard operation (but identify common blocks usable for LKr/L0 operation) Any sub-detector could be (in principle) L0-triggering or not Firmware will likely be (slightly) different for different sub- detectors: strictly confine sub-detector differences in few well- identified blocks. Generalities Mainz 2011

Use a common development environment Firmware must be centrally stored, consistent, modular, scalable and understantable to others besides the author Strict name-space and memory-space constraints Documentation is not an optional feature to be provided “later”, it must come with the code Each firmware module must be simulated and should come together with its own test bench code Live testing and diagnostic features should be incorporated General principles Mainz 2011

PP0 PP1 PP2 PP3 SL CCPC (CPU) GLUE TTCrx Quad GbE QDR DDR AUX Daughter card (TDCB) TEL62

Input buffer MONECS PP- FPGA V2 Output buffer MONECS = OK = In test DDR writer DDR Frame buffers PP-SL tester Test mem 1 ECS Logger Log FIFO ECS Triginfo RX SL Data FIFO Data organizer DDR arbiter Common Monitoring DDR reader TDC IB PP-SL transmitter SL FB mux DDR mux ECS Data mux Trigger generator Trig FIFO ECS Trigger mux MON FIFO arbiter OB Black box ECS Data compressor Hit counters Fake trigger generator PP-SL transmitter SL Trigger mux Subdetector Monitoring Choke generator ECS = accessible from CCPC MON = monitored FIFO Data processor All firmware is VHDL files, except test benches for simulation and some obscure verilog simulation files Pure text: great bonus for version control

Firmware tree

PP-FPGA SL-FPGA Sub-detector specific versions later: pp_rich, pp_lav, etc. Sub-detector specific versions (if needed) later: sl_rich, sl_lav, etc.

TEL62 Common and sub- detector memory spaces

Subdetector specific stuff Example: LAV (by dr. F. Gonnella)

Address space All registers, and most FIFOs/memories have direct ECS access (great bonus for debugging) Local bus is memory mapped from CCPC (all FPGAs and QDR memory) Separate sub-detector memory space: COMMON: - Generic registers - Monitoring registers - FIFOs - Memories SUB-DETECTOR: - Generic registers - Monitoring registers - FIFOs - Memories Sticking to the common structure helps readability, maintenance and software integration

Re-usability Several general-purpose modules available If you develop a module which can be useful to others you can ask to have it stored in a general library

PP SL DDR TDC Gbit QDR TTCrx CCPC GLUE (ECS) Simulation blocks Pisa

ItemNotesLHCb?WhoStatus Core & monitoringRegisters, memories, clocks, spy YES/NOPisaStarted Live test blocksData transfer testsNOPisaStarted Daughter-card commFor TDCBNOPisaStarted Data correctionOffsets, data manipolationNO Data formatter/writerTime framing, countingNOPisa DDR controlBurst writing/reading, arbitrationNOPisaStarted Data extractionTimestamp translation, framingNOPisa Trigger handlingTrigger type handling, arbitrationYES/NOPisa Trigger primitive genRICH version (time multiplicity) Can be different for other SD NOPerugiaStarted Inter-PP commTrigger primitive exchangeNO PP-FPGA FIRMWARE BLOCKS

ItemNotesLHCb?WhoStatus Core & monitoringRegisters, memories, clocks, spy YES/NOPisaStarted Live test blocksData transfer testsNOPisa Data mergerMerge 4 PP dataYES Data formatter/writerMulti-event packetsYES Gigabit controlEthernet (as in TELL1 + receiver) YES/NO QDR controlBurst writing/reading, arbitrationYESPisa Trig primitive mergerMerge 4 PP trigger dataNO Trig primitive handling Merge other board trigger data, format, send to Gbit NO TTCrx handlingTimestamps, trigger type, resetsYES/NOPisa AUX-board handlingInter-board communicationNO SL-FPGA FIRMWARE BLOCKS

ItemNotesLHCb?WhoStatus TDC simulation block NOPisaStarted DDR simulation blockNOPisa QDR simulation blockNOPisa TTCrx simulation blockYES/NOPisa CCPC simulation blockYES/NOPisa Gbit simulation blockYES/NO AUX cardDesign and building-- JTAG test vectors--Roma 2 OTHER TASKS

Some NA62 TDAQ concepts Refer to NA62 TDR for more information Three root concepts characterize the NA62 TDAQ 1.Full-digital integrated Trigger and DAQ (no separate sources for trigger and main data) 2.Unique communication channels (all commands through TTC, all replies through data path) 3.Strong state checking (mandatory reply to commands)

Timing (NA62) sob (pulse): start of burst (from TTC system) eob (pulse): end of burst (from TTC system) done (pulse): cleanup finished after eob (from every module) in_burst, in_endburst (levels) When in RUN mode: in_burst=0 in_endburst=0 in_burst=1 in_endburst=0 in_burst=0 in_endburst=1 sob eob done

(L0) triggers (NA62) Valid sequence (see NA62 TDR): SOB signal SOB trigger Any other trigger EOB signal EOB trigger Any other trigger sequence is invalid. Note that “EOB” is derived from SPS machine signal EE with a (common) arbitrary delay to allow for out-of-beam triggers, calibration, etc. Both “signals” and “triggers” are actually transmitted through TTC - Signal have direct effect on hardware - Triggers have a “L1” (TTC naming for LHC) pulse determining the 25ns timestamp, followed by a mandatory “Broadcast message” (6-bit trigger type for NA62)

Choke/error (NA62) Experiment-wide levels from each sub-detector to central L0 Trigger Processor (only 1 choke + 1 error line for a whole sub-detector) CHOKE: anomalous condition with buffers filling up, system still working fine, no data loss ERROR: anomalous condition, possibly unrecoverable, which might have caused data loss All systems are notified of start and finish of such conditions and must acknowledge this Monitored and recorded

Some project-wide signals Clocks: mostly 160 MHz Multiple clock domains sometimes a necessity: - ECS works at 20 MHz - TDCBs communicate at 40 MHz - GbE works at 120 MHz Reset (pulse): everything, at initialization reset_error (pulse): clear error conditions (normally not to be done at SOB) reset_monitor (pulse): clear counters, etc. freeze (level): stop everything for debugging (level) snapshot (pulse): store some monitoring values in registers sob: start of burst (pulse) eob: end of burst (pulse) done: cleanup finished after eob (pulse) in_burst, in_endburst

Module tests Providing a module self-test feature is recommended: -Poor man’s way: a script setting registers properly and triggering a sequence of actions resulting in some known data to be displayed -Nicer way: use the internal self-test firmware framework (TEST mode): inject test stimuli stored in ROM on a “test start” pulse and set “test result” lines

Error reporting Most modules are allowed a (single) error line Error condition must be sticky (not automatically reset with new burst) Error conditions can raise global ERROR line If more than one error condition must be distinguished, connection to the logger module must be used (recommended for debugging) Severe errors must be unrecoverable Also consider “crazy” illegal input to your module Protocol for handling of each kind of error will consolidate with time Modules can also provide input (maskable) for CHOKE generation

Logging The logger module (both in each PP and in SL) is a small (circular or linear) memory recording desired module conditions (usually errors) with automatic time-stamping and storage capability of a few module-specific 32-bit words In practice: a LOGGER_INTERFACE module exists to simplify operation (see examples). If you have multiple sources (i.e. state machines) for your (single) log lane some tricks are required (see examples).

End of Burst data In NA62 End of Burst data (monitoring, status, diagnostics, statistics, etc.) is stored together with event data, making offline data quality selection much easier and convenient. This is obtained because EOB data is the same as event data, with a different format as allowed for a different trigger. Your modules might want to provide some EOB data: describe the format and interface to the appropriate module (TRIGINFO_RX in PP and SL_DATA_SOURCE in SL) EOB data is (mostly) stored in ASCII format (!) for ease of inspection in raw data files.