“The Dry Run” Gianluca Lamanna (CERN) TDAQ - NA62 Siena Meeting

Introduction July 16 thThe Dry Run started July 16 th 2 daysWe had regular meeting each 2 days 35Good participation (peak of more than 35 people) InstallationInstallation –Most of the time –Several small problems (network, cabling, bugs in the firmware, …) CommissioningCommissioning –Only partially done, but all the device are working TestingTesting –Not all the test foreseen has been finished Very useful experienceVery useful experience: TDAQ –To converge in the building of the TDAQ issues and solutions –To found issues and solutions

LAVFEE, TEL62 and TDCB LAVFEE:LAVFEE: Installed since the beginning. No major issues. TEL62+TDCB:TEL62+TDCB: arrived after few days. Very small problems fixed in the lab.

Installation Status: what we have… DetectorrackCrateTel62TDCBcables CEDARCEDAR-01CEDARtel62cedar11 CHANTILAV1-01LAV1tel62chanti11 LAV1LAV1-01LAV1tel62lav1310 LAV2LAV2-01LAV2tel62lav2310 LAV3LAV3-01LAV3tel62lav3310 STRAWSTRAW1-02VME9UNO CHODCHOD-01RICHtel62chod28 LKREBFBNO MUV2EB24MUVtel62muv226 MUV3EB24MUVtel62muv3310 SACEB24MUVtel62muv211 L0TPEB21ROMA2tel62l0tp (tell1) NO

Installation: FEE DetectorFEEN. boardsNote CEDARCEDAR proto1 CHANTICHANTI proto+LAVFEE 1+1 LAV1LAVFEE5 LAV2LAVFEE5 LAV3LAVFEE5 STRAWSRB+Cover1+1 CHODLAVFEE4 LKRCPD+SLM40% TOT MUV2LAVFEE4 MUV3TRAM+CFD10+16 SACLAVFEE1

Installation: cabling LAV1MUV3

Installation: LKr & Straw Straw:Straw: –Installation of the SBC+SRB+Cover –Standard VME crate in STRAW1- 02 rack –Temporary solution for the network connection –Problems with the VME readout LKRLKR: –SLM recabling –SLM software completed –New trigger distribution system installed LKr

Installation: Trigger partition DetectorLTUTTCexOwner(TTC ex) TTCitSplitter CEDARLTU_CEDARTTC-CEDARTDAQYES CHANTILTU_CHANTITTC-CHANTICHANTIYESYES(LAV) LAV1LTU_LAVTTC-LAVLAVNOYES LAV2LTU_LAVTTC-LAVLAVNOYES LAV3LTU_LAVTTC-LAVLAVNOYES STRAWLTU_LKRTTC-LKRCHANTINO CHODLTU_CHODTTC-CHODPOOLNO LKRLTU_LKRTTC-LKRCHANTINO MUV2LTU_MUVTTC-MUVMUVYES MUV3LTU_MUVTTC-MUVMUVYES SACLTU_MUVTTC-MUVMUVYES TALKLTU_LKRTTC-LKRCHANTINO

Installation: LTUs and TALKs 2 TALK2 TALK boards installed LTUConnection with LTU installed and tested Small problems in the beginning (LVDS termination, TALK firmware, LTU delays,…) LTU and TALKControl software of LTU and TALK almost completed

Installation: choke/error infrastructure DetectorPATH CEDARCEDAR-01  GTK3-01  EB20 CHANTILAV1-01  CHEF  LAV3-01  CHEF  STRAW1-02  RICH- 01  EB20 LAV1LAV1-01  CHEF  LAV3-01  CHEF  STRAW1-02  RICH- 01  EB20 LAV2LAV2-01  LAV3-01  CHEF  STRAW1-02  RICH-01  EB20 LAV3LAV3-01  CHEF  STRAW1-02  RICH-01  EB20 STRAWNO CHODIRC-01  EB20 LKRNO MUV2EB24  CHEF  EB20 MUV3EB24  CHEF  EB20 SACEB24  CHEF  EB20

Installation: choke/error infrastructure choke/errorCEDAR, LAV123+CHANTI, CHOD, MUV1+MUV2+SAC4 choke/error lines for this year (CEDAR, LAV123+CHANTI, CHOD, MUV1+MUV2+SAC) RJ45  RJ11RJ45  RJ11 adaptors

Installation: SPS signals SPS signals NIM logicThe SPS signals distribution is done using the old NIM logic connected to the new trigger partition PC-TIMER SOB/EOBThe PC-TIMER is devoted to distribute the SOB/EOB to the software systems (PCFARM, RunControl, SLM…) NIM LTU+ TTCex TALK TTC PC- TIMER

Installation: Network & Farm 6 switches6 switches installed downstairs 1 Router1 Router in the PCFARM LKr systemAll the equipments to boot the LKr system are installed and working DetectorSwitch CEDARCEDAR-01 CHANTILAV1-01 LAV1LAV1-01 LAV2LAV2-01 LAV3LAV3-01 STRAWTemp. solution CHODIRC-01 LKREB MUV2EB MUV3EB SACEB PCFARMRouter 6 PCs6 PCs installed and configured Merger PCMerger PC data acquisitionSoftware for data acquisition data mergingSoftware for data merging online monitoringSoftware for online monitoring

Installation: Run Control is working!Run control is working! TEL62, TALK, LTU, SLM, PCTIMERConnection with TEL62, TALK, LTU, SLM, PCTIMER tested PCFARMPCFARM connection almost ready DBDB to be define better DCSInteraction with DCS

Installation: DCS & DSS DCSDCS control program installed in Control Room –Possibility monitor and control the crates DSSDSS control program installed in Control Room –Alarm working for all the installed racks

Installation: GPU and Ferrara L0TP GPU pc CHOD rackGPU pc installed close to CHOD rack real data from CHOD –Possible tests with real data from CHOD simulated RICH-like data –Possible tests with simulated RICH-like data P2P connection APENet GroupSpecial card for P2P connection with the GPU installed by APENet Group (from Roma1) L0TP PCFerrara L0TP PC for test installed (and removed)

Installation: Elog and Twiki ElogElog –Installed –Essential for the technical run –User guide: NA62 note TwikiTwiki Very very useful A lot of information Several contribution from subsystems Additional information are welcome

Installation: Control Room private networkAll the PCs (but the DSS) are in the DCS private network WiFi availableWiFi available in Control and Conference room notebookSome outlet configured for notebook (label PB) System control PC (temporary) Run Control Beam Control Online monitor & Event display DSS DCS 3 PCs Data Acquisition Run Control 3 PCs for Data Acquisition and Run Control (5 screens) 2 PCs Beam Control 2 PCs for Beam Control (2 screens) 1 PC DSS 1 PC for DSS (1 screen) 1+1 PCs DCS 1+1 PCs for DCS (4+1 screens)

Installation: additional facilities

… what is missing FEE electronicsFEE electronics for some detectors –Cedar, Chanti HVHV for most of the detectors: –Cedar, Chanti, LAV, MUV2, MUV3 PCFARM softwareSmall details in PCFARM software: –Driver license, DIM connection, “last trigger” signal DCSDCS to be completed –LAVFEE control (software ready) –…

… LKrLKr –Fastbus branch –Test the chain (CPD+SLM) –Test the Pcfarm readout –Calibration system TEL62 firmwareTEL62 firmware –Still missing some part (special trigger, eob,…) –Trigger primitive generation CDRCDR Recabling in EBRecabling in EB Longer fibers installationLonger fibers installation Straw switchStraw switch

What is missing: cables CEDAR:CEDAR: from the KTAG CHANTI:CHANTI: from the final FEE prototype LAV1-2-3:LAV1-2-3: from the detector to the FEE STRAW:STRAW: everything CHOD:CHOD: OK MUV2:MUV2: from the patch pannel to the FEE MUV3MUV3: OK SAC:SAC: from the flange to the FEE

Issues & critical points CratesCrates –Critical situation Choke/error adaptorsChoke/error adaptors –To be understood Network driver licenseNetwork driver license –Buy license? Different NIC in the future PCs? CrateSituation CEDARPS+FT in reparation LAV1OK LAV2OK LAV3PS in reparation STRAWOK (in CR) CHODOK ROMA2OverVoltage on 48V, 3V3 MUVOK exMUVRepaired (to be installed in CEDAR) TTCexTTCex 2 boards with problem with the clock phase stability after power up

Issues & critical points TEL62 firmware:TEL62 firmware: –Bug in the merging of the data before the preparation of the packet –It’s possible to extract the data from the DDR using the triggers but it isn’t possible to send these data using the ethernet Data correction Gbit handling Data formatter and writer Data marger

Test done Connection to the TEL62Connection to the TEL62 –Problems with the network in the beginning –Single server for all the boards Power on of the TDCbsPower on of the TDCbs –JTAG & I2C working –Initialization without errors LAVFEELAVFEE –Remote control using the DCS (tested on LAV1) –Local control using serial connection  pulser JBC load of firmwareJBC load of firmware –Possibility to upload the firmware remotely LKr new trigger distributionLKr new trigger distribution –TALK board trigger distribution is working  the old TS can be dismounted

Test done: noise in TDCs DetectorTDCB 0TDCB 1TDCB 2TDCB CEDAR ok ??? CHANTI ok 20, ,6 LAV1 ok 18 ok 23, 16 ok 31 ok LAV2 ok 4 27, 29 ok LAV3???????????? CHOD ok MUV2 ok 238, 12 MUV3 ok 1925 SAC ok In some case the noise can be fixed moving the connector! The periodic noise in the not plugged TDCs shows some effect due to the periodical digital generated signals inside the FPGA, with the TDCs

Test done: trigger count TALK LTUTrigger sent from the TALK to the LTU –Correct number –Correct number counted in several condition bugs –Useful to found bugs in TALK & LTU firmware clock phase –The result of the test depends on the clock phase –BC delay & TTC_interface –BC delay & TTC_interface registers in the LTU should be set in correct way TALK –The TALK have to be synchronized with the TTC clock TALKTEL62Trigger sent from the TALK to the TEL62 in all the TEL62 –We are able to measure the correct number of triggers in all the TEL62

Test done: synchronization synchronization25 ns synchronization LTU +TT cEx TEL62 TALK L1 L2 D1 d TALKTEL62reset LTU level The timestamp counters, both in TALK and TEL62, are reset using a signal synchronized at the LTU level. TALKTEL62depends only on (D1+d+L1) Can be easily demonstrated that the time difference for the trigger between the TALK and the TEL62 counters depends only on (D1+d+L1) (where D1 is the cable for trigger info from TALK to LTU, d is the time inside the LTU+TTCex, and L1 is the fiber between LTU and TALK + the delay inside the TTCrx chip) the stability We care only about the stability of this quantity SOB

Test done: synchronization DetectorDelay (HEX) in clk cycles LTU CEDAR+ALTU_CEDAR CHANTI+ALTU_CHANTI MUV2+ALTU_MUV MUV3+ALTU_MUV LAV1+BLTU_LAV LAV2+BLTU_LAV CHOD+BLTU_CHOD BC_DELAY and TTC_INTERFACE LTU The residual difference is due to different BC_DELAY and TTC_INTERFACE in the LTU It’s constant with time TEL62 This value will be compensate in the TEL62

Test done: PCFARM TEL62PCFARMSending packets from the TEL62 to the PCFARM (1 single PC for the moment) and build the event –Troubles in the beginning to send the packets through the network –Now we are able to send packets from each board to the PCFARM PCFARM merger PC final burst fileThe PCFARM receives the packets and send the data to the merger PC to build the final burst file empty packetsTest done with empty packets TEL62 Switch Router PC Merger

Test done: DDR data extraction One of the most delicate and difficult things: huge DDR –The data are stored in a given position in the huge DDR memory (for instance in the position of the timestamp 0XC1A0C1A0) TALK “correct” –The TALK sends a trigger in the “correct” instant to extract the data local buffer –Result observed in the local buffer latency compensation –Test done only with simulated data and without latency compensation (a part the delay compensation) xC1A0C1A TALK Send a simulated trigger at timestamp 0XC1A0C106 Packet preparation DDR control TTC receiver Local Buffer

Test done: P2P GPU data transfer GPUGPU Network interface GPU –Direct transmission of data from Custom made Network interface and GPU –A standalone test has been carry out to check the compatibility with our system CPU RAM BUS PCI-Express GPU FPGA Data

Test missing: Trigger generation 2 ways2 ways to trigger Both ways need to be tested (everything is ready in the TALK board) Some part in common Lemo: NIM cratesLemo: additional NIM crates (for instance close to the CHOD), and cabling Ethernet:Ethernet: firmware for primitives generation Lemo Timestamp latch Trigger logic Ltu Eth Primitives matching Ltu Trigger logic

Test missing: Data extraction with trigger bug TEL62 readout data with the triggerDue to the bug in the TEL62 we aren’t able, at the moment, to readout data with the trigger full chain will be testedAs soon as the bug will be fixed the full chain will be tested (packet preparation, addressing round robin logic, counters,…) xC1A0C1A TALK Send a simulated trigger at timestamp 0XC1A0C106 Packet preparation DDR control TTC receiver Local Buffer

Test missing: Pulse the TDC pulseTDCWe have several ways to pulse the TDC: LAVFEE –From the LAVFEE in free running external pulser TDCB –Controlling the external pulser with a signal coming from the TDCB TALK board TDCB –Using the TALK board to trigger the external pulser (through the TDCB) Few tests done (CEDAR, LAV, CHOD, MUV3,…) systematic check TDCNot yet systematic check to qualified all the TDC channels we need to use

Test missing: EOB and Choke/error tested EOB trigger it works EOB doesn’t containsWe tested the answer of the TEL62 to a special EOB trigger (it works), but at the moment the EOB doesn’t contains information RJ45/RJ11 didn’t test choke/error signal TALK testedDue to the problem with the RJ45/RJ11 connectors we didn’t test the sending of the choke/error signal to the TALK (but the infrastructure has been tested).

Test missing: others Long run acquisition, multiple PCsLong run acquisition, multiple PCs bugTEL62license problem –Both due to the bug in the TEL62 and the license problem Rate testRate test TDCB, TEL62, trigger distribution and data acquisition system –To characterize the bottle neck in the TDCB, TEL62, trigger distribution and data acquisition system Online MonitoringOnline Monitoring is ready –The infrastructure is ready. Reconstruction and decoding of the detectors should be prepared from the sub-system groups (volunteers?). proto event-display –Possibility to design a proto event-display for the technical run CDRCDR isn’t readyshould be tested –The link isn’t ready, but the scripts should be tested using the standard network

Test summary SPS signals Clk distribution Eth. primitives Lemo trigger Trigger distribution Choke/error CDR TALKLTU TEL62 PCFARM FEE >75% 25%<x<75% <25% Legenda: Run Control

Conclusions complete (almost) readout electronics learn a lotThe dry run was useful to complete (almost) the installation of the readout electronics and to learn a lot about the system. installed something missingMost of the equipments are installed, but still there is something missing both in infrastructure (choke/error, full chain test, …) and detector side (cabling, set of the threshold,…) very positive has been testedThe experience has been very positive but not everything has been tested as foreseen can’tis ready to take data for the technical run: additional period before the particles!We can’t say that the system is ready to take data for the technical run: we need an additional period before the particles!

COLLABORATION WEEK Final tests period Starting period Technical Run

Acknowledgment all the participantsMany thanks to all the participants… … particularly to all the persons allowed us to have a successful DryRun: Riccardo Ferdi Marco Michael Jeckel Gilles Tonio Valeri (and many others)