Run coordination Summary C. Gemme – INFN Genova August 16th, 2016 5197 2172b 5181 2076b 5183 2173b 5199 2220b 5187 2172b 5194 3b 5198 2172b
LHC LHC Cardiogram
LHC dipole
LHC More bunches on Sunday!
Daily Operations - AC & AO LHC Last 24h Plan No beam in the morning. Then: new scheme with 2220b, 225ns spacing Aim to push to 1.2e11 bunch intensity in the coming days. Friday, November 23, 2018 Daily Operations - AC & AO
Daily Operations - AC & AO LHC status A31L2 Thursday: analysis of measurements No sign of changes in short Two “quenches” while ramping down RB.A12 from 6 kA with -10 A/s High current quenches must be avoided: would destroy the magnet 10 June 2016 @ 547 A FPA would trigger QPS heaters destroy magnet 3 Aug 2016 @ 295 A The second event triggered detailed investigations. Leave diagnostic tools in Mitigation measures: Could be explained with inter-turn short Remove “global protection mechanism” [link with FPA in other circuits in the same sector] present at high current and absent at lower currenttransition triggers QPS Follow-up (Wednesday) Reduce UFO-induced quenches: BLM thresholds to end of Run 1 values additional instrumentation Measurement campaign Increase QPS thresholds to minimise quench heaters intervention during FPA Various types of cycles evening and overnight. Friday, November 23, 2018 Daily Operations - AC & AO
Daily Operations - AC & AO LHC Plans Remove global protection mechanism: implemented on Thursday and validated Reduce BLM thresholds on complete MB and MQ arc families: thresholds changed on Thursday and changes to be continued on Friday Increase QPS thresholds on A31L2: new board installed on Thursday and successfully validated PS recovering from the vacuum leak More news at 10 am (hopefully the sector valves can be opened by midday) Cryogenic system: Pt 4 and Pt 6 were back around midnight RB.12 cycled to 6.5 TeV as of 21:30 on Thursday: no trips or other abnormal behaviour observed Two scenarios: PS is back in operation before the change of BLM thresholds is completed LHC can be safely operated and injection tests carried out (in view of p-Pb run) Once BLM changes are ready, resume physics with the same filling scheme used before the stop The LHC is back in operation before the PS While waiting for beam, access might be given Friday, November 23, 2018 Daily Operations - AC & AO
ATLAS One slide eachfor Data Quality and TDAQ Trigger L!calo L1topo e new item Id, Lar, Tile, Muons NCB/BCM/BLM/DBM/RADMON
Details C’e’ il run di domenica notte che forse non e’ stato ancora detto con casino sulle TGC??
Daily operations Peak lumi: 1.02x1034cm-2s-1 Lumi Ready Rec: 322 pb-1 Stable: 13h 24m 2076b_2064_1692_1765_96bpi_23inj Fill 5181 – Run 305777 [02:13] Stable beam Smooth running [15:30] Programmed dump Interfill and LHC injection tests [17:38] Quick Test to configure/unconfigure L1CTLatencyChecker for CTP on-call. [17:46] Combined quick L1Calo/L1topo [18:05] Start of Run: FATAL thrown by L1CTLatencyChecker, CTP on-call enquired. [18:17] Several attempts to restart L1Muctpi and LatencyChecker unsuccessful; Rebooting the L1muctpi machine fixed it.
Fill 5183 – Run 305811 Peak lumi: 1.04x1034cm-2s-1 INDIV bunch added for IBL timing scan. [19:15] Stable beam 20:07 TGC busy for several seconds, in coincidence with exception from TGC-RCD-C05. Such error appeared later on repeatedly with no busy. 22:07 LAR HEC A HV (Not at Optimal voltage) 22:28 TGC HV A M2 Sector 12 tripped and recovered 23:05 IBL timing test begin: Go from 2BC to 3BC readout. Trigger with special keys to keep the rate < 65 kHz and enhance individual bunch. 01:08 Reverting at 3 BC and to standard keys. 04:15 SCT Busy [05:37] Beams dumped Peak lumi: 1.04x1034cm-2s-1 Lumi Ready Rec: 273 pb-1 Stable: 10h 22m 2173b_2161_1838_1916_96bpi_25inj SCT is still 100% busy despite the sct-rod-removal and sct-rod-recovery. Expert called. 04:28 SCT expert enabled data taking again. However, one SCT ROD now missing. Expert working on recovering it. 04:38 SCT ROD back again. SCT RC called back and said that a link was being very problematic, but it was not showing up as a link error. Interfill Calibrations Problems in starting the run HLT expert informed. Access until 12
Peak lumi: 1.06x1034cm-2s-1 Lumi Ready Rec: 260 pb-1 Stable: 9h 33m 2172b_2160_1855_1948_96bpi_24inj Fill 5187 – Run 305920 [20:28] Stable beam 01:35 ERROR TGCFE RCD_ECA03 TTC restart on TGC needed according to the expert. Successful. 03:19 DCS alarm: TRT ENDCAPC LV Stack26 … LV has been switched off in this FE board. expert called, performing intervention. Solved. Few occurrences for IBL causing 100% deadtime [06:03] beam dump, trip of RQ4.L5 DQ: InDet-Eff: Hit efficiency in EA and EC are 1% lower than reference similar in yesterday's run. InDet-Tracking: Hot spot in eta-phi tracks with no IBL hit but a hit Is expected in eta~-2.5, phi~3: also appeared in previous runs, inform the ID shiftershould not be a problem Trigger: Many Histograms in DQMD are flagged red (see summary previous shifter), becomes difficult to spot if a new issue pops up.
Weekend Summary Friday Saturday Sunday C. Gemme, INFN Genova, RM report 15 August Weekend Summary Friday Saturday Sunday
Weekend summary: Friday Beam recovery LHC not available yet in the morning PS problem (fixed at noon) LHC implementing protections for the faulty dipole ATLAS calibrations CTP sw tests (very quick), not done the longer test announced in the morning; CSC test: From now on, CSC will NOT give "LatenctMismatch" error, and will just set the latency correctly without any expert intervention. CTP + L1 topo test Afternoon standalone long tests for several subdetectors: Tile, CSC, TGC, MDT (testing recovery of the chambers in standalone) ATLAS data taking LHC planning a first full round with 3b only Defining keys as STANDBY [~19] a first injection but beam dumped 21:35 New access needed to fix beam dump system at point6 [00:38] Stable beam for 3bunches Fill 5194, just few minutes [05:51] Stable beam for 2722b Fill 5196 just few minutes
Fill 5194 – Run 306247 – Single_3b_2_2_2_with_nc_probes [00:00] Switch to standby keys for 3b. However L1_RD0_UNPAIRED_ISO went up to ~2 kHz since prescale of ~65000 got set to 1 when moving to 3b standby keys activate FFTV protection for IBL and cause 100% IBL busy. [00:20] Need a TTC reset of IBL was issued. [00:39] Stable beam 3 bunches 7’ with stable beam [00:43] Beam dump Fill 5196 – Run 306252 - 25ns_2172b_2160_1855_1948_96bpi_24inj 05:51 stable beams 05:55 beam dump due to electrical perturbation
Weekend summary: Saturday Physics 08:30 Stable beam for 2722b Fill 5197 16:43 Stable beam for 2722b Fill 5198
Fill 5197 – Run 306269 Peak lumi: 1.06 x 1034cm-2s-1 Lumi Ready Rec: 159 pb-1 Stable: 5h 15m 2172b_2160_1855_1948_96bpi_24inj Fill 5197 – Run 306269 [08:34] Pixel: noticed by chance during ADJUST that the PPC of ROD_L1_S10 was stuck. Tried to recover before declaration of stable beams Ready for physics delayed by ~3’. PPC status should be checked routinely until an automatic monitoring can be established. [08:38] Stable beam Enabled several additional L1Topo items. The L1 rates all seem fine and the stream rate is around 40 Hz. Trigger shifter noticed that some Predictions need to be updated. L1_3J50 looks wrong to be followed up. [13:52] Beam lost due to BLM UFO Some busy by RPC, TGC, Pixel, SCT
Fill 5198 – Run 306278 Peak lumi: 1.08 x 1034cm-2s-1 Lumi Ready Rec: 392 pb-1 Stable: 16h 23m 2172b_2160_1855_1948_96bpi_24inj Fill 5198 – Run 306278 16:35 noticed in adjust some delay (1-2’) between the ATLAS pref lumi and the ‘same’ information in page1. It is correct in ATLAS toLHC tab [16:43] Stable beam 21:45 LAR EMEC HV TRIP - recovered 01:12 power cycle of Tile Module LBC44 DQ In histogram "Eta-phi of tracks with no IBL hit but hit is expected, ratio to total tracks" in OHP, there are some hits expected (but without IBL hit) around (eta,phi)=(-2.5 , -1.5). MDT got 50% busy on and off around 07:42 for 5 minutes. The offending channel was BC-0 and it was fixed by itself at 07:48. [09:02] Beam dumped Some busy by IBL, SCT, MDT
Weekend summary: Sunday Data Taking Changed EMECA HV Module as 4 channels do not work LHC raising to 2220b
Fill 5199 – Run 306310 Peak lumi: 1.04x1034cm-2s-1 Lumi Ready Rec: pb-1 Stable: 25ns_2220b_2208_1940_2036_96bpi_24inj Fill 5199 – Run 306310 25ns_2220b_2208_1940_2036_96bpi_24inj [13:01] Stable beam [] Beam lost