Recall: CSC TF Halo rate spikes during run 136066 (23 May) Each spike in beam halo was correlated with a CSC FMM response… Looking at one of these events (8:28:38), the FED crate monitoring logfiles, the ME+ DCC FIFO's are filling up, backpressuring to DDU's... 7 spikes in 3.5 hours, with “bursts” of trouble… Note: this plot is UTC (i.e. 5:00 = 7:00GVA time) Halo triggers per 10 seconds 26 May 2010 G. Rakness (UCLA)

Lindsey Gray (UW) presented a talk on behalf of CSC Prompt Feedback, looking into this problem… It appears we have tracked down the cause of these spikes, but we still have problems… In addition, see next slide… Beam loss plot from A. Ryd (Cornell) 8 June 2010 G. Rakness (UCLA) http://indico.cern.ch/conferenceDisplay.py?confId=97196

Saturday 5 June Trigger elog… “In collision run 137028 we got quite some CSC error (solved by reset) at; LS:170, 183, 201, 210 CSC experts have seen that it was in coincidence with spikes in L1_SingleMu20” L1A L1_SingleMu20 L1_BeamHalo L1_BeamHalo was quiet L1_SingleMu20 was spiking (N.B. it was also spiking on 23 May) There is no nice “Beam loss” plot (like on page 2) that I have seen 8 June 2010 G. Rakness (UCLA)

If you look closer at the problem, it turns out there are many… What is the difference between 23 May (beam halos) and 5 June (no beam halos)? L. Gray (UW) is looking into that FMM response may be buggy S. Durkin (OSU) is aware there may be a problem Carl Vuosalo (OSU) is going to start looking at that S. Sakharov (WSU) is creating status digis so that one doesn’t have to look at event dumps Looking at DDU/DCC monitoring logs, it looks like the FED software is buggy. This is what “releases” the FMM response from DDU… P. Killewald (OSU) is looking at that 8 June 2010 G. Rakness (UCLA)

Resistance to Hard Reset Automation FMM=ERROR is an explicit request for TTC Hard Reset Why does everybody (except us) thinks that the Hard Resets should be periodic, and not on request? Questions that Ivan, Wesley, Joao, and Tiziano are asking include… Why are we filling up FIFO’s and going out-of-sync? We are supposed to send FMM=WARNING and FMM=BUSY to prevent this kind of thing from happening. Why do we need a hard reset if we are just out-of-sync? See previous page… It makes sense to “clean house” before we can start requesting things from CMS. 8 June 2010 G. Rakness (UCLA)

Other stuff CEO training CSC Activity trigger has been approved https://twiki.cern.ch/twiki/bin/view/CMS/CeoTraining I have been talking to several people at different levels (some beginning students, some post-docs) about becoming CEO’s, following the above recipe List of CEO’s for next round of shifts to go to Richard Breedon next week CSC Activity trigger has been approved Got Laria’s talk on-shell, gave talk at Trigger Studies Group meeting Will continue to encourage Laria about getting it into the HLT 8 June 2010 G. Rakness (UCLA)

To do Tomorrow Event dumps for UCLA Elog ME-3/1/10 and other cable lengths Record TTCci configuration information Send “Expert system” analysis to Misha, then to Evaldas and Jinghua… Analysis Look at AFEB time bins ON for timing scan runs Point 5 Continue to encourage automated Hard Resets automated… Documentation Look at/input transitions into TriDAS/emu/doc/html/EmuFSM.xhtml To do at 904 Online software Put ALCT firmware database check into ALCT firmware write routine Resurrect ALCT self-test ALCT hot-channel mask in xml file/config DB Expert config check Configuration check with separated DMB and CFEB Managerial stuff Talk to Valentin Sulimov and Sergey Vavilov regarding responsibilities w.r.t. 904 and calibration runs 8 June 2010 G. Rakness (UCLA)