Inner Detector Combined Shift Tools and Procedures version 0.9 September 6th, 2011

Table Of Contents ID General FSM Panel IDE FSM Panel ID General Alarm Panel DAQ Tools ID Operations & Problems Database Checklists & Elog Special Actions: Warm start/stop, beam dump Online Histogram Presenter Data Quality Monitoring Display

ID General FSM Panel

FSM Panel Explained ID FSM Tree FSM Main Panel FSM Help LHC Handshake Status Fast Navigation Alarm Summary ID FSM Tree FSM Main Panel FSM Help Secondary Panel

ID FSM Main Panel ID Env. SCT Pixel TRT BCM/BLM Background Conditions SCT Automatic Turn On Pixel TRT BCM/BLM & Post mortem Injection permit

ID Environment Evaporative Cooling Status Compressor Configuration Liquid level Environmental Cooling Status Thermal Enclosure Heater (TEH) Trip Recovery Button TEH Status

SCT Endcap C Status Barrel Status Endcap A Status Pix SCT Infrastructure Status Snapshot shows normal status in STANDBY Should ALL be GREEN when SCT is ON

Pixel Barrel Layers C-Side Disks A-Side Disks Pixel Infrastructure Snapshot shows normal status in STANDBY Should ALL be GREEN when PIX is ON

ALL indicators should ALWAYS be green TRT ALL indicators should ALWAYS be green Endcap C Barrel C Endcap A Barrel A TRT Infrastructure Gas Gain Stabilization System (GSSS) Status

Beam Conditions (see later slides for descriptions) BLM, RAD Mon SCT Auto-ON Injection Permit Status Ready For Inj. Status Inj. Permit Mask Configuration BCM post mortem counter State 6: normal State 5: PM analysis)

Backgrounds Green: OK Red: Too high (or no data) SCT Occupancies Pixel BCM Rates Background monitoring active? Stable beam yes/no?

Use Tabs to navigate panel IDEnvironment FSM Reached via IDE button Useful for more detailed look at ID environment & beam conditions, eg in preparation for warm start Use Tabs to navigate panel

IP1 Collimator Check Panel (IDE FSM) Absorbers Background rates Beam pipe pressure Tertiary collimators Beam parameters

Cleaning Collimator Panel (IDE FSM)

Beam Rates Panel (IDE FSM) LUCID, ZDC rates Beam pipe pressure, BCM rates, LHC params SCT/TRT occupancies

Collapse of the separation bumps Example - Beam Rates SQUEEZE Lumi optimization RAMP Collapse of the separation bumps → collisions

IDG Alarm Screen Click On Alarm for Sort According to Severity details, help, and logging Sort According to Severity

IDG Alarm Help Right click on alarm to see options Insert Directly to elog Get Alarm Details Open Alarm Help (Twiki page) All alarms shall be logged Errors and Fatals shall be followed by a call to the expert

IDG Alarm Help (Opened by selecting ‘Alarm Help’ in options list) Alarm explanation along with actions to be taken. Please report any missing help pages!

DAQ Panel Busy monitor ATLAS partition OHP DQMD

DAQ Tools Shifter Assistant Alerts the shifter to specific DAQ error conditions that require shifter action Eg ROD busy or disabled etc Complements IGUI panel indicators Includes instructions for the shifter IGUI Panels Indicates DAQ related alerts and warnings Provides shifter tools to take the appropriate action MRS Warnings/Errors/Fatals from DAQ applications Eg missing essential files, application crashes etc Usually require expert actions, but shifter should monitor MRS for ERRORs or FATALs

Shifter Assistant Tool New alert Expand if new alert http://pc-tdq-mon-61:8081/alerts/Detectors Link is available under the ID Menu at the ID Desk

Shifter Assistant Tool Read the messages and mark them as read Info & Warnings can be ignored Errors and Fatals shall be followed by a call to the expert

More details in SA

ATLAS DAQ IGUI Loading IGUI Panels

IGUI Panels The IDPanel under the ID tab of the TDAQ IGUI provides: Visible indications of any SCT and TRT DAQ-related alerts or warnings that the ID shifter should react to In principle, the same alerts/warnings are displayed on the Shifter Assistant Tools to react to those alerts The IDPanel should be open and visible at all times on the ID shifter desktop At the start of the shift, the shifter should open IDPanel (if not already open) by ticking ‘IDShifterGui’ in the list of tabs shown when clicking the ‘Load Panels’ button at the top of the TDAQ IGUI For info, the IDPanel can be closed by unticking ‘IDShifterGui’ in the same list, but the shifter should leave the panel open

Panel Structure To the left of the panel is an expandable tree indicating the DAQ status for both SCT and TRT Green: OK, Yellow: WARNING, Red: ALARM The ‘ID DAQ Status’ icon colour indicates the combined status for both SCT and TRT The ‘TRT’ and ‘SCT’ icons in the tree indicate the overall status of TRT and SCT respectively If yellow or red, click on the sub-tree component to see the origin of the warning/alarm and the tools to resolve the problem

Overall SCT Control and Status panel, visible by clicking on the ‘SCT’ icon in the tree

SCT Control & Status Panel : 1 Control Mode: Enables recovery tools Only one TDAQ IGUI has control mode enabled (if someone opens another IGUI and takes control mode, this IGUI will have control mode disabled) Module Recovery: Automatic recovery of link errors Should be enabled by default If Control Mode is disabled, module recovery gets disabled too. Global Reconfiguration: Time between automatic global reconfiguration of the SCT to counteract chip configuration corruption by SEUs Set to 30minutes by default

SCT Control & Status Panel : 2 Disabled ROD count across full SCT Green: 0 Yellow: 1 (take action!) Red: >1 (take action!) Link Error count across full SCT Green: <50 Yellow: <100 (call expert!) Red: >100 (take action!) Busy ROD count across full SCT Green: 0 Red: >0 (take action!) If any of these 3 indicators are not green, click on the crate icons which are non-green to perform actions

ROD status in first column (Grayed out slots mean no ROD present in that slot) Green: ROD is Enabled, and not BUSY

Useful to see if 100% busy or not ROD percent busy Useful to see if 100% busy or not

Link Errors per ROD Should be ideally 0 If non-zero, module recovery is attempted automatically (if enabled in control&status panel) Green: <10 Yellow <20 Red >20

Visible by clicking on TRT Recovery TRT Panel Visible by clicking on TRT Recovery

Synchronization: - If the # of sync problems is > 0, an automatic resync is performed (no shifter action necessary) - If prompted by the Shifter Assistant or an expert, click the Request Resync button to manually request a resync 8/25/11

- If this doesn't work, call the TRT DAQ expert on 160531 ROD Recovery: - If the # of disabled RODs is > 0, click the Recover Disabled RODs button to attempt to recover these. - Run Control will have to acknowledge a popup window for this procedure. - If this doesn't work, call the TRT DAQ expert on 160531 8/25/11

8/25/11 Specific ROD Recovery: - If instructed by the shifter assistant or an expert, you can force a ROD recovery. - Type the ROD address into the text box and click Request ROD Recovery. - Run Control will have to acknowledge a popup window for this procedure. - If problems occur, call the TRT DAQ expert on 160531 8/25/11

MRS Call experts for ERRORs or FATALs Enter subscription string to filter for SCT, TRT, PIX and BCM “SCT|TRT|PIX|BCM” Call experts for ERRORs or FATALs

Not supposed to be an ID shifter tool Pixel Rod Mon Not supposed to be an ID shifter tool

ID Operations and Problems DB Web interface opened via ID menu on desktop (keep it open for easy/quick reference)

Searching Prob. DB Specify System Specify Category

Click on DB entry to get details Searching Prob. DB Click on DB entry to get details Enter String to search DB

Details with explanation of DB entry and solution prescribed Searching Prob. DB Details with explanation of DB entry and solution prescribed

Checklists and Procedures Elog Start of Shift Warm Start Warm Stop Beam Dump During Shift End of Shift

Elog Maintain a log throughout your shift of what happened, and upload it at the end of the shift. A new elog interface will be available “real soon” for use by the ID shifter: - uploads your entries directly as you type them, line by line, and appends them to the shift elog summary - useful for experts and run coordinators to see the status of your shift in ‘real time’ - will only append (not delete/change previous lines)

Start of Shift Arrive to Shift 15 Minutes before Start of Shift to get summary from previous shifter! Login to Elog DCS ID FSM Panel DCS Alarm Screens Run Com Tool (not yet possible) Introduce yourself to the shift leader (if they haven’t done so already). Read latest elog messages from last 24 hours. Keep elog regularly updated Ensure that OHP, DQMD, SA, and IGUI tools are up and running with the proper configurations. Ensure that individual ID desks are running their proper tools as well. Check the alarm screen for any alerts. Check the DAQ status (IGUI ID panel and SA) Check the status of the pixel infrastructure (pixel igui, checked by SA) Check the FSM Status (for silicon should be READY/OK during data taking, otherwise in STANDBY/OK, TRT should always be READY/OK).

During Shift Monitor and acknowledge any alerts from the SA tool, and take action as needed Monitor the IGUI ID panel and take action as needed Log any alarms in the FSM. Acknowledge them once they have been logged and WENT. Take any necessary actions needed (use Alarm Help and/or Problems DB tool). Check the trend in evaporative cool plant liquid level (~ 1kg / 8 hours). Check ROS occupancy is non-zero in FSM. If it is zero and beam is present, then ask the Run Control Shifter to restart the SCT ROS Monitoring. During data taking, check hourly DQ checklists Number of disabled pixel modules, especially b-layer (OHP) Check Pixel Module Errors (i.e. LVL1ID errors) (Pixel Rod Mon / DQMD)

Start of Run It is important to be specially vigilant at the beginning of a run. Problems most likely occur at this point. Be aware of: FATAL / ERROR messages from the Pixels, SCT, TRT, or BCM appearing in the MRS Check BUSY status using the BUSY status display. If no triggers are coming through one of the systems could be busy. Check the IGUI ID panel to ensure all is green Stop-less removal could happen at this stage, the run control shifter should tell you if an ID System goes busy and may disable one or more of the RODs. Follow the instructions in the operations database if this happens. Ensure that all the Pixel RODs are allocated by the Central DAQ. If not an ERROR should appear in the SA

During a run Keep checking the SA and IGUI ID panel. If the run control shifter did not tell you he/she disabled an SCT ROD, you would see a big increase in link errors. Check SCT ROS Monitoring every hour. Look for errors and anomalous values in the results Continue to check for errors in MRS Watch out for busy of any of the detectors. If the run control shifters asks to disable a ROD always say yes. Document everything throughout your shift and upload a single e-log entry at the end.

Before beam injection Before and during beam injection the Pixel and SCT need to be in the STANDBY state, ie for Pixel preamplifiers and HV off and for SCT reduced HV (50V instead of 150V) Make sure that neither Pixels nor SCT masks the stable beam flag / "Injection Permits" If SCT is not already in STANDBY, Issue GOTO_STANDBY from the top level SCT node in the FSM. Recover any modules as necessary. If you can't get all branches to be in the STANDBY state, call the SCT DCS on-call expert (162034). Check that Pixel is in STANDBY and gives injection permit. If not please call the pixel on-call (160032).

Before stable beams (i.e. start of SQUEEZE) Open "IP1 Collimator Check" tab in the IDE panel and monitor the beam background, BCM activity, SCT occupancy with ROS monitoring and overall beam rates. You need to do these checks continuously and very closely, and check the trends in the rates. If the rates are too high (indicators are RED) and the trend is increasing, its unlikely we will be ready to switch on when stable beam is declared. Call the Pixel expert in case they are not in the ACR yet. (160032) Check at the end of the squeeze that all collimators are in there right position, i.e. all boxes have to be green.

Warm Start Warm Start happens when the LHC declares Stable Beams at the beginning of a physics fill. There is an audio indication in the ACR when this happens. The silicon detectors need to ramp their voltages in the warm start procedure. The ramping of voltages for Pixels is done by experts so the ID shifter has no responsibility other to than to make sure the appropriate expert is called (well before stable beams is declared) for this procedure. The ramping of voltages for SCT is automated, so long as the background conditions are ok TRT requires no special actions

Warm Start – SCT Special Actions The AUTOMATON widget will automatically switch on the SCT on declaration of stable beams, provided all warm start requirements (collimator positions, background rates, SCT occupancy) are met On declaration of stable beams, watch the SCT voltages switch on, recover (manually switch on) any modules which fail to switch on by navigating down through the SCT FSM (should not be necessary – afterwards log this action so the SCT expert knows) If warm start conditions are not met, AUTOMATON will not grant the automated switch on. Call the SCT run co who may ask you to switch on the SCT manually (GOTO_READY in SCT FSM) after he/she verifies that it is safe to switch on

SCT AUTOMATON state summary DISABLED: not active, manuals switch-on required NO_BEAM: no beam in LHC BEAM_PRESENT: beam in LHC BEAM_CONFIRMED: ID agrees there is beam (background rates etc) BEAM_OK: ID declares beam ok GRANTED: flag set for automated switch on STABLE_BEAM: stable beam declared ON: SCT has successfully ramped ON NOT_CONNECTED: no connection to GCSLHC or IDESCS – call SCT expert on 162034 NOT_VALID: no data – call SCT expert on162034

Warm Stop Warm stop happens when the the Stable Beams Flag is removed or an “LHC handshake” is started. The Stable Beams Flag can be removed for several reasons: After an unscheduled Beam Dump. Variable delay: from immediate to 15 min. When the LHC schedules a beam dump and initiates a Beam Dump handshake. When the LHC attempts a beam adjustment and initiates a Beam Adjustment Handshake These are all indicated on the LHC Page 1 web page, and also associated with an audio indicator in the control room During the Warm Stop the silicon detectors ramp down their voltages. The Ramping down of voltages on the silicon detectors is automatic. No action required. If by accident one or more SCT channels remain ON (never happened before) and prevent handshake completion, switch to stand by manually the individual channels, then call the SCT on-call 162034 The ID Shifters responsibility is to make sure that the silicon detectors end up in their Stand-By State. If there is a problem at any point, then the appropriate On-Call Expert should be called. TRT requires no action at Warm Stop

Warm Stop Both SCT and PIX react to “end of physics” signals automatically, in different ways PIX Action: Switch off HV, turn off pre-amplifier HV on, preamps on = READY HV on, preamps off = ON (Pixel partition NOT_READY) HV off, preamps on = LV_ON (Pixel partition NOT_READY) HV off, preamps off = STANDBY Invoked by start of BEAMDUMP or ADJUST handshake: Invoked by unplanned beam dump Reacts on post mortem signal If stable beam flag is removed immediately → HV interlock Interlock needs to be acknowledged SCT Action: Ramp down HV to 50V (READY → STANDBY) Invoked by start of BEAMDUMP or ADJUST handshake, or removal of stable beam flag

Beam Dump When a Beam Dump Occurs, the BCM RODs Readout state will change to 5. This means that the RODs are reading out the post mortem from their buffer. After 5 minutes if they do not change to a state of 6, call the BCM On-Call: 16-3881. Once the BCM is finished reading out, check the post-mortem histograms in the BCM OHP (at the BCM desk). If the histograms are Green, then everything was good. If they are not Green then call the BCM On-Call: 16-3881.

End of Shift Fill out Shift Summary elog template. Inform the next shift of current system status and any significant events (significant means did you do something that is not normal operation or did you call an expert). If the next shifter does not arrive within 30 minutes after the start of the next shift, call 16-XXXX (reserver shifter), if all else fails call your favorite RC.

ID Monitoring – check list Help or more explanations

OHP Sub detector tabs

DQMD

DQMD SCT Pixel TRT

Desktop Layout Screen 1 Screen 2 Screen 3 Screen 4 Screen 1 Screen 2 IGUI, SA DQMD, OHP ID ALARMS Liquid trend Busy ID FSM Desktop 1 Screen 1 Screen 2 Screen 3 Screen 4 Pixel Alarms SCT Alarms TRT Alarms ID FSM Desktop 2

Desktop Layout II Screen 1 Screen 2 Screen 3 Screen 4 Screen 1 Whiteboard Elog RunCom Prob. DB ID FSM Desktop 3 Screen 1 Screen 2 Screen 3 Screen 4 BCM Alarms PM OHP IDE Alarms ID FSM Desktop 4

Feedback Please follow all the check list and give feedback! Don’t be shy to put all inconsistencies, problems which could not be solved alone, etc into the elog. We need to be prepared also for not experienced shifters!