1. Inner Detector Combined Shift
Tools and Procedures
version 0.9
September 6th, 2011

2. Table Of Contents ID General FSM Panel IDE FSM Panel
ID General Alarm Panel
DAQ Tools
ID Operations & Problems Database
Checklists & Elog
Special Actions: Warm start/stop, beam dump
Online Histogram Presenter
Data Quality Monitoring Display

3. ID General FSM Panel

4. FSM Panel Explained ID FSM Tree FSM Main Panel FSM Help
LHC Handshake Status
Fast Navigation
Alarm Summary
ID FSM Tree
FSM Main Panel
FSM Help
Secondary Panel

5. ID FSM Main Panel ID Env. SCT Pixel TRT BCM/BLM Background Conditions
SCT Automatic
Turn On
Pixel
TRT
BCM/BLM
& Post mortem
Injection permit

6. ID Environment Evaporative Cooling Status Compressor Configuration
Liquid level
Environmental
Cooling Status
Thermal Enclosure
Heater (TEH) Trip
Recovery Button
TEH Status

7. SCT Endcap C Status Barrel Status Endcap A Status
Pix
SCT Infrastructure Status
Snapshot shows normal status in STANDBY
Should ALL be GREEN when SCT is ON

8. Pixel Barrel Layers C-Side Disks A-Side Disks Pixel Infrastructure
Snapshot shows normal status in STANDBY
Should ALL be GREEN when PIX is ON

9. ALL indicators should ALWAYS be green
TRT
ALL indicators should ALWAYS be green
Endcap C
Barrel C
Endcap A
Barrel A
TRT Infrastructure
Gas Gain
Stabilization System
(GSSS) Status

10. Beam Conditions (see later slides for descriptions)
BLM,
RAD Mon
SCT
Auto-ON
Injection
Permit Status
Ready For
Inj. Status
Inj. Permit Mask Configuration
BCM post mortem counter
State 6: normal State 5: PM analysis)

11. Backgrounds Green: OK Red: Too high (or no data) SCT Occupancies Pixel
BCM Rates
Background monitoring
active?
Stable beam
yes/no?

12. Use Tabs to navigate panel
IDEnvironment FSM Reached via IDE button Useful for more detailed look at ID environment & beam conditions, eg in preparation for warm start
Use Tabs to navigate panel

13. IP1 Collimator Check Panel (IDE FSM)
Absorbers
Background rates
Beam pipe pressure
Tertiary collimators
Beam parameters

14. Cleaning Collimator Panel (IDE FSM)

15. Beam Rates Panel (IDE FSM)
LUCID, ZDC
rates
Beam pipe pressure, BCM rates, LHC params
SCT/TRT occupancies

16. Collapse of the separation bumps
Example - Beam Rates
SQUEEZE
Lumi optimization
RAMP
Collapse of the separation bumps
→ collisions

17. IDG Alarm Screen Click On Alarm for Sort According to Severity
details, help, and logging
Sort According to Severity

18. IDG Alarm Help Right click on alarm to see options
Insert Directly to elog
Get Alarm Details
Open Alarm Help (Twiki page)
All alarms shall be logged
Errors and Fatals shall be followed by a call to the expert

19. IDG Alarm Help (Opened by selecting ‘Alarm Help’ in options list)
Alarm explanation along with actions to be taken.
Please report any missing help pages!

20. DAQ Panel
Busy monitor
ATLAS partition
OHP
DQMD

21. DAQ Tools Shifter Assistant
Alerts the shifter to specific DAQ error conditions that require shifter action
Eg ROD busy or disabled etc
Complements IGUI panel indicators
Includes instructions for the shifter
IGUI Panels
Indicates DAQ related alerts and warnings
Provides shifter tools to take the appropriate action
MRS
Warnings/Errors/Fatals from DAQ applications
Eg missing essential files, application crashes etc
Usually require expert actions, but shifter should monitor MRS for ERRORs or FATALs

22. Shifter Assistant Tool
New alert
Expand if new alert
Link is available under the ID Menu at the ID Desk

23. Shifter Assistant Tool
Read the messages and mark them as read
Info & Warnings can be ignored
Errors and Fatals shall be followed by a call to the expert

24. More details in SA

25. ATLAS DAQ IGUI
Loading IGUI Panels

26. IGUI Panels The IDPanel under the ID tab of the TDAQ IGUI provides:
Visible indications of any SCT and TRT DAQ-related alerts or warnings that the ID shifter should react to
In principle, the same alerts/warnings are displayed on the Shifter Assistant
Tools to react to those alerts
The IDPanel should be open and visible at all times on the ID shifter desktop
At the start of the shift, the shifter should open IDPanel (if not already open) by ticking ‘IDShifterGui’ in the list of tabs shown when clicking the ‘Load Panels’ button at the top of the TDAQ IGUI
For info, the IDPanel can be closed by unticking ‘IDShifterGui’ in the same list, but the shifter should leave the panel open

27. Panel Structure
To the left of the panel is an expandable tree indicating the DAQ status for both SCT and TRT
Green: OK, Yellow: WARNING, Red: ALARM
The ‘ID DAQ Status’ icon colour indicates the combined status for both SCT and TRT
The ‘TRT’ and ‘SCT’ icons in the tree indicate the overall status of TRT and SCT respectively
If yellow or red, click on the sub-tree component to see the origin of the warning/alarm and the tools to resolve the problem

28. Overall SCT Control and Status panel, visible by clicking on the ‘SCT’ icon in the tree

29. SCT Control & Status Panel : 1
Control Mode: Enables recovery tools
Only one TDAQ IGUI has control
mode enabled (if someone opens another IGUI and takes control mode, this IGUI will have control mode disabled)
Module Recovery:
Automatic recovery of link errors
Should be enabled by default
If Control Mode is disabled, module recovery gets disabled too.
Global Reconfiguration:
Time between automatic global reconfiguration of the SCT to counteract chip configuration corruption by SEUs
Set to 30minutes by default

30. SCT Control & Status Panel : 2
Disabled ROD count across full SCT
Green: 0
Yellow: 1 (take action!)
Red: >1 (take action!)
Link Error count across full SCT
Green: <50
Yellow: <100 (call expert!)
Red: >100 (take action!)
Busy ROD count across full SCT
Green: 0
Red: >0 (take action!)
If any of these 3 indicators are not green, click on the crate icons which are non-green to perform actions

31. ROD status in first column
(Grayed out slots mean no ROD present in that slot)
Green: ROD is Enabled, and not BUSY

32. Useful to see if 100% busy or not
ROD percent busy
Useful to see if 100% busy or not

33. Link Errors per ROD
Should be ideally 0
If non-zero, module recovery is attempted automatically (if enabled in control&status panel)
Green: <10
Yellow <20
Red >20

34. Visible by clicking on TRT Recovery
TRT Panel
Visible by clicking on TRT Recovery

35. Synchronization:
- If the # of sync problems is > 0, an automatic resync is performed (no shifter action necessary)
- If prompted by the Shifter Assistant or an expert, click the Request Resync button to manually request a resync
8/25/11

36. - If this doesn't work, call the TRT DAQ expert on 160531
ROD Recovery:
- If the # of disabled RODs is > 0, click the Recover Disabled RODs button to attempt to recover these.
- Run Control will have to acknowledge a popup window for this procedure.
- If this doesn't work, call the TRT DAQ expert on
8/25/11

37. 8/25/11 Specific ROD Recovery:
- If instructed by the shifter assistant or an expert, you can force a ROD recovery.
- Type the ROD address into the text box and click Request ROD Recovery.
- Run Control will have to acknowledge a popup window for this procedure.
- If problems occur, call the TRT DAQ expert on
8/25/11

38. MRS Call experts for ERRORs or FATALs Enter subscription string
to filter for
SCT, TRT, PIX and BCM
“SCT|TRT|PIX|BCM”
Call experts for ERRORs or FATALs

39. Not supposed to be an ID shifter tool
Pixel Rod Mon
Not supposed to be an ID shifter tool

40. ID Operations and Problems DB
Web interface opened via ID menu on desktop
(keep it open for easy/quick reference)

41. Searching Prob. DB
Specify
System
Specify
Category

42. Click on DB entry to get details
Searching Prob. DB
Click on DB entry to get details
Enter String
to search DB

43. Details with explanation of DB entry and solution prescribed
Searching Prob. DB
Details with explanation of DB entry and solution prescribed

44. Checklists and Procedures
Elog
Start of Shift
Warm Start
Warm Stop
Beam Dump
During Shift
End of Shift

45. Elog
Maintain a log throughout your shift of what happened, and upload it at the end of the shift.
A new elog interface will be available “real soon” for use by the ID shifter:
- uploads your entries directly as you type them, line by line, and appends them to the shift elog summary
- useful for experts and run coordinators to see the status of your shift in ‘real time’
- will only append (not delete/change previous lines)

46. Start of Shift
Arrive to Shift 15 Minutes before Start of Shift to get summary from previous shifter!
Login to
Elog
DCS ID FSM Panel
DCS Alarm Screens
Run Com Tool (not yet possible)
Introduce yourself to the shift leader (if they haven’t done so already).
Read latest elog messages from last 24 hours. Keep elog regularly updated
Ensure that OHP, DQMD, SA, and IGUI tools are up and running with the proper configurations.
Ensure that individual ID desks are running their proper tools as well.
Check the alarm screen for any alerts.
Check the DAQ status (IGUI ID panel and SA)
Check the status of the pixel infrastructure (pixel igui, checked by SA)
Check the FSM Status (for silicon should be READY/OK during data taking, otherwise in STANDBY/OK, TRT should always be READY/OK).

47. During Shift
Monitor and acknowledge any alerts from the SA tool, and take action as needed
Monitor the IGUI ID panel and take action as needed
Log any alarms in the FSM. Acknowledge them once they have been logged and WENT. Take any necessary actions needed (use Alarm Help and/or Problems DB tool).
Check the trend in evaporative cool plant liquid level (~ 1kg / 8 hours).
Check ROS occupancy is non-zero in FSM. If it is zero and beam is present, then ask the Run Control Shifter to restart the SCT ROS Monitoring.
During data taking, check hourly
DQ checklists
Number of disabled pixel modules, especially b-layer (OHP)
Check Pixel Module Errors (i.e. LVL1ID errors) (Pixel Rod Mon / DQMD)

48. Start of Run
It is important to be specially vigilant at the beginning of a run. Problems most likely occur at this point. Be aware of:
FATAL / ERROR messages from the Pixels, SCT, TRT, or BCM appearing in the MRS
Check BUSY status using the BUSY status display. If no triggers are coming through one of the systems could be busy.
Check the IGUI ID panel to ensure all is green
Stop-less removal could happen at this stage, the run control shifter should tell you if an ID System goes busy and may disable one or more of the RODs. Follow the instructions in the operations database if this happens.
Ensure that all the Pixel RODs are allocated by the Central DAQ. If not an ERROR should appear in the SA

49. During a run
Keep checking the SA and IGUI ID panel. If the run control shifter did not tell you he/she disabled an SCT ROD, you would see a big increase in link errors.
Check SCT ROS Monitoring every hour. Look for errors and anomalous values in the results
Continue to check for errors in MRS
Watch out for busy of any of the detectors. If the run control shifters asks to disable a ROD always say yes.
Document everything throughout your shift and upload a single e-log entry at the end.

50. Before beam injection
Before and during beam injection the Pixel and SCT need to be in the STANDBY state, ie for Pixel preamplifiers and HV off and for SCT reduced HV (50V instead of 150V)
Make sure that neither Pixels nor SCT masks the stable beam flag / "Injection Permits"
If SCT is not already in STANDBY, Issue GOTO_STANDBY from the top level SCT node in the FSM. Recover any modules as necessary. If you can't get all branches to be in the STANDBY state, call the SCT DCS on-call expert (162034).
Check that Pixel is in STANDBY and gives injection permit. If not please call the pixel on-call (160032).

51. Before stable beams (i.e. start of SQUEEZE)
Open "IP1 Collimator Check" tab in the IDE panel and monitor the beam background, BCM activity, SCT occupancy with ROS monitoring and overall beam rates. You need to do these checks continuously and very closely, and check the trends in the rates. If the rates are too high (indicators are RED) and the trend is increasing, its unlikely we will be ready to switch on when stable beam is declared.
Call the Pixel expert in case they are not in the ACR yet. (160032)
Check at the end of the squeeze that all collimators are in there right position, i.e. all boxes have to be green.

52. Warm Start
Warm Start happens when the LHC declares Stable Beams at the beginning of a physics fill. There is an audio indication in the ACR when this happens. The silicon detectors need to ramp their voltages in the warm start procedure.
The ramping of voltages for Pixels is done by experts so the ID shifter has no responsibility other to than to make sure the appropriate expert is called (well before stable beams is declared) for this procedure.
The ramping of voltages for SCT is automated, so long as the background conditions are ok
TRT requires no special actions

53. Warm Start – SCT Special Actions
The AUTOMATON widget will automatically switch on the SCT on declaration of stable beams, provided all warm start requirements (collimator positions, background rates, SCT occupancy) are met
On declaration of stable beams, watch the SCT voltages switch on, recover (manually switch on) any modules which fail to switch on by navigating down through the SCT FSM (should not be necessary – afterwards log this action so the SCT expert knows)
If warm start conditions are not met, AUTOMATON will not grant the automated switch on. Call the SCT run co who may ask you to switch on the SCT manually (GOTO_READY in SCT FSM) after he/she verifies that it is safe to switch on

54. SCT AUTOMATON state summary
DISABLED: not active, manuals switch-on required
NO_BEAM: no beam in LHC
BEAM_PRESENT: beam in LHC
BEAM_CONFIRMED: ID agrees there is beam (background rates etc)
BEAM_OK: ID declares beam ok
GRANTED: flag set for automated switch on
STABLE_BEAM: stable beam declared
ON: SCT has successfully ramped ON
NOT_CONNECTED: no connection to GCSLHC or IDESCS – call SCT expert on
NOT_VALID: no data – call SCT expert on162034

55. Warm Stop
Warm stop happens when the the Stable Beams Flag is removed or an “LHC handshake” is started.
The Stable Beams Flag can be removed for several reasons:
After an unscheduled Beam Dump. Variable delay: from immediate to 15 min.
When the LHC schedules a beam dump and initiates a Beam Dump handshake.
When the LHC attempts a beam adjustment and initiates a Beam Adjustment Handshake
These are all indicated on the LHC Page 1 web page, and also associated with an audio indicator in the control room
During the Warm Stop the silicon detectors ramp down their voltages.
The Ramping down of voltages on the silicon detectors is automatic. No action required. If by accident one or more SCT channels remain ON (never happened before) and prevent handshake completion, switch to stand by manually the individual channels, then call the SCT on-call
The ID Shifters responsibility is to make sure that the silicon detectors end up in their Stand-By State. If there is a problem at any point, then the appropriate On-Call Expert should be called.
TRT requires no action at Warm Stop

56. Warm Stop
Both SCT and PIX react to “end of physics” signals automatically, in different ways
PIX
Action: Switch off HV, turn off pre-amplifier
HV on, preamps on = READY
HV on, preamps off = ON (Pixel partition NOT_READY)
HV off, preamps on = LV_ON (Pixel partition NOT_READY)
HV off, preamps off = STANDBY
Invoked by start of BEAMDUMP or ADJUST handshake:
Invoked by unplanned beam dump
Reacts on post mortem signal
If stable beam flag is removed immediately → HV interlock
Interlock needs to be acknowledged
SCT
Action: Ramp down HV to 50V (READY → STANDBY)
Invoked by start of BEAMDUMP or ADJUST handshake, or removal of stable beam flag

57. Beam Dump
When a Beam Dump Occurs, the BCM RODs Readout state will change to 5. This means that the RODs are reading out the post mortem from their buffer.
After 5 minutes if they do not change to a state of 6, call the BCM On-Call:
Once the BCM is finished reading out, check the post-mortem histograms in the BCM OHP (at the BCM desk). If the histograms are Green, then everything was good. If they are not Green then call the BCM On-Call:

58. End of Shift Fill out Shift Summary elog template.
Inform the next shift of current system status and any significant events (significant means did you do something that is not normal operation or did you call an expert).
If the next shifter does not arrive within 30 minutes after the start of the next shift, call 16-XXXX (reserver shifter), if all else fails call your favorite RC.

59. ID Monitoring – check list
Help or more explanations

60. OHP
Sub detector tabs

61. DQMD

62. DQMD
SCT
Pixel
TRT

63. Desktop Layout Screen 1 Screen 2 Screen 3 Screen 4 Screen 1 Screen 2
IGUI, SA
DQMD, OHP
ID ALARMS
Liquid trend
Busy
ID FSM
Desktop 1
Screen 1
Screen 2
Screen 3
Screen 4
Pixel Alarms
SCT Alarms
TRT Alarms
ID FSM
Desktop 2

64. Desktop Layout II Screen 1 Screen 2 Screen 3 Screen 4 Screen 1
Whiteboard
Elog
RunCom
Prob. DB
ID FSM
Desktop 3
Screen 1
Screen 2
Screen 3
Screen 4
BCM Alarms
PM OHP
IDE Alarms
ID FSM
Desktop 4

65. Feedback Please follow all the check list and give feedback!
Don’t be shy to put all inconsistencies, problems which could not be solved alone, etc into the elog.
We need to be prepared also for not experienced shifters!