Run period 2010 Updated on 05.01.2010 C. Garabatos and C. Lippmann TPC shift guide Run period 2010 Updated on 05.01.2010 C. Garabatos and C. Lippmann

Foreword This shift guide is intended for the TPC shifter to be able to operate and monitor the TPC: Identify the relevant machines: aldaqacr34 and aldaqacr33, the Linux machines in the Working Room (pw=TPC.501) alitpcon001, the Windows DCS Operator Node (Use your NICE account) The shifter should know: How to monitor the TPC and react to alarms How to take Pedestal, Pulser and Laser (PPL) runs How to prepare the TPC for a global run How to use the online monitor to look at events and plots How to use AMORE to check the quality of the data being taken How to use the electronics logbook 18-Dec-09 TPC Shift Guide

Shifters tasks Verify that the TPC is in safe and operational state Check for alarms at least every 30 minutes Monitor data quality and DA output with AMORE Keep track of what happens Add entries to electronic logbook for runs longer than 10 minutes Add shift summary to electronic logbook Study this manual to have the overview Know how to take calibration runs (PPL) Know how to prepare the TPC for a global run 18-Dec-09 TPC Shift Guide

Data taking with beams Pre-Cycling: LHC magnets are prepared. No beam (~15 minutes) Injection warning If needed, DCS shifter ramps down TPC ROC HV (Transition from READY to BEAM_TUNING) Injection of 450GeV/c protons (~15 minutes) Ramping up to nominal energy (~30 minutes) Preparation for physics Squeeze: Make interaction points smaller Put beams to collision Adjust collimators to minimise background Declaration of stable beams TPC ROC HV is ramped up (Transition from BEAM_TUNING to READY) by the TPC shifter on request of the SRC Data taking Data taking continues for many hours until beams are dumped ALICE can be UNSAFE TPC can be READY ALICE must be SAFE TPC must be BEAM_TUNING Data taking: ALICE is UNSAFE Data taking: TPC must be READY 18-Dec-09 TPC Shift Guide

Tips In PVSS: Do not click too fast. Wait for the system to respond If a monitor (screen) of aldaqacr3x becomes black: Press “Alt+Ctrl+F6”, wait a second, then press “Alt+Ctrl+F7” If you cannot type in a PVSS panel, go to a linux terminal, type something there, and try again There are two kinds of locks: DCA (ECS/DAQ) and DCS. Don’t confuse them! Keep each application in its assigned desktop, in order to avoid confusion 18-Dec-09 TPC Shift Guide

PART I: DCS (PVSS, Alarms and FSM)

Connecting to the Operator Node Make sure there is a remote desktop session to alitpcon001; if there is not one, a registered user does: Open remote desktop connection Enter your NICE username and password The TPC DCS control panel (a GUI) opens automatically (see next slide) Click on the key icon (top left) to login as tpc (pw=tpc): The TPC DCS control panel is composed of: The Alarm panel PVSS: Display and control of operational parameters of the TPC FSM (Finite State Machine): A tree structure of the various states of the TPC This is the command in case the shortcut does not exist: rdesktop -g1280x1024 - a16 -u <your_username> 18-Dec-09 TPC Shift Guide

The PVSS Main Panel Log in Lock To restart FSM Alarm panel Click to open FSM control panel FSM tree HV State and Control: BEAM_TUNING READY The PVSS Main Panel Overview Status of the communication to different control systems

Alarm Panel Watch the alarm panel! Click on the Alarm symbol: You see the list of recent alarms: warnings , errors and fatals Right-click on active alarms and choose ‘Alarm help’ to see descriptions and actions to be taken After the action, some alarms have to be acknowledged by clicking on ‘!!!‘ 18-Dec-09 TPC Shift Guide

Alarm Types WARNING no ACK Alarms are defined Watch Warning ACK Working hours ERROR ACK One hour Fatal Without delay Alarms are defined according to the reaction time 18-Dec-09 TPC Shift Guide

FSM Control Panel To open the FSM control panel: Right-click on TPC_DCS in FSM tree and select ‘view panel’ Click Check the lock Closed and green: you have the control. Release the lock only if you need to close the session Closed and red: someone else has the control Grey and open: No one has the control Note: In order to take data, someone must have the lock Closed, green Lock Note: If you need to release the lock, click on it and use ’Release’ 18-Dec-09 TPC Shift Guide

FEE Configuration In the FSM control panel of the TPC_DCS click on the state, choose CONFIGURE and send the run type (PHYSICS, LASER, ...) Wait for the state of TPC_DCS to return to READY. This may take 20 to 30 seconds. 18-Dec-09 TPC Shift Guide

Excluding sectors or partitions In case of problems, partitions have to be excluded from data taking This is in general done ONLY in the DAQ How to do it: See Part II: Taking data (DCA and DAQ) Only if the partition has a severe problem (for example overheating) it should be switched off (GO_IDLE) and excluded from the FSM control How this is done is shown on the next slides 18-Dec-09 TPC Shift Guide

Excluding a sector from FSM control Open the TPC_FERO node in the FSM tree in the main panel Open the FSM control panel Select ‘Exclude& LockOut’ for the relevant sector In this example we want to exclude sector A03 18-Dec-09 TPC Shift Guide

Excluding a partition from FSM control Open the TPC_ESC node in the FSM tree in the main panel Open the FSM control panel for the relevant FED node Disable the relevant readout partition In this example we want to exclude partition 1 in sector A07. Partition 0 is already excluded. 18-Dec-09 TPC Shift Guide

Cooling panel Watch the alarm panel! TPC_COOL (in Operator Node, screenshot next page) FEC Cooling plant must be RUNNING Resistor Rod Cooling plant must be RUNNING In case of problems (STANDBY or STOPPED) call expert Note: if the cooling goes to standby, a software interlock will turn off the FEE after 2 min, and the LV power supplies (the RCUs) after 7 min. A hardware interlock shuts off everything after 10 minutes. Note: There is also a Cooling tab in the FERO panel, showing the temperatures and pressures of the FEE cooling circuits 18-Dec-09 TPC Shift Guide

FERO panel In TPC_FERO you can select different monitors: FED MON: shows by default the temperature of the FEE The tab ‘FSM State’ shows you the state of all readout partitions Other tabs show currents and voltages BUSY BOX: monitor and control of the busy box LV MON: shows by default the digital currents, which should be 2Amp (FEE off) to 130Amp (all FEE on) COOL PLANT: shows the status of the cooling it should be OK and RUNNING (all green) COOL LINES: shows the temperatures in all FEE cooling circuits All these panels will be shown on the next slides 18-Dec-09 TPC Shift Guide

Partition with no control. This is not good. Watch for Alarms! Readout partitions rp0 to rp5 Status of the communication software chain

One sector with lower current, since one out of six partitions is off

This will show BUSY or OK One disabled partition, all others are ready for a run In case of BUSY: Click here to get more details

HV Monitoring and Control Watch the alarm panel! Drift HV: voltage 100 kV and normal current 364A Readout Chamber (ROC) HV: currents should be < 1mA There are two states of the ROC HV: READY: full voltage on the ROCs BEAM_TUNING: safe, reduced voltages for beam injections Actions: In case of unstable beams in the LHC, the ROC HV must be changed to BEAM_TUNING by the TPC or the DCS shifter Upon request of the SRC, the TPC shifter changes the ROC HV to READY (in the TPC_ROC) main panel Any other operations on HV are done ONLY by experts 18-Dec-09 TPC Shift Guide

OROC IROC ROC High Voltage Panel

Drift High Voltage Panel

Gas Watch the alarm panel! TPC_GAS (Operator Node) Status must be CONNECTED and OK Mixer, Distribution, Pump and Exhaust modules should be RUN or RECIRCULATING; if STOP call expert 18-Dec-09 TPC Shift Guide

Part II: Taking data (DCA and DAQ)

The DCA The TPC Detector Control Agent (DCA) controls all what is necessary to take standalone data (DAQ, DCS, trigger and HLT) Closed, green lock: You have control of the DCA Click here to start a run Note: If the lock is red and nobody seems to have it (check with DAQ shifter), make sure there isn't a second, hidden DCA window! This one might have the lock! 18-Dec-09 TPC Shift Guide

Taking data using the DCA For PPL data taking (calibration runs PEDESTAL, PULSER, LASER) use the DCA (see next slides): Click Options and select ‘Ignore ACT’ Get the lock (ask DAQ shifter to release, if necessary) Do you want to save the raw data? Select the LDC mode of operation (Local recording on/off) Select the GDC mode of operation ‘Event Building Off’  data are not saved, monitoring of full TPC events not possible ‘No Recording’  events are built for the online monitor ‘mStream Recording’  events are built for the online monitor and data are saved Select the HLT mode (Usually HLT unlocked and mode A) Start the run by selecting the run type (PEDESTAL, PULSER, LASER, COSMIC or STANDALONE) PEDESTAL, PULSER and LASER runs will stop automatically after 100 or 1000 events. STANDALONE (and COSMIC) runs have to be stopped manually 18-Dec-09 TPC Shift Guide

Exclude partitions from Readout Each of the 216 TPC partitions are called ‘equipment’ in the Data Acquisition System (DAQ) Find the ‘Alice DAQ’ window Click on ‘select equipment’ Click on the green button(s) to disable the equipment (see next slide) Remember to commit the changes Example: A09 partition 1

How to take a Pedestal run In the FSM Send CONFIGURE command with run_type=PEDESTAL to the FSM In the DCA Get the DCA lock Click Options and select ‘Ignore ACT’ Set the GDC to ‘Event building OFF’ Local recording on the LDCs is activated automatically Select HLT mode A and release the HLT lock Start the run by selecting PEDESTAL_RUN The run stops automatically after 100 events The Detector Algorithm (DA) then starts on the LDCs (see the progress on the InfoBrowser, may take 90 seconds) When the DA is finished, look at the output using AMORE 18-Dec-09 TPC Shift Guide

How to take a Pulser run In the FSM In the DCA Send CONFIGURE command with run_type=PULSER to the FSM In the DCA Get the DCA lock Click Options and select ‘Ignore ACT’ Set the GDC to ‘No Recording’ In this way we can use the online monitor to look at events from the data flow Local recording on the LDCs is activated automatically Select HLT mode A and release HLT lock Start the run by selecting PULSER_RUN This will automatically activate the calibration pulsers The run stops automatically after 1000 events The pulsers are disabled automatically The Detector Algorithm (DA) starts on the LDCs (see the progress on the InfoBrowser, may take 30 seconds) When the DA is finished, look at the output using AMORE 18-Dec-09 TPC Shift Guide

How to take a Laser run In the FSM In the DCA Send CONFIGURE command with run_type=LASER to the FSM In the DCA Get the DCA lock Click Options and select ‘Ignore ACT’ Set the GDC to ‘mStream recording’ Also the raw data should be recorded Set the LDCs to ‘Local recording OFF’ Select HLT mode A and release HLT lock Start the run by selecting LASER_RUN This will automatically activate the laser system The run stops automatically after 1000 events The laser system will be stopped automatically The Detector Algorithm (DA) starts on a monitoring machine (see the progress on the InfoBrowser) When the DA is finished, look at the output using AMORE 18-Dec-09 TPC Shift Guide

How to prepare for a global run In DCA Click ‘release resources’ Release the lock of the DCA In FSM Send CONFIGURE command with run_type=PHYSICS to the FSM Make sure someone has the lock of the FSM control (you or DCS shifter) Check for alarms The state of the TPC_DCS should be READY (TPC_ROC ON) COSMIC runs: If in BEAM_TUNING, go to READY on request of the SRC LHC runs: Go from BEAM_TUNING to READY ONLY in case of STABLE BEAMS and ONLY on request of the SRC Inform the Shift Leader: ‘The TPC is ready’ 18-Dec-09 TPC Shift Guide

Online Monitor Click on Online Monitor Select File/Stream and choose LDC or GDC to be used. Examples: mem://@aldaqpc200: mem://@gdc-GLOBAL-01: (Don’t forget the colon ”:”) Monitoring has to be enabled on that machine press Next Event button You can look at pad signals from individual sectors If the OM crashes or is hanging: open a terminal, ssh to aldaqdqm04 killall root root.exe 18-Dec-09 TPC Shift Guide

How to run AMORE (1) Click on a window appears (see next slide): start the update (left bottom corner) choose what you want to view 18-Dec-09 TPC Shift Guide

How to run AMORE (2) 18-Dec-09 TPC Shift Guide

HLT online display Only on aldaqacr33 and only when HLT mode is not A 18-Dec-09 TPC Shift Guide

Logbook Use shortcut to start browser with logbook Log in using NICE account do you have write privileges? For each run add an entry with the run conditions: missing partitions HV settings problems other useful information You may attach a screenshot (event display, ...) Add a shift summary at the End Of your Shift Add Log Entry (Type=General, Subsystem=TPC, Title: TPC EOS dd.mm.yy) Click ‘Submit’ twice! 18-Dec-09 TPC Shift Guide

List of experts DCS, Busy Box FEE, DAQ, Trigger HV Gas Cooling Laser C. Lippmann 162430, U. Frankenfeld 164048 FEE, DAQ, Trigger C. Lippmann 162430, L. Musa 163119 HV D. Vranic 163691, C. Garabatos 165327 Gas C. Garabatos 165327, Gas piquet 162516 Cooling C. Garabatos 165327, Cooling piquet 72201 Laser C. Garabatos 165327, B. Nielsen 164221 Amore, Online Monitor C. Lippmann 162430 18-Dec-09 TPC Shift Guide

More information 18-Dec-09 TPC Shift Guide

FSM Troubleshooting in case of problems with FSM (careful, nut during a run!) if the lock is taken but nobody seems to have it (careful!) On the PVSS main panel click on and choose ‘FSM’ Choose ”STOP ALL” and wait for activity in status boxes on lower part of main panel to stop (patience!) Click ”KILL FSM” and wait 5 seconds Click ”START/RESTART ALL” All status boxes at the bottom of the main panel should turn green again 18-Dec-09 TPC Shift Guide

Busy Box Troubleshooting In case you suspect that the busy box does not work properly, you can try to reprogram it: Click here type: cd /mnt/dcbrw/busy/busybox_files ./program Also reconfigure it from he PVSS main panel (TPC_FERO  BUSY BOX) Click ‘Reconfigure’ 18-Dec-09 TPC Shift Guide

How to change L2_DELAY Click Stdfuncs and open “VME r/w” Click “CODE_ADD” and choose L2_DELAY Click on “read”. The default value is 0xf70 Click on “write” to write new value: 0x1040 Read again to verify that it was changed 18-Dec-09 TPC Shift Guide