1. Run period 2010 Updated on 05.01.2010 C. Garabatos and C. Lippmann
TPC shift guide
Run period 2010
Updated on
C. Garabatos and C. Lippmann

2. Foreword
This shift guide is intended for the TPC shifter to be able to operate and monitor the TPC:
Identify the relevant machines:
aldaqacr34 and aldaqacr33, the Linux machines in the Working Room (pw=TPC.501)
alitpcon001, the Windows DCS Operator Node (Use your NICE account)
The shifter should know:
How to monitor the TPC and react to alarms
How to take Pedestal, Pulser and Laser (PPL) runs
How to prepare the TPC for a global run
How to use the online monitor to look at events and plots
How to use AMORE to check the quality of the data being taken
How to use the electronics logbook
18-Dec-09
TPC Shift Guide

3. Shifters tasks Verify that the TPC is in safe and operational state
Check for alarms at least every 30 minutes
Monitor data quality and DA output with AMORE
Keep track of what happens
Add entries to electronic logbook for runs longer than 10 minutes
Add shift summary to electronic logbook
Study this manual to have the overview
Know how to take calibration runs (PPL)
Know how to prepare the TPC for a global run
18-Dec-09
TPC Shift Guide

4. Data taking with beams
Pre-Cycling: LHC magnets are prepared. No beam (~15 minutes)
Injection warning
If needed, DCS shifter ramps down TPC ROC HV (Transition from READY to BEAM_TUNING)
Injection of 450GeV/c protons (~15 minutes)
Ramping up to nominal energy (~30 minutes)
Preparation for physics
Squeeze: Make interaction points smaller
Put beams to collision
Adjust collimators to minimise background
Declaration of stable beams
TPC ROC HV is ramped up (Transition from BEAM_TUNING to READY) by the TPC shifter on request of the SRC
Data taking
Data taking continues for many hours until beams are dumped
ALICE can be UNSAFE
TPC can be READY
ALICE must be SAFE
TPC must be BEAM_TUNING
Data taking: ALICE is UNSAFE
Data taking: TPC must be READY
18-Dec-09
TPC Shift Guide

5. Tips In PVSS: Do not click too fast. Wait for the system to respond
If a monitor (screen) of aldaqacr3x becomes black: Press “Alt+Ctrl+F6”, wait a second, then press “Alt+Ctrl+F7”
If you cannot type in a PVSS panel, go to a linux terminal, type something there, and try again
There are two kinds of locks: DCA (ECS/DAQ) and DCS. Don’t confuse them!
Keep each application in its assigned desktop, in order to avoid confusion
18-Dec-09
TPC Shift Guide

6. PART I: DCS (PVSS, Alarms and FSM)

7. Connecting to the Operator Node
Make sure there is a remote desktop session to alitpcon001; if there is not one, a registered user does:
Open remote desktop connection
Enter your NICE username and password
The TPC DCS control panel (a GUI) opens automatically (see next slide)
Click on the key icon (top left) to login as tpc (pw=tpc):
The TPC DCS control panel is composed of:
The Alarm panel
PVSS: Display and control of operational parameters of the TPC
FSM (Finite State Machine): A tree structure of the various states of the TPC
This is the command in case the shortcut does not exist: rdesktop -g1280x a16 -u <your_username>
18-Dec-09
TPC Shift Guide

8. The PVSS Main Panel Log in Lock To restart FSM Alarm panel
Click to open FSM control panel
FSM tree
HV State and Control: BEAM_TUNING READY
The PVSS Main Panel
Overview
Status of the communication to different control systems

9. Alarm Panel Watch the alarm panel! Click on the Alarm symbol:
You see the list of recent alarms: warnings , errors and fatals
Right-click on active alarms and choose ‘Alarm help’ to see descriptions and actions to be taken
After the action, some alarms have to be acknowledged by clicking on ‘!!!‘
18-Dec-09
TPC Shift Guide

10. Alarm Types WARNING no ACK Alarms are defined
Watch
Warning ACK
Working hours
ERROR ACK
One hour
Fatal
Without delay
Alarms are defined
according to the reaction time
18-Dec-09
TPC Shift Guide

11. FSM Control Panel
To open the FSM control panel:
Right-click on TPC_DCS in FSM tree and select ‘view panel’
Click
Check the lock
Closed and green: you have the control. Release the lock only if you need to close the session
Closed and red: someone else has the control
Grey and open: No one has the control
Note: In order to take data, someone must have the lock
Closed, green Lock
Note: If you need to release the lock, click on it and use ’Release’
18-Dec-09
TPC Shift Guide

12. FEE Configuration
In the FSM control panel of the TPC_DCS click on the state, choose CONFIGURE and send the run type (PHYSICS, LASER, ...)
Wait for the state of TPC_DCS to return to READY. This may take 20 to 30 seconds.
18-Dec-09
TPC Shift Guide

13. Excluding sectors or partitions
In case of problems, partitions have to be excluded from data taking
This is in general done ONLY in the DAQ
How to do it: See Part II: Taking data (DCA and DAQ)
Only if the partition has a severe problem (for example overheating) it should be switched off (GO_IDLE) and excluded from the FSM control
How this is done is shown on the next slides
18-Dec-09
TPC Shift Guide

14. Excluding a sector from FSM control
Open the TPC_FERO node in the FSM tree in the main panel
Open the FSM control panel
Select ‘Exclude& LockOut’ for the relevant sector
In this example we want to exclude sector A03
18-Dec-09
TPC Shift Guide

15. Excluding a partition from FSM control
Open the TPC_ESC node in the FSM tree in the main panel
Open the FSM control panel for the relevant FED node
Disable the relevant readout partition
In this example we want to exclude partition 1 in sector A07. Partition 0 is already excluded.
18-Dec-09
TPC Shift Guide

16. Cooling panel Watch the alarm panel!
TPC_COOL (in Operator Node, screenshot next page)
FEC Cooling plant must be RUNNING
Resistor Rod Cooling plant must be RUNNING
In case of problems (STANDBY or STOPPED) call expert
Note: if the cooling goes to standby, a software interlock will turn off the FEE after 2 min, and the LV power supplies (the RCUs) after 7 min. A hardware interlock shuts off everything after 10 minutes.
Note: There is also a Cooling tab in the FERO panel, showing the temperatures and pressures of the FEE cooling circuits
18-Dec-09
TPC Shift Guide

18. FERO panel In TPC_FERO you can select different monitors:
FED MON: shows by default the temperature of the FEE
The tab ‘FSM State’ shows you the state of all readout partitions
Other tabs show currents and voltages
BUSY BOX: monitor and control of the busy box
LV MON: shows by default the digital currents, which should be 2Amp (FEE off) to 130Amp (all FEE on)
COOL PLANT: shows the status of the cooling
it should be OK and RUNNING (all green)
COOL LINES: shows the temperatures in all FEE cooling circuits
All these panels will be shown on the next slides
18-Dec-09
TPC Shift Guide

19. Partition with no control. This is not good. Watch for Alarms!
Readout partitions rp0 to rp5
Status of the communication software chain

20. One sector with lower current, since one out of six partitions is off

22. This will show BUSY or OK
One disabled partition,
all others are ready for a run
In case of BUSY: Click here to get more details

23. HV Monitoring and Control
Watch the alarm panel!
Drift HV: voltage 100 kV and normal current 364A
Readout Chamber (ROC) HV: currents should be < 1mA
There are two states of the ROC HV:
READY: full voltage on the ROCs
BEAM_TUNING: safe, reduced voltages for beam injections
Actions:
In case of unstable beams in the LHC, the ROC HV must be changed to BEAM_TUNING by the TPC or the DCS shifter
Upon request of the SRC, the TPC shifter changes the ROC HV to READY (in the TPC_ROC) main panel
Any other operations on HV are done ONLY by experts
18-Dec-09
TPC Shift Guide

24. OROC
IROC
ROC High Voltage Panel

25. Drift High Voltage Panel

26. Gas Watch the alarm panel! TPC_GAS (Operator Node)
Status must be CONNECTED and OK
Mixer, Distribution, Pump and Exhaust modules should be RUN or RECIRCULATING; if STOP call expert
18-Dec-09
TPC Shift Guide

27. Part II: Taking data (DCA and DAQ)

28. The DCA
The TPC Detector Control Agent (DCA) controls all what is necessary to take standalone data (DAQ, DCS, trigger and HLT)
Closed, green lock: You have control of the DCA
Click here to start a run
Note: If the lock is red and nobody seems to have it (check with DAQ shifter), make sure there isn't a second, hidden DCA window! This one might have the lock!
18-Dec-09
TPC Shift Guide

29. Taking data using the DCA
For PPL data taking (calibration runs PEDESTAL, PULSER, LASER) use the DCA (see next slides):
Click Options and select ‘Ignore ACT’
Get the lock (ask DAQ shifter to release, if necessary)
Do you want to save the raw data?
Select the LDC mode of operation (Local recording on/off)
Select the GDC mode of operation
‘Event Building Off’  data are not saved, monitoring of full TPC events not possible
‘No Recording’  events are built for the online monitor
‘mStream Recording’  events are built for the online monitor and data are saved
Select the HLT mode (Usually HLT unlocked and mode A)
Start the run by selecting the run type (PEDESTAL, PULSER, LASER, COSMIC or STANDALONE)
PEDESTAL, PULSER and LASER runs will stop automatically after 100 or 1000 events. STANDALONE (and COSMIC) runs have to be stopped manually
18-Dec-09
TPC Shift Guide

30. Exclude partitions from Readout
Each of the 216 TPC partitions are called ‘equipment’ in the Data Acquisition System (DAQ)
Find the ‘Alice DAQ’ window
Click on ‘select equipment’
Click on the green button(s) to disable the equipment (see next slide)
Remember to commit the changes
Example: A09 partition 1

31. How to take a Pedestal run
In the FSM
Send CONFIGURE command with run_type=PEDESTAL to the FSM
In the DCA
Get the DCA lock
Click Options and select ‘Ignore ACT’
Set the GDC to ‘Event building OFF’
Local recording on the LDCs is activated automatically
Select HLT mode A and release the HLT lock
Start the run by selecting PEDESTAL_RUN
The run stops automatically after 100 events
The Detector Algorithm (DA) then starts on the LDCs (see the progress on the InfoBrowser, may take 90 seconds)
When the DA is finished, look at the output using AMORE
18-Dec-09
TPC Shift Guide

32. How to take a Pulser run In the FSM In the DCA
Send CONFIGURE command with run_type=PULSER to the FSM
In the DCA
Get the DCA lock
Click Options and select ‘Ignore ACT’
Set the GDC to ‘No Recording’
In this way we can use the online monitor to look at events from the data flow
Local recording on the LDCs is activated automatically
Select HLT mode A and release HLT lock
Start the run by selecting PULSER_RUN
This will automatically activate the calibration pulsers
The run stops automatically after 1000 events
The pulsers are disabled automatically
The Detector Algorithm (DA) starts on the LDCs (see the progress on the InfoBrowser, may take 30 seconds)
When the DA is finished, look at the output using AMORE
18-Dec-09
TPC Shift Guide

33. How to take a Laser run In the FSM In the DCA
Send CONFIGURE command with run_type=LASER to the FSM
In the DCA
Get the DCA lock
Click Options and select ‘Ignore ACT’
Set the GDC to ‘mStream recording’
Also the raw data should be recorded
Set the LDCs to ‘Local recording OFF’
Select HLT mode A and release HLT lock
Start the run by selecting LASER_RUN
This will automatically activate the laser system
The run stops automatically after 1000 events
The laser system will be stopped automatically
The Detector Algorithm (DA) starts on a monitoring machine (see the progress on the InfoBrowser)
When the DA is finished, look at the output using AMORE
18-Dec-09
TPC Shift Guide

34. How to prepare for a global run
In DCA
Click ‘release resources’
Release the lock of the DCA
In FSM
Send CONFIGURE command with run_type=PHYSICS to the FSM
Make sure someone has the lock of the FSM control (you or DCS shifter)
Check for alarms
The state of the TPC_DCS should be READY (TPC_ROC ON)
COSMIC runs: If in BEAM_TUNING, go to READY on request of the SRC
LHC runs: Go from BEAM_TUNING to READY ONLY in case of STABLE BEAMS and ONLY on request of the SRC
Inform the Shift Leader: ‘The TPC is ready’
18-Dec-09
TPC Shift Guide

35. Online Monitor Click on Online Monitor
Select File/Stream and choose LDC or GDC to be used. Examples:
(Don’t forget the colon ”:”)
Monitoring has to be enabled on that machine
press Next Event button
You can look at pad signals from individual sectors
If the OM crashes or is hanging:
open a terminal,
ssh to aldaqdqm04
killall root root.exe
18-Dec-09
TPC Shift Guide

36. How to run AMORE (1) Click on a window appears (see next slide):
start the update (left bottom corner)
choose what you want to view
18-Dec-09
TPC Shift Guide

37. How to run AMORE (2)
18-Dec-09
TPC Shift Guide

38. HLT online display Only on aldaqacr33 and only when HLT mode is not A
18-Dec-09
TPC Shift Guide

39. Logbook Use shortcut to start browser with logbook
Log in using NICE account
do you have write privileges?
For each run add an entry with the run conditions:
missing partitions
HV settings
problems
other useful information
You may attach a screenshot (event display, ...)
Add a shift summary at the End Of your Shift
Add Log Entry (Type=General, Subsystem=TPC, Title: TPC EOS dd.mm.yy)
Click ‘Submit’ twice!
18-Dec-09
TPC Shift Guide

40. List of experts DCS, Busy Box FEE, DAQ, Trigger HV Gas Cooling Laser
C. Lippmann , U. Frankenfeld
FEE, DAQ, Trigger
C. Lippmann , L. Musa HV
D. Vranic , C. Garabatos
Gas
C. Garabatos , Gas piquet
Cooling
C. Garabatos , Cooling piquet 72201
Laser
C. Garabatos , B. Nielsen
Amore, Online Monitor
C. Lippmann
18-Dec-09
TPC Shift Guide

41. More information
18-Dec-09
TPC Shift Guide

42. FSM Troubleshooting
in case of problems with FSM (careful, nut during a run!)
if the lock is taken but nobody seems to have it (careful!)
On the PVSS main panel click on and choose ‘FSM’
Choose ”STOP ALL” and wait for activity in status boxes on lower part of main panel to stop (patience!)
Click ”KILL FSM” and wait 5 seconds
Click ”START/RESTART ALL”
All status boxes at the bottom of the main panel should turn green again
18-Dec-09
TPC Shift Guide

43. Busy Box Troubleshooting
In case you suspect that the busy box
does not work properly, you can try to
reprogram it:
Click here
type:
cd /mnt/dcbrw/busy/busybox_files
./program
Also reconfigure it from he PVSS main panel (TPC_FERO  BUSY BOX)
Click ‘Reconfigure’
18-Dec-09
TPC Shift Guide

44. How to change L2_DELAY Click Stdfuncs and open “VME r/w”
Click “CODE_ADD” and choose L2_DELAY
Click on “read”. The default value is 0xf70
Click on “write” to write new value: 0x1040
Read again to verify that it was changed
18-Dec-09
TPC Shift Guide