1. TPC shift guide
Run period ??.??.2008
Updated on

2. Foreword
This shift guide is intended for the TPC shifter to be able to operate and run the TPC:
Identify the relevant machines:
aldaqacr34, the local TPC linux machine in the Working Room (TPC.501)
alitpcon001, the Windows PVSS Operator Node
alitpcwn008, the Busy Box linux machine
Power on/off and configure the FEE for any specific run
Configure and initialise the Busy Box
Use the Local Trigger Unit
Start/stop various standalone runs, or get ready for a global run
Periodic pedestal and cal. pulser runs (DAs); laser runs
Use the online monitor to look at event displays, and use AMORE
Monitor the TPC: LV, FEC temperatures, cooling, ROC HV, Drift HV, gas, run status
Use the electronics logbook
Troubleshooting, list of expert

3. Connecting to the Operator Node
Make sure there is a remote desktop session to alitpcon001; if there is not one, a registered user does:
rdesktop –g1280x1024 –a16 –u username alitpcon001
Enter your NICE username and password
Start  TPC.bat
Click on the key icon (top left) to login as tpc tpc
You now have the PVSS panel
On the left there is the tree with all the panels
Bottom left, a synoptic with the colour-coded status of all systems
The top level node, TPC_DCS, will be the most extensively used
Note: if you can't type in a field, go to a linux terminal session, type something there, and go back to PVSS

4. Functionality of top node
TPC_DCS: Top node of the TPC FSM Tree Take the lock of the FSM Tree from here. Before closing the main panel release the lock with the option RELEASE ALL FSM Commands: State: OFF Action: GO_STANDBY -> Switches ON LV PS State: XXX Action: CONFIGURE  -> Configuration of the FEC according to the sent parameter State: XXX Action: SOR or EOR -> Actual condition values are send to database for Shuttle
If the FSM has to be restarted (it also releases the lock) press 'Start/Restart All'. Wait until the 12 boxes (tpc_ui etc.) on the lower part of the panels are green.
Forced restart: Press 'Stop All' wait ~20 seconds, press 'KILL FSM' wait ~5 seconds, press 'Start/Restart All'

5. Cooling Experts: TPC_COOL (in Operator Node) (next page)
FEC Cooling plant must be RUNNING
Resistor Rod Cooling plant must be RUNNING
In case of problems ('Standby' or 'Stopped') call expert
Experts:
Danilo Vranic
Chilo Garabatos
CCC (ask for CV/Detector Cooling, answer ALICE TPC)

7. Power up and configure FEE
Switch ON LV Power Supplies from PVSS
Use the FSM of either the FERO or the top level TPC_DCS
select GO_STANDBY either for one particular sector or for all
Or click on a sector in the LV MONITOR panel and select ON
Go to the FED MONITOR, select FSM state and wait for required partitions to be blue
In the (sub)system state tab, select CONFIGURE and enter, e.g., COSMICS as parameter. This will turn on the FECs and configure them for running with cosmic triggers.
Other commands are
GO_IDLE (FEC power off)
CONFIGURE with entry RESET (a popular command)
CONFIGURE with entry PEDESTAL_RUN (for pedestal run)
Before each run it is recommended to CONFIGURE with entry RESET and once more with, e.g., entry COSMICS (and reinitialise the BB, see later)

8. FEE Monitoring In TPC_FERO you can select different monitors:
FED Monitor: shows by default the temperature of the FECs, which should be lower than ~30 oC.
In the lower third of the panel, all boxes should be green. If one is red, try Restart. If still red, call expert (CL).
LV Monitor: shows by default the analog currents, which should be 2 Amp (FECs off) to 130 Amp (all FECs on).
Cooling Monitor: shows the status of the cooling: it should be OK and RUNNING (all green).
In case of problems call
Christian Lippmann
Luciano Musa

9. FEE Configuration (after power up)
On the operator node (alitpcon001) right-click on TPC_FERO in the FSM tree to the left.
Choose view panel.
Click on the FSM button (just above).
Take the lock by clicking on the lock symbol.
In the System TPC_FERO (on the top) click on the state and choose CONFIGURE and type RESET as run parameter. The states should all be (or change to) STANDBY.
Choose again CONFIGURE ,and type as run parameter the right name from the table of next slide.
Wait for all states to change to ON. This may take ~10 seconds.
In case of ERRORs choose RECOVER and repeat steps 6 and 7.

10. Configurations PEDESTAL_RUN - No For Pedestal DA BLACK_FPED ZS_FPED
Name
Pedestal Subtr.
Zero Suppression
Remarks
PEDESTAL_RUN - No
For Pedestal DA
BLACK_FPED
Black evens for, e.g., signal studies
ZS_FPED
Fixed
Yes (Threshold 3)
1 Pre- & 2 Postsamples. FPED values are part of configuration.
ZS_PEDMEM
Pedestal Memory
Yes
(Threshold 3)
1 Pre- and 2 Postsamples
COSMICS
0 Pre-,0 Postsamples, Gating Grid Suppressed, Glitch Filter On pp
Like cosmics. Only relevant acceptance readout

11. Configuration via DCS of single RCUs
On the operator node (alitpcon001) open the FSM tree and find the desired partition (RCU): TPC_RCU_xxx_p.
Check monitoring values and change states here.
You get the same functionality by clicking on the individual partitions (RCUs) on the FED Monitor panel.

12. Troubleshooting: FEE Configuration / Monitoring via DCS (1)
The FSM tree is locked:
Find out who has the lock, and ask him/her to release it.
From TPC_DCS top node, Restart ALL, Kill FSM, Start/Restart all
No response from FEE:
If it is a single Partition (DCS board / RCU), try
Restart feeserver on this DCS board (ask expert),
Reboot this DCS board (ask expert) or
Do a power cycle on this sector: GO_IDLE, GO_OFF, GO_STANDBY
It takes 20 to 60s for the DCS board to reboot and for the data from this RCU to appear on the monitoring panel.
Restart PVSSdim by clicking on “Kill PVSSdim“ in the FED Monitor panel.
Restart interComLayer(s) (see next slide).

13. Troubleshooting: FEE Configuration / Monitoring via DCS (2)
Restart interComLayers:
Log in to alitpcwn008 (A side) or alitpcwn009 (C side) (user tpc, no password needed from aldaqacr34).
Check if interComLayer is running (type „ps –a“ in a shell).
Restart it by executing „killall interComLayer“.
Additional information:
The interComLayer is an intermediate Layer between the FeeServer on the DCS boards and the PVSS and FSM Control Layer you are using. There is one per side.
It runs in the directory /home/tpc/icl-rundir.
In case it exits (or is killed), the interComLayer is automatically restarted by the script /home/tpc/icl- rundir/startInterComLayer.sh.

14. FEE Configuration via PERL scripts
Go to aldaqacr34 and type the command:
./SECTORS_do.pl has the following options: reset/config/on/off/armSYNC
choose the appropriate one, the side, and the sector number
Examples of commands:
./SECTORS_do.pl A config 0-17 (to configure all FEE on A side)
./SECTORS_do.pl C reset
The normal sequence is on, reset, config
The option armSYNC is used in stand alone runs only, in case one has just reconfigured a part of the system

15. FEE Configuration via scripts in each LDC
If you log into one LDC, you should go to /local/home/tpc/TPCrdo and type:
source TPCrdo_commands.sh
Commands: SECTOR_on SECTOR_off SECTOR_reset
SECTOR_configuration SECTOR_armSYNC
If this failed for a partition, you can address it individually:
rpx_on rpx_off rpx_reset rpx_config rpx_armSYNC
x=1,2,3,4,5,6

16. FEE Configuration via TPCC (1)
1) Find a desktop with a session in aldaqecs01 (/date/tpcc/Linux)
2) Type: ./tpcc_server (and maybe type after this:
./dumpOracle.tcl)
3) Find a desktop in /local/home/tpc/rundir
4) For each LDC type
/date/tpcc/Linux/tpcc_client
5) Move to aldaqpc126 (should be in /local/home/tpc/TPCrdo)
-should see configurations (1-15), different commands for FECs( ), check the status of server (200) and exit (300)
Not yet fully operational

17. FEE Configuration via TPCC (2)
If you don't see the above options, type:
/date/tpcc/Linux/tpcc_control
To see online the informations online go to ALICE DAQ
window, press infoBrowser and another window should
appear; in this new window change from Stream (should be TPC and you have to clear this field) to Facility (write TPCC) and then press Online
Not yet fully operational

18. Trigger (Standalone) To run CTP Emulator (LTU, trigger window):
Go to alidcscom026 (tpc tpc145) and type vmecrate tpc
The VME crate window should appear
select Configuration -> TTCinit
(Busy box is used by default)
Open the CTP Emulator window and here
choose sequence L2a.slm
load it by pressing the button Load sequence
select Signal
for pulser trigger choose BC and write the rate, e.g., 1s
for laser runs choose pulser/edge
In case the CTP emulator gets stuck, kill the 'ltu client' process from alidcscom026, and use the command 'xkill' and mouse-clickto close the dead windows. In case of doubt or problems, please ask the CTP shifter.

19. DAQ (Standalone)
In DAQ Run Control window (click Date HI button in task bar to open):
Select LDCs for Readout
make sure you have the lock
press left arrows until you get to Configuration
press Define
go to GDC and choose e.g. aldaqpc046
go to Detector: choose TPC -> A/C side -> choose LDCs
save configuration and press ok
Select the HLT mode (Mode A: no HLT)
In DCA window (for DAs for example):
Make sure you have the locks
Select the run type: pulser, laser, etc
Start a run

20. Configuration of Busy Box
Log in to alitpcwn008 (user tpc, no password) - cd /home/tpc/busy - execute ./bbox-conf.pl - select the partitions - save and exit
From here log into busy box dcs board: ssh alidcsdcb1555
- cd /mnt/dcbrw/busy/busybox_files - execute ./bbinit.sh (before each run!) - to check status, execute ./regpoll.sh status
If one of the two top registers is zero, execute
./program busybox_fpga1.bit busybox_fpga2.bit

21. Start/Stop stand-alone run
In DAQ Run Control
press right arrows to get from Configuration to Ready to Start
press 'Start process', wait for ready and check for errors
press 'Start'
press 'Start emulation' in LTU window
Note: Each sector (6 RCUs) is readout via optical fibre with a Local Data Concentrator, a DAQ machine. The data is then transferred to one or several Global Data Concentrators which may or may not perform Event Building and Data Recording
To stop the run, quit the emulator (LTU) first, then click stop in Run Control

22. Run control from ECS (standalone)
Use the ECS panel TPC Detector Control Agent (DCA) to control the run.
get the lock from the „TPC DAQ – Run Control“ (release it there first)
Select the operation of the GDC
No event building
No data recording
mStream
get the LTU, DAQ_RC, DETECOR, (HLT) locks
Select the HLT mode (ask the HLT shifter)
Select the run type (pedestal, pulser, laser, cosmic, ...)
this will start a run
pedestal and pulser will stop automatically
Eventually stop the run
Note: if your lock is red and nobody seems to have it, make sure there isn't a second, hidden window in your panel: this one might have the lock!

23. HV Monitoring Drift HV (SIM_TPC_FC next page)
check Voltage (100 kV) and Current (355 A)
ROC HV (SIM_TPC_ROC next page)
In FSM state, all sectors must be green
If red (trip) call expert
If electronics is turned off >1 hour
click on the corresponding chambers
Switch ON button to INTERMEDIATE
Check the currents: call expert if > 1 mA
Experts:
Danilo Vranic
Chilo Garabatos

26. Gas TPC_GAS (Operator Node) Status must be 'OK' and 'Connected'
Mixer, Exhaust, Pump and Distribution modules should be run or circulating; if stop, call expert.
You may navigate around and look at trends
In case of problems, call expert:
Chilo Garabatos
Gas piquet:

27. Gating Grid pulsers
The Gating Grid Pulser voltages are operated by experts.
Experts: Danilo Vranic , Rainer Renfordt , Chilo Garabatos

28. Laser One PVSS panel per side. To turn on and run do: GO_STANDBY
LOAD RECIPY
GO_ON
SUTTER_OPEN
Expert: Borge Nielsen

29. Calibration pulser There is one unit per side. In each of them click
Open COM port
INIT
All ON
The corresponding automatic run (DA) is called Pulser_Calibration_run
Click on All OFF when you are done

30. Online monitoring
Log in as and go to /local/home/daquser/OM
type root
in root type: .x Monitor.C
another window appears:
select as format (DATE file/stream)
press Select and OK
select File/Stream -> write LDC or GDC name
Monitoring has to be enabled on that machine
press Next Event button
If root crashed or hangs, open another window in aldaqpc081 and type: killall root root.exe
To take a screenshot, open a terminal on aldaqacr34 and run ksnapshot; paste it in the logbook

31. How to run AMORE /1
1) Open two terminal windows on aldaqacr33 (as user tpc)
2) In both do: ssh aldaqdqm04, source .balice, cd /tmp
3) Check that you get the monitoring data by doing e.g. eventDump –i
4) On the first machine start the amoreAgent: amoreAgent -aTPCQA
5) On the second machine start the display: amore  -dTPC -mUIQA  - select init  - select start  - when you receive data, make the window bigger and you should see the different variables

32. How to run AMORE /2
The variables are: MaxCharge – Qmax of the cluster MeanCharge – Qtot (integral over the 5x5 cell) NlocalMaxima – total number of clusters in this pad NPads – average width of cluster in pad direction NTimebins – average width of cluster in time direction (Noisy pads will typically have a small average since the noise is not shaped, see example in guide) NoThreshold – fraction of digits above threshold (ZS reduction) This is the only variable which is not a cluster variable but shows the average occupancy for each pad (between 0 and 1) TimePosition – mean Z (time) position of clusters (if you have many noise clusters in a pad this will give a peak centered at the middle of the time bin interval because the noisy clusters are typically evenly spread out in time - perhaps a bit counterintuitive).
Some more info can be found here:

33. Logbook Use it. For each run add an entry with the run conditions:
FEC configuration
Missing partitions
Gain
Magnet status
Define run quality for TPC (good, bad)

34. List of on-call experts
FEE, DAQ, DCS, Trigger
C. Lippmann
L. Musa
FC HV, ROC HV, Cooling
D. Vranic
C. Garabatos
Gas
Gas piquet
Laser
B. Nielsen
GG Pulser, Calibration Pulser
R. Renfordt
Online, Offline
M. Ivanov 77761
P. Christiansen 78956