1 TRT DCS: Shifter Basics Main goal – make sure TRT Detector is in good condition: - TRT FSM node: state READY and status OK - no alarms in the alarm screen (alarm screen ALWAYS empty) In case of changing TRT state/status from READY/OK and/or appearance of an alarm in the alarm screen: act immediately: navigate in the TRT hierarchy tree to find the reason enter information to the e-log if needed call OCE/RC (accordingly to severity of the problem) Carefully go through the detector check-list Legend: SL - Shift Leader, RC – Run Coordinator, OCE: On-call expert, DCS – Detector Control System

2 In the beginning of your shift: 1. Check setup of the TRT console in ACR: TRT Desktop 1 Screen III and Screen IV should contain: Screen III: Alarm Panel Screen IV: FSM Panel 2. Login to FSM and to Alarm panels using your nice credentials: Alarm Panel: FSM Panel At the end of your shift: logout from both panels by right-click on key icon and choosing Log out from the popup menu Left click on key icon the login dialog will be opened:

3 Short Overview of TRT DCS 1.Basics of FSM: this slide 2.TRT Hierarchy: slide 4, FSM: ways of navigation, monitoring and control: slides TRT Infrastructure: slides Alarm screen: slides ATLAS FSM Architecture (slide from Central DCS Manual) Control Unit CU: a part of tree, can be excluded, included, taken; contains children of any type – an independent process Logical Unit LU: bottom level of a tree, can be enabled, disabled, can contain other LUs or DUs – an object within a process Device Unit DU: “real” device in PVSS, can be enabled, disabled, cannot have children The detector is broken down into finite state machine units that are hierarchically controlled by other FSMs. Each unit reacts on changes of the internal status of the individual device or groups of devices it is representing and allow simplified control, error handling and interaction with other detector components in the hierarchy. Each unit is descirbed by its STATE and STATUS. STATEs and STATUSes are transferred up in the hierarchy, while commands – down. CUs CUs/ LUs DUs

4 TRT INFRASTRUCTUREBARREL CENDCAP CBARREL AENDCAP A COOLING DAQ VME RACKS LV BULK CANPSU FE_LV TEMP HWI GAS GG SS ALARMS States/Statuses HV FE_LV TEMP HV FE_LV TEMP HV FE_LV TEMP HV TRT Hierarchical Structure DAQ Partitions Commands Comments there is ONE HV for Barrel, it is visible from both sides: C and side A GGSS (Gas Gain Stabilization System) regulates voltage level of HV

5 TRT FSM Panel Navigation Part Fast NavigationFSM Status MessagesFSM Hierarchy Secondary Panel Main Panel left click: start FSM command left click: open Main Panel right click: open Secondary Panel Control Widget Parameter Widget Whether all PVSS systems in TRT are connected. Any missing: red background Whether all PVSS systems in TRT are connected. Any missing: red background

6 FSM Hierarchy: STATE, STATUS and Tree CONTROL Basic TRT States: Available statuses: Control Units – padlocks: OK FATAL ERROR WARNING Everything is fine Low severity, the system can go on working, to fix in the following working hours High severity, serious error, to be fixed ASAP, contact DCS OCE or RC Very high severity, the system cannot work. Immediate action: inform SL, RC, DCS OCE Background colours of padlock/device control: grey: FSM tree is complete orange: FSM tree incomplete, part of a tree is excluded and locked out yellow: FSM tree incomplete, individual devices disabled Logical Units & Device Units: enabled disabled UNKNOWNTRANSITIONNOT_READYSHUTDOWNREADY Tree Control Yellow backgrounds are connected with disabled hardware (HV lines or LV FE boards) – should match exactly information posted in the White Board. STATESTATUSTree Control Legend: SL - Shift Leader, RC – Run Coordinator, OCE: On-call expert, DCS – Detector Control System Obligatory (and the only) for: TRT node and DAQ Partitions nodes ANY orange background shows a problem

7 Navigation through the hierarchy 1. FSM hierarchy tree General rule: Left mouse click on Control mentioned below: navigates to the node described by the control and opens its Main Panel. Right mouse click: opens its Secondary Panel without changing navigation. 2. Subsystems in Fast Navigation 3. Control Widget This is a graphical representation of an FSM node within the detector granularity: any of a plane figure (e.g. a polygon, a circle ), with a circle inside it. Background colour of the figure shows the node state. Background colour of the inside circle shows the node status. When you move your mouse over that shape, a pointer (small finger) and a tooltip describing control name is shown. Then just press Left or Right mouse button. A simplified and fast version of the navigation. This panel is present always on the FSM screen and is fixed. It consits of 4 columns containg DAQ partitions, each consisting of 3 subsystems: HV – High Voltage, LV – Front End Low Voltage and Temperature and of 8 members of Infrastructure (overview on p. 9,10). Background colour of letters describing subsystems show their state, where status is shown by two letters on the right: OK, W, ERR, F and subsequent background colour. Navigation partCommand part How to start a command: 1.Left click on a state field of a node 2.When command name is shown choose it 3.Then a confirmation window appears: choose YES or NO

8 Displaying data Right Click on any Parameter Widget shows a trend of its value: if a parameter is stored in the DB, the view is like on a picture below. if not: there is no line, only current value is shown. On some panels left click on the following buttons opens an independent page with trends: Background colour of a Parameter Widget contains an information: This background colour is shown, if there is no alarm connected to the value If an alarm is connected to the value, the background colour can be on of: green: the value is in valid range yellow: WARNING limit was exceeded, usually (but not always) connnected with WARNING STATUS orange: ERROR limit was exceed, is connected with ERROR STATUS red: FATAL limit was exceeded, is connected with FATAL STATUS grey: an alarm needs to be acknowledged, as source of the alarm dissappeared dark grey: bad (invalid) value Several trends for Cooling systems Plot of a spectrum in GGSS

9 Some hints on displaying trends Moving mouse show tooltip text: 1.over Y (value) axis 2.over X (time) axis 3.over value (datapoint) name Menu: Time Range allow display pre-chosen time interval or jump into any moment in the history 1:1 unzoom both axis Left mouse click on any part of trend area opens a ruler window. Then moving the ruler line shows value in the window. Keeping pressed left- mouse button while moving allows for dynamic zoom 1 2 3

10 TRT Basic Granularity Barrel: LV, HV, some Temperature Groups End Cap: LV and some Temperature Groups: 32 units HV: 16 units – marked with magenta lines HV: 16 units – marked with magenta lines 32 units, called: DCS Slice - when controlling 1/32 of both End Cap and Barrel: used in LV systemDCS Slice - when controlling 1/32 of both End Cap and Barrel: used in LV system Barrel / End Cap Readout Stack – when controlling 1/32 of either Barrel or End Cap, used in LV and some groups of TemperatureBarrel / End Cap Readout Stack – when controlling 1/32 of either Barrel or End Cap, used in LV and some groups of Temperature Barrel HV Stacks – when controlling 1/32 of Barrel: used in HV systemBarrel HV Stacks – when controlling 1/32 of Barrel: used in HV system 16 units, called: HV End Cap Double-Sector (e.g. S3S4)– when controlling End Cap HV (signed in magenta on right picture)HV End Cap Double-Sector (e.g. S3S4)– when controlling End Cap HV (signed in magenta on right picture) Temperature Groups, divided by roles. 1 (side A) 2 (side A) 32 (side A) (side C) 1 (side C) 2 (side C) 32 (side C) 1 (side C) 2 (side C) 32 1 (side A) 2 (side A) 32 (side A) Lower granularity depends on either type on controlled hardware or a detector part. The detailed description is placed in: DESCRIPTION OF TRT DCS FOR SHIFTERS HV: S3S4

11 TRT Basic Granularity: temperature groups Temperatures are grouped by their role: ENDCAP FE Electronics, Electronics Cooling, Straw Cooling, Mechanics and Environment BARREL FE Electronics, Cooling Plates, Detector Sensors and CO2 Ventilation

12 TRT Basic Granularity: HV, LV System and Temperature Systems The status and state of HV and LV systems is shown in different places with different granularity. Also there are some TRT-dedicated states on lower levels, with meaning, name and colour dedicated to subsystem Subsystems in Fast Navigation: for entire DAQ Partition: states are shown with colours: States of HVStates of LV READY SAFE_HV RAMPING PREPARED SHUTDOWN NOT_READY UNKNOWN READY UNPREPARED ALL_OFF NOT_READY UNKNOWN Subsystems in Main TRT Panel: for Barrel or EndCap Stack/Sector states are shown with colours: States of HVStates of LV READY SAFE_HV RAMPING PREPARED SHUTDOWN NOT_READY UNKNOWN READY UNPREPARED ALL_OFF NOT_READY UNKNOWN States of Temperature READY NOT_READY UNKNOWN States of Temperature READY NOT_READY UNKNOWN

13 TRT Infrastructure Overview: monitor + control DAQ CRATESMARATONS CAN BUSES PSUsGGSS VME Crates containing TRT DAQ electronics. FSM NODEFN Legend: Node name in Navigation Part Node name in Fast Navigation BULKVME CANGGSS Gas Gain Stabilization System: basic overview of the system functions as well as the display of Fe-55 spectra collected on different detectors within the GGSS monitoring boxes. Shifter: no action Shifter: EMERGENCY action: Low Voltage bulk power supplies. 20 MARATON power boxes, housing 12 independent power supply channels each. 40 channels used for powering PP2 boxes, 192 channels used for Front End electronic supply. ATLAS Standard power supplies, for CANBUS and ELMBs.

14 TRT Infrastructure Overview: monitor COOLINGRACKS HWIGASALARMS Information about FE electronics and cable cooling stations. FSM NODEFN Legend: Node name in Navigation Part Node name in Fast Navigation COOL HWIALsGAS There is also a table with the temperatures of all TRT racks in USA15 as well as of cooling water for racks and UX level8 maraton power supplies. Information from Central DCS CIC (Common Infrastructure) on TRT racks in USA15. Racks in row 14: 23 – 27 LV 28 – 30 HV Racks in row 11: 22 – 26 DAQ 27 – 29 DCS PCs, CANPSU and HWI Information from 2 external systems: DSS - Detector Safety System and CIC. DSS monitors cooling of FE, cabling, racks, power failure and activation of UPS. CIC data show status of leak sensors in Maraton LV racks located in UX. HWI – Hardware Interlock System automatically switches off LV Maraton Bulks in case of: over temperature in any of FE electronic card on the detector lack of supply voltage in any of PP2 boxes Monitoring of TRT Active Gas

15 Alarm Screen All Alarms coming from TRT detector are shown in the alarm screen. If an alarm occurs, its information appears immediately in the alarm screen, with severity described in the first column: with a letter in appropriate colour: F – fatal – red,E – error – orange,W – warning – yellow and other information in further columns. There are 2 types of alarms: - requiring acknowledgement: after source of an alarm dissappears, the alarm is not removed automatically from the alarm screen, but an action of an operator (shifter or expert) is expected. - without acknowledgement: after source of an alarm dissappears, the alarm is automatically removed from the alarm screen Several alarms can be acknowledged here (choose one option in submenu) or an alarm can be acknowledged individually here (double click on !!! mark) Alarm severity

16 Alarm Screen - controls Dubble click opens a list with available PVSS system and allows to choose systems Write click on Online Value opens a submenu Display alarms with chosen severity or containing a pattern Login field Press ‘Apply Filter’ button after every change of any part of FILTER (blinking green) Filter Settings