Inner Detector Combined Shift Tools and Procedures version 0.8 September 6th, 2011.

Slides:



Advertisements
Similar presentations
PC Encryption installation progress/password screen Includes comments from: Encryption team Sarah Deane Tony Stieber Selected people who took part in the.
Advertisements

Creating and Submitting a Necessary Wayleave Application
X-Call Version V3.0 User Guide
X-Media V2.0 Healthcare Training Jayex Technology Limited X-Media V2.0 March 2010 v
1 Vendor Reverse Auction - Event User Guide. 2 Minimum System Requirements Internet connection - Modem, ISDN, DSL, T1. Your connection speed determines.
X-Call Version 2.0 user guide A manual to explain how to identify arrived patients and call into a preferred location. A manual to explain.
1 Burn-in Viewer User’s Guide Version 2.0 February 17, 2005 LukasTomasek
Chapter 11 - Monitoring Server Performance1 Ch. 11 – Monitoring Server Performance MIS 431 – created Spring 2006.
Inventory Throughout this slide show there will be hyperlinks (highlighted in blue) follow the hyperlinks to navigate to the specified Topic or Figure.
Application Process USAJOBS – Application Manager USA STAFFING ® —OPM’S AUTOMATED HIRING TOOL FOR FEDERAL AGENCIES.
School of Psychology Online Ethics Guide No. 1 Submitting an application for ethical review. JRC/13/10/2014.
DEMONSTRATION FOR SIGMA DATA ACQUISITION MODULES Tempatron Ltd Data Measurements Division Darwin Close Reading RG2 0TB UK T : +44 (0) F :
ShelterPoint™ Data-Entry Workflows. ShelterPoint v5.2.3.
A walkthrough of the SageQuest Mobile Control Online & ESC integration.
1 Vendor RFI - Event User Guide. 2 Minimum System Requirements Internet connection - Modem, ISDN, DSL, T1. Your connection speed determines your access.
LGC Website and Customer On-line Tools LGC RESOURCE 2014.
Downloading and Installing PAF Insight PAF Insight can be easily downloaded Or can be installed from a CD A license is needed t0 activate the program.
Central DQM Shift Tutorial Online/Offline. Overview of the CMS DAQ and useful terminology 2 Detector signals are collected through individual data acquisition.
5. Data Manager 1. Introduction 2. Data Manager Duties 3. Quality Checking 4. Problem Reporting 5. Data Monitoring 6. Histogram Presenter 7. Trend Presenter.
Scoring Program Updates & XML upload to the NSRCA web site July 2013.
User Interface Elements of User Interface Group View.
Moodle (Course Management Systems). Assignments 1 Assignments are a refreshingly simple method for collecting student work. They are a simple and flexible.
1 How to start the Online Labs module ?. 22 System Requirement for doing Online Labs Before you start your course, make sure that you have following system.
Copyright © 2007, Oracle. All rights reserved. Managing Concurrent Requests.
Access Online Cardholder Transaction Approval Training 1 Client Logo.
1. To start the process, Warehouse Stationery (WSL) will invite you to use The Warehouse Group Supplier Electronic Portal and will send you the link to.
General (Add/Edit General Ledger Accounts) The Add/Edit General Ledger Accounts screen allows you to: Add new or maintain existing GL Accounts View/edit.
André Augustinus 10 September 2001 DCS Architecture Issues Food for thoughts and discussion.
22 October 20151/23Fabio Vinagre -- TileCal DCS TileCal DCS Expert tutorial - DCS 22 October 2015.
1 How to start the Online Labs module ?. 22 System Requirement for doing Online Labs Before you start your course, make sure that you have following system.
An Introduction to Designing and Executing Workflows with Taverna Aleksandra Pawlik materials by: Katy Wolstencroft University of Manchester.
Page 1 Non-Payroll Cost Transfer Enhancements Last update January 24, 2008 What are the some of the new enhancements of the Non-Payroll Cost Transfer?
OFFLINE TRIGGER MONITORING TDAQ Training 5 th November 2010 Ricardo Gonçalo On behalf of the Trigger Offline Monitoring Experts team.
1 ECS CALO HV Control CALO Piquet Training Session Anatoli Konoplyannikov /ITEP/ Outline  ECS HV control of the ECAL/HCAL sub-detectors.  Introduction.
Optimizing Your Computer To Run Faster Using Msconfig Technical Demonstration by: Chris Kilkenny.
Stefan Schlenker, CERN ATLAS Post-LS1 Operations Workshop, 24th June 2013, CERN DCS Tools during ATLAS Run 1 Questionnaire Results Stefan Schlenker, CERN.
L0 DAQ S.Brisbane. ECS DAQ Basics The ECS is the top level under which sits the DCS and DAQ DCS must be in READY state before trying to use the DAQ system.
Part I – Shifter Duties Part II – ACR environment Part III – Run Control & DAQ Part IV – Beam Part V – DCS Part VI – Data Quality Monitoring Part VII.
Handshake with the experiments 04/09/20091 R. Alemany (BE/OP/LHC) LHC Experiments Handshake.
Office of Housing Choice Voucher Program Voucher Management System – VMS Version Released October 2011.
Pixel DQM Status R.Casagrande, P.Merkel, J.Zablocki (Purdue University) D.Duggan, D.Hidas, K.Rose (Rutgers University) L.Wehrli (ETH Zuerich) A.York (University.
DT Shifter Training Infrastructure A. Benvenuti, M.Giunta 07/07/2010.
1 eCoRepair New Release Slide Expanded view of Circuit Details Change to Circuit Looping text4 View of closed faults up to 30 days old5 - 7 Minor.
DCS Instructions K. Grogg, M. Weinberg, M. Grothe 1/24/20161.
Alarm Handling Oliver Holme 7 th November Guidelines & Strategies All provide recommendation to use Framework Classes Standard severities/behaviour/colours.
6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching.
Collaborate. Coordinate. Evaluate. Connecting Communities > Demonstrating Outcomes ™ / I&R Housing Youth & Family Services Older Adult Services ShelterPoint™
1 TRT DCS: Shifter Basics Main goal – make sure TRT Detector is in good condition: - TRT FSM node: state READY and status OK - no alarms in the alarm screen.
ID Week 13 th of October 2014 Per Johansson Sheffield University.
SNG via Webinar. Where’s Webinar??  Double click Aflac 2000 folder  Highlight “SNGWebCommunicator”  Right Click and “Send To - Desktop”
Orders and Invoices Supply Chain Platform: Rolls-Royce Training for Indirect Suppliers March 2016.
1 Top Level of CSC DCS UI 2nd PRIORITY ERRORS 3rd PRIORITY ERRORS LV Primary - MaratonsHV Primary 1 st PRIORITY ERRORS CSC_COOLING CSC_GAS CSC – Any Single.
ISite Main Window Open iSite v7.0 from your laptop (the Main window will appear) Auto-Discovery occurs on if connected to a iNFINITI Modem via a LAN, port.
CACI Proprietary Information | Date 1 PD² v4.2 Increment 2 SR13 and FPDS Engine v3.5 Database Upgrade Name: Semarria Rosemond Title: Systems Analyst, Lead.
Downloading and Installing GRASP-AF Workshop Ian Robson Information Analyst, North of England Cardiovascular Network.
The Next Step Hudson Fare Files 102 – Import & upload Rev. 10/14.
Orders – Create Responses Boeing Supply Chain Platform (BSCP) Detailed Training July 2016.
SchoolSuccess for Coordinators
Operation of CSC DCS Ryan Carroll Carnegie Mellon 29-Sept-2009.
Shift instructions August 16, 2017 Antoni Aduszkiewicz
Inner Detector Combined Shift
Data Entry Interface (DEI) Overview
How to Create and Start a Test Session
Managing Rosters Screener Training Module Module 5
Skype for Business Webinar Meeting
Data Entry Interface (DEI) Overview
Data Entry Interface (DEI) Overview
Presentation transcript:

Inner Detector Combined Shift Tools and Procedures version 0.8 September 6th, 2011

Table Of Contents 1. ID General FSM Panel 2. IDE FSM Panel 3. ID General Alarm Panel 4. DAQ Tools 5. ID Operations & Problems Database 6. Checklists & Elog 7. Special Actions: Warm start/stop, beam dump 8. Online Histogram Presenter 9. Data Quality Monitoring Display

ID General FSM Panel

FSM Panel Explained Fast Navigation Alarm Summary ID FSM Tree Secondary Panel FSM Help LHC Handshake Status FSM Main Panel

ID FSM Main Panel ID Env. TRT SCT Pixel BCM/BLM & Post mortem Background Conditions Injection permit SCT Automatic Turn On

ID Environment Evaporative Cooling Status Compressor Configuration Environmental Cooling Status Thermal Enclosure Heater (TEH) Trip Recovery Button TEH Status Liquid level

SCT Pix Endcap C Status Endcap A Status Barrel Status SCT Infrastructure Status Snapshot shows normal status in STANDBY Should ALL be GREEN when SCT is ON

Pixel Pixel Infrastructure C-Side DisksA-Side Disks Barrel Layers Snapshot shows normal status in STANDBY Should ALL be GREEN when PIX is ON

TRT Gas Gain Stabilization System (GSSS) Status Endcap CEndcap A Barrel C Barrel A TRT Infrastructure ALL indicators should ALWAYS be green

Beam Conditions (see later slides for descriptions) BLM, RAD Mon SCT Auto-ON Injection Permit Status Inj. Permit Mask Configuration Ready For Inj. Status BCM post mortem counter State 6: normal State 5: PM analysis)

Backgrounds SCT Occupancies BCM Rates Stable beam yes/no? Pixel Occupancies Background monitoring active? Green: OK Red: Too high (or no data)

IDEnvironment FSM Reached via IDE button Useful for more detailed look at ID environment & beam conditions, eg in preparation for warm start

IP1 Collimator Check Panel (IDE FSM) Absorbers Beam pipe pressure Background rates Tertiary collimators Beam parameters

Cleaning Collimator Panel (IDE FSM)

Beam Rates Panel (IDE FSM) Beam pipe pressure, BCM rates, LHC params SCT/TRT occupancies LUCID, ZDC rates

Example - Beam Rates RAMP SQUEEZE Collapse of the separation bumps → collisions Lumi optimization

IDG Alarm Screen Sort According to Severity Click On Alarm for details, help, and logging

IDG Alarm Help Insert Directly to elog Get Alarm Details Open Alarm Help (Twiki page) All alarms shall be logged Errors and Fatals shall be followed by a call to the expert Right click on alarm to see options

IDG Alarm Help Alarm explanation along with actions to be taken. Please report any missing help pages! (Opened by selecting ‘Alarm Help’ in options list)

DAQ Panel ATLAS partition Busy monitor DQMDOHP

DAQ Tools 1.Shifter Assistant Alerts the shifter to specific DAQ error conditions that require shifter action Eg ROD busy or disabled etc Complements IGUI panel indicators Includes instructions for the shifter 2.IGUI Panels Indicates DAQ related alerts and warnings Provides shifter tools to take the appropriate action 3.MRS Warnings/Errors/Fatals from DAQ applications Eg missing essential files, application crashes etc Usually require expert actions, but shifter should monitor MRS for ERRORs or FATALs

Shifter Assistant Tool New alert Expand if new alert (Link available soon from desktop) (Link available soon from desktop)

Shifter Assistant Tool Info & Warnings can be ignored Errors and Fatals shall be followed by a call to the expert Read the messages and mark them as read

More details in SA

ATLAS DAQ IGUI Loading IGUI Panels

IGUI Panels The IDPanel under the ID tab of the TDAQ IGUI provides: Visible indications of any SCT and TRT DAQ-related alerts or warnings that the ID shifter should react to In principle, the same alerts/warnings are displayed on the Shifter Assistant Tools to react to those alerts of warnings The IDPanel should be open and visible at all times on the ID shifter desktop At the start of the shift, the shifter should open IDPanel (if not already open) by ticking ‘IDShifterGui’ in the list of tabs shown when clicking the ‘Load Panels’ button at the top of the TDAQ IGUI For info, the IDPanel can be closed by unticking ‘IDShifterGui’ in the same list, but the shifter should leave the panel open

Panel Structure To the left of the panel is an expandable tree indicating the DAQ status for both SCT and TRT Green: OK, Yellow: WARNING, Red: ALARM The ‘ID DAQ Status’ icon colour indicates the combined status for both SCT and TRT The ‘TRT’ and ‘SCT’ icons in the tree indicate the overall status of TRT and SCT respectively If yellow or red, click on the sub-tree component to see the origin of the warning/alarm and the tools to resolve the problem

Overall SCT Control and Status panel, visible by clicking on the ‘SCT’ icon in the tree

Control Mode: Enables recovery tools Only one TDAQ IGUI has control mode enabled (if someone opens another IGUI and takes control mode, this IGUI will have control mode disabled) Module Recovery: Automatic recovery of link errors Should be enabled by default If Control Mode is disabled, module recovery gets disabled too. Global Reconfiguration: Time between automatic global reconfiguration of the SCT to counteract chip configuration corruption by SEUs Set to 30minutes by default SCT Control & Status Panel : 1

Disabled ROD count across full SCT Green: 0 Yellow: 1 (take action!) Red: >1 (take action!) Busy ROD count across full SCT Green: 0 Red: >0 (take action!) Link Error count across full SCT Green: <50 Yellow: <100 (call expert!) Red: >100 (take action!) SCT Control & Status Panel : 2 If any of these 3 indicators are not green, click on the crate icons which are non-green to perform actions

ROD status in first column (Grayed out slots mean no ROD present in that slot) Green: ROD is Enabled, and not BUSY

ROD percent busy Useful to see if 100% busy or not

Link Errors per ROD Should be ideally 0 If non-zero, module recovery is attempted automatically (if enabled in control&status panel) Green: <10 Yellow <20 Red >20

TRT Panel Visible by clicking on TRT Recovery

8/25/11 Synchronization: - If the # of sync problems is > 0, an automatic resync is performed (no shifter action necessary) - If prompted by the Shifter Assistant or an expert, click the Request Resync button to manually request a resync

8/25/11 ROD Recovery: - If the # of disabled RODs is > 0, click the Recover Disabled RODs button to attempt to recover these. - Run Control will have to acknowledge a popup window for this procedure. - If this doesn't work, call the TRT DAQ expert on

8/25/11 Specific ROD Recovery: - If instructed by the shifter assistant or an expert, you can force a ROD recovery. - Type the ROD address into the text box and click Request ROD Recovery. - Run Control will have to acknowledge a popup window for this procedure. - If problems occur, call the TRT DAQ expert on

MRS Call experts for ERRORs or FATALs Enter subscription string to filter for SCT, TRT, PIX and BCM SCT|TRT|PIX|BCM (? To be confirmed)

Pixel Rod Mon Not supposed to be an ID shifter tool

ID Operations and Problems DB Web interface opened via ID menu on desktop (keep it open for easy/quick reference)

Searching Prob. DB Specify Category Specify System

Searching Prob. DB Enter String to search DB Click on DB entry to get details

Searching Prob. DB Details with explanation of DB entry and solution prescribed

Checklists and Procedures 1. Elog 2. Start of Shift 3. Warm Start 4. Warm Stop 5. Beam Dump 6. During Shift 7. End of Shift

Elog Maintain a log throughout your shift of what happened, and upload it at the end of the shift. A new elog interface will be available “real soon” for use by the ID shifter: - uploads your entries directly as you type them, line by line, and appends them to the shift elog summary - useful for experts and run coordinators to see the status of your shift in ‘real time’ - will only append (not delete/change previous lines)

Start of Shift 1. Arrive to Shift 15 Minutes before Start of Shift to get summary from previous shifter! 2. Login to 1. Elog 2. DCS ID FSM Panel 3. DCS Alarm Screens 4. Run Com Tool (not yet possible) 3. Introduce yourself to the shift leader (if they haven’t done so already). 4. Read latest elog messages from last 24 hours. Keep elog regularly updated 5. Ensure that OHP, DQMD, SA, and IGUI tools are up and running with the proper configurations. 6. Ensure that individual ID desks are running their proper tools as well. 7. Check the alarm screen for any alerts. 8. Check the DAQ status (IGUI ID panel and SA) 9. Check the status of the pixel infrastructure (pixel igui, checked by SA) 10. Check the FSM Status (for silicon should be READY/OK during data taking, otherwise in STANDBY/OK, TRT should always be READY/OK).

During Shift 1. Monitor and acknowledge any alerts from the SA tool, and take action as needed 2. Monitor the IGUI ID panel and take action as needed 3. Log any alarms in the FSM. Acknowledge them once they have been logged and WENT. Take any necessary actions needed (use Alarm Help and/or Problems DB tool). 4. Check the trend in evaporative cool plant liquid level (~ 1kg / 8 hours). 5. Check ROS occupancy is non-zero in FSM. If it is zero and beam is present, then ask the Run Control Shifter to restart the SCT ROS Monitoring. 6. During data taking, check hourly 1. DQ checklists 2. Number of disabled pixel modules, especially b-layer (OHP) 3. Check Pixel Module Errors (i.e. LVL1ID errors) (Pixel Rod Mon / DQMD)

Start of Run It is important to be specially vigilant at the beginning of a run. Problems most likely occur at this point. Be aware of: 1. FATAL / ERROR messages from the Pixels, SCT, TRT, or BCM appearing in the MRS 2. Check BUSY status using the BUSY status display. If no triggers are coming through one of the systems could be busy. 3. Check the IGUI ID panel to ensure all is green 4. Stop-less removal could happen at this stage, the run control shifter should tell you if an ID System goes busy and may disable one or more of the RODs. Follow the instructions in the operations database if this happens. 5. Ensure that all the Pixel RODs are allocated by the Central DAQ. If not an ERROR should appear in the SA

During a run 1. Keep checking the SA and IGUI ID panel. If the run control shifter did not tell you he/she disabled an SCT ROD, you would see a big increase in link errors. 2. Check SCT ROS Monitoring every hour. Look for errors and anomalous values in the results 3. Continue to check for errors in MRS 4. Watch out for busy of any of the detectors. If the run control shifters asks to disable a ROD always say yes. 5. Document everything throughout your shift and upload a single e-log entry at the end.

Before beam injection Before and during beam injection the Pixel and SCT need to be in the STANDBY state, ie for Pixel preamplifiers and HV off and for SCT reduced HV (50V instead of 150V) 1. Make sure that neither Pixels nor SCT maskes the stable beam flag / "Injection Permits" 2. If SCT is not already in STANDBY, Issue GOTO_STANDBY from the top level SCT node in the FSM. Recover any modules as necessary. If you can't get all branches to be in the STANDBY state, call the SCT DCS on-call expert (162034). 3. Check that Pixel is in STANDBY and gives injection permit. If not please call the pixel on-call (160032).

Before stable beams (i.e. start of SQUEEZE) 1. Open "IP1 Collimator Check" tab in the IDE panel and monitor the beam background, BCM activity, SCT occupancy with ROS monitoring and overall beam rates. You need to do these checks continuously and very closely, and check the trends in the rates. If the rates are too high (indicators are RED) and the trend is increasing, its unlikely we will be ready to switch on when stable beam is declared. 2. Call the Pixel expert in case they are not in the ACR yet. (160032) 3. Check at the end of the squeeze that all collimators are in there right position, i.e. all boxes have to be green.

Warm Start Warm Start happens when the LHC declares Stable Beams at the beginning of a physics fill. There is an audio indication in the ACR when this happens. The silicon detectors need to ramp their voltages in the warm start procedure. The ramping of voltages for Pixels is done by experts so the ID shifter has no responsibility other to than to make sure the appropriate expert is called (well before stable beams is declared) for this procedure. The ramping of voltages for SCT is automated, so long as the background conditions are ok TRT requires no special actions

Warm Start – SCT Special Actions 1. The AUTOMATON widget will automatically switch on the SCT on declaration of stable beams, provided all warm start requirements (collimator positions, background rates, SCT occupancy) are met On declaration of stable beams, watch the SCT voltages switch on, recover (manually switch on) any modules which fail to switch on by navigating down through the SCT FSM (should not be necessary – afterwards log this action so the SCT expert knows) If warm start conditions are not met, AUTOMATON will not grant the automated switch on. Call the SCT run co who may ask you to switch on the SCT manually (GOTO_READY in SCT FSM) after he/she verifies that it is safe to switch on

SCT AUTOMATON state summary DISABLED: not active, manuals switch-on required NO_BEAM: no beam in LHC BEAM_PRESENT: beam in LHC BEAM_CONFIRMED: ID agrees there is beam (background rates etc) BEAM_OK: ID declares beam ok GRANTED: flag set for automated switch on STABLE_BEAM: stable beam declared ON: SCT has successfullu ramped ON NOT_CONNECTED: no connection to GCSLHC or IDESCS – call SCT expert on NOT_VALID: no data – call SCT expert on162034

Warm Stop Warm stop happens when the the Stable Beams Flag is removed or an “LHC handshake” is started. The Stable Beams Flag can be removed for several reasons: After an unscheduled Beam Dump. Variable delay: from immediate to 15 min. When the LHC schedules a beam dump and initiates a Beam Dump handshake. When the LHC attempts a beam adjustment and initiates a Beam Adjustment Handshake These are all indicated on the LHC Page 1 web page, and also associated with an audio indicator in the control room During the Warm Stop the silicon detectors ramp down their voltages. The Ramping down of voltages on the silicon detectors is automatic. No action required. If by accident one or more SCT channels remain ON (never happened before) and prevent handshake completion, switch to stand by manually the individual channels, then call the SCT on-call The ID Shifters responsibility is to make sure that the silicon detectors end up in their Stand-By State. If there is a problem at any point, then the appropriate On-Call Expert should be called. TRT requires no action at Warm Stop

Warm Stop Both SCT and PIX react to “end of physics” signals automatically, in different ways PIX Action: Switch off HV, turn off pre-amplifier HV on, preamps on = READY HV on, preamps off = ON (Pixel partition NOT_READY) HV off, preamps on = LV_ON (Pixel partition NOT_READY) HV off, preamps off = STANDBY Invoked by start of BEAMDUMP or ADJUST handshake: Invoked by unplanned beam dump Reacts on post mortem signal If stable beam flag is removed immediately → HV interlock Interlock needs to be acknowledged SCT Action: Ramp down HV to 50V (READY → STANDBY) Invoked by start of BEAMDUMP or ADJUST handshake, or removal of stable beam flag

Beam Dump 1. When a Beam Dump Occurs, the BCM RODs Readout state will change to 5. This means that the RODs are reading out the post mortem from their buffer. 2. After 5 minutes if they do not change to a state of 6, call the BCM On-Call: Once the BCM is finished reading out, check the post-mortem histograms in the BCM OHP (at the BCM desk). If the histograms are Green, then everything was good. If they are not Green then call the BCM On-Call:

End of Shift 1. Fill out Shift Summary elog template. 2. Inform the next shift of current system status and any significant events (significant means did you do something that is not normal operation or did you call an expert). 3. If the next shifter does not arrive within 30 minutes after the start of the next shift, call 16-XXXX (reserver shifter), if all else fails call your favorite RC.

ID Monitoring – check list Help or more explanations

OHP Sub detector tabs

DQMD

Pixel SCT TRT

Desktop Layout IGUI, SA DQMD, OHP ID ALARMS Liquid trend Busy ID FSM Screen 1Screen 2Screen 3Screen 4 Desk top 1 Pixel Alarms SCT Alarms TRT Alarms ID FSM Screen 1Screen 2Screen 3Screen 4 Desk top 2

Desktop Layout II WhiteboardElogRunCom Prob. DB ID FSM Screen 1Screen 2Screen 3Screen 4 Desk top 3 BCM Alarms PM OHP IDE Alarms ID FSM Screen 1Screen 2Screen 3Screen 4 Desk top 4

Feedback 1. Please follow all the check list and give feedback! 2. Don’t be shy to put all inconsistencies, problems which could not be solved alone, etc into the elog. 3. We need to be prepared also for not experienced shifters!