Liaison Training Seminar Brought to you by the BaBar/PEP-II Liaison Team (First Version, Sep 2000 TIMeyer)

2 Outline The job Accepting backgrounds Mice, EPICS, and other clicks Hands-on tutorial

3 Being a Good Liaison READ the online shift manual – SIGN UP for Liaison hypernews – VISIT two (2) shifts before your first for hands-on experience ATTEND this training workshop BE smart while on shift; DON’T pass the buck GIVE feedback to other Liaisons and Liaison Coordinator(s)

4 Job Description

5 Job Details Shifts are taken at MCC, Bldg 005 Answer x5121, call x5255 –Call/page experts when appropriate Use your toolbox to stay up-to-date –Make notes in e-log SWING should attend 3:30pm IR2 meeting Bring next shifter up-to-date

6 PEP-II Hierarchy Typically 3-4 PEP-II operators on a shift –1 or 2 for PEP-II –1 for Linac/FFTB –1 “EOIC” – engineer operator in charge This is the person with whom you should speak most often They determine short term schedule They know a LOT 1 Program Deputy (PD) who manages weekly goals –An accelerator physicists, and very friendly –Ask them technical question, but be prepared to answer some in return!

7 Responsibilities Protect BaBar –from user errors –from background conditions Communicate Maximize integrated luminosity Help understand and mitigate backgrounds

8 Why Protect BaBar? Machine backgrounds at PEP-II are serious –High b/c of unprecedented currents and luminosities Consist primarily of –lost particles scattered by beam-gas interactions which then shower into detector –Low angle radiatiove bhabhas which scatter off machine elements into detector High backgrounds can cause –Detector deadtime / reco ineffeciencies –Radiation damage

9 Communication Liaison is primary source of information –For BaBar about PEP-II –For PEP-II about BaBar Responsible for –Coordinating downtime and repair work BBR or PEP won’t always TELL you; you need to ask –Answering questions –Condensing, distilling, and CONTROLLING information –Writing it down in the e-log The Liaison makes BaBar and PEP-II a team

10 What PEP-II Will Ask PEP-II really appreciates Liaison presence –Backgrounds Are they okay? How are they measured? What is their impact? –Beam aborts Who did it? Why? –Luminosity What is it? How is it measured? How much is it –BaBar detector and physics Why can’t we inject? Anything else…

11 What IR-2 Will Ask IR2 wants to know, too –Are backgrounds okay? –What “just happened?” –What’s going on over there? –What happened to beams? –Who aborted the beam and why? –What is taking so long? You’ll soon discover that true DIPLOMACY is required…answer the important questions without bothering too many people, and keep both sides synchronized and content.

12 Who did the Beam Abort? This is an important question whose LACK of an answer is sure to irritate many, many people. LEARN HOW TO TELL. Only the SVT diodes will abort the beam from the BaBar side –Check the SVT Diode EPICS panel for a RED Trip light (more on this later) PEP-II keeps a running logfile of Beam Abort Triggers on a monitor on their side of the room –Ask another Liaison if you don’t know it –Ask operators per “MPS CUD” and/or SLCX9 or MCCX38 –YES, you’ll have to move from your chair to read that display Key to messages you’ll see –SVT_H(L)ABT: SVT diode beam abort –EMC_H(L)INH: EMC_TOTL H(L)ER injection inhibit –BBRHVH(L)INH: BaBar !INJECTABLE injection inhibit –MCC_ABORT: PEP-II op big red button abort –KLYS_xxxxx: RF trip –BOAT: Bad Orbit Abort Trigger Don’t BLAME the operators; share, and work on it together This is the only abort that BaBar can cause, and it should show up in BaBar  SVT  Background EPICS

13 Backgrounds

14 Backgrounds at BaBar High rate of backgrounds can threaten –SVT, DCH, DRC, EMC, IFR, TRG Several subsystems have dedicated background monitoring and protection systems –It is the Liaison’s job to care and feed them or contact an expert when appropriate –Different systems sensitive to different things Sensors are not generally correlated!! When are backgrounds typically the highest? –injection

15 Backgrounds Differ Daily Plot shows bkgs during StableBeams for 1 SVT diode as compared to best bkgs in July,2000 Note that “ratio” decreases, but a LOT of noise, and lately, even central value getting worse! Scrubbing of the vacuum August ROD Vac leak

16 What To Do What does high background mean? –“Pain thresholds” are defined for each detector such that running within 5% of them is OKAY –“Top of the stripchart” = pain threshold When backgrounds are high –Verify that it makes sense –Be sure that not just injection backgrounds –Observe what PEP-II operators are doing –If situation persist for more than 90 seconds, ask operators nicely about it (be armed w/info!)

17 SVT Diodes : Intro System of 12 PIN diodes & dedicated hardware –Monitor and control radiation exposure 4 MID-plane diodes –Receive highest doses (5-10x top/btm diodes) –Are in a special ABORT circuit which aborts beam 8 TOP/BTM diodes (more precise) Signal in mR/s (~5-20 mR/s) –Requires exact knowledge of pedestal Temperature sensitive Changes with radiation damage, too Biggest source of error is pedestal subtraction!

18 SVT Diodes : Summary Panel Latched trip status Can be reset under Details  Trip History See this menu for: Calibrations FastHistory Captains Panel

19 SVT Diodes : Trips Frontend hardware offers “fast” protection –Typically visible on stripcharts, but not always –After “fast” trip, post-mortem buffer read out to disk which will flatline stripchart signal for 60 sec EPICS use stripchart signals to implement 5- minute “Soft Abort” timer (typically 50 mR/s) –Whistles at MCC to warn operators Easily affected by BAD calibrations! BW:MID disabled right now

20 SVT Diodes : Calibration Dose rate signals calibrated AUTOMATICALLY after most beam aborts Frontend ABORT hardware needs calibration 2 times per day MANUALLY by Liaison w/ NO BEAMS More dose received, more often we need to calibrate –Radiation damage HUGE issue –BW:MID pedestal has gone from 1 nA  1100 nA !! During DAILY_CALIB is a guaranteed time to do this!!

21 SVT Diodes : Tips Use the SVT stripcharts to watch bkg levels Keep the SVT Diode EPICS panel open TOP/BTM diodes MUCH more accurate, although less sensitive –They are a good way to verify degree of bad bkgs All 4 MID diodes ocassionally exhibit large oscillations/noise of +/- 100 mR/s for 3-8 min. –Cause unknown, but typically harmless (doesn’t trip) BW:MID diode has unpredictable pedestal –expect poor quality dose rates When in doubt, calibrate the diodes

22 DCH Currents and DCH Trips DCH total current is the most precise signal for measuring some types of bkg –Available on BKG stripchart Trips occur when HV segment exceeds its power supply current threshold for more than 5 seconds DCH trips cause inefficiency and stress the chamber –Can tolerate 3-4 per shift –Often trips at top of fill; ops can fix this if occurs repeatedly

23 EMC Pin Diode EMC worries about long-term integrated radiation dose leading to light losses in crystals EMC has placed several diode/CsI-crystal sensors on the endcap The sum of these sensors’ is used to inhibit injection and cause EPICS alarms –Referred to as EMC_TOTL –Inhibits injection at ~0.7V –Most sensitive to poor HER injection Rare, but can suffer problems –Pedestal drift in which signal is >0.1V during NO BEAMS –Sometimes can get saturated beyond 1.0 V

24 EMC Readout Diodes EMC crystals are read out using biased photosensitive diodes Poor injection or high backgrounds can actually trip the readout diodes’ bias supply –Mild history of power supply spike-trips, too Watch stripcharts on BKG panel –COLORS carry no alarm value (just match color to plot trace)

25 DRC Scalers DRC phototube rates sensitive to LER tunes, LUMI, and global orbit DRC has connected several phototubes to scalers –Some read out in EPICS –Some read out into MCC High DRC ‘tube rates == DAQ deadtime –Pain threshold is 200 kHz –EPICS alarm and stripchart signals Often excessive during injection and at top of fill

26 DRC CsI Sensors DRC has added multiple diode/CsI packages on beamline to understand source and distribution of backgrounds In EPICS –DRC  SOB/CsI-Mon  CsI-Monitors Available on BKG stripchart Not mission critical, but a vast wealth of information

27 TRG Rates and Deadtime Fundamentally important –High TRG rate incurs deadtime Available on FCT stripchart EPICS alarms Well correlated with other bkg signals –If isolated, probably BaBar problem FCT LUMI not always well calibrated wrt BaBar LUMI

28 Common Background Issues ProblemCauseSolution SVT spike or step changeDust (photoionized micron scale material drawn electrostatically into HER beam, typically vaporizes and disperses quickly, but can lead to sustained “trapped event”) Wait High DRC ratesLERLER tunes, skews, collimators SVT diodes oscillateNoiseNone (has never tripped the beam) High L1 rates (sudden)Background spike or step change Wait or dump beam High L1 rates (chronic)LER if also high DRC Trapped event if also high SVT See above SVT diode drifts upward during course of fill Radiation damageWait and re-calibrate when beams lost (Thanks to B. Murphy for contributions)

29 Common Beam Loss Reasons ProblemCause * Solution Both beams aborted1)BaBar abort (SVT diode, 90%) 2)Vacuum pipe temp 3)Magnet failure Ask Liaison to confirm BaBar abort, ops reset PR02 latch in SCP. If curious, ask Liaison for “post mortem” history HER beam aborted1)RF trip (90%) 2)dI/dt Dump LER beam (usually) and re-fill both from scratch LER beam aborted1)RF trip (90%) 2)dI/dt Refill LER beam, usually keeping existing HER fill * Look for cause on white text screen near ceiling above LINAC desk in MCC If you can’t find it, ask operators to point out “MPS CUD” display (SLCX9 or MCCX38)

30 What You MUST Watch Liaison Alarm Hander –Launched from BaBarMain  BKG 3 Liaison-specific stripcharts –Launched from BaBarMain  BKG BaBarMain –BaBar State –Injectable status –Alarms

31 BaBar State Machine BSM mediates activity between BaBar and PEP-II BaBar accepts requests from PEP- II for injection PEP-II declares StableBeams, after which BaBar can go Runnable

32 Your Toolbox

33 Your Toolbox Liaison workspace/environment Starting EPICS Using EPICS Maintaining EPICS Using StripTool Responding to the Liaison Alarm Handler

34 Liaison Computing Two computers available Sun/Solaris and NCD term Since Netscape and EPICS often color- conflict, we strongly recommend –Use NCDterm as an xterm from Username babarpep, password ****** –Use to run EPICS If you have time to and do other work, use additional windows on NCDterm

35 Your Two Screens

36 EPICS You MUST learn EPICS to survive EPICS is the universal BaBar detector control and monitoring environment –Hardware connected to VME single-board computers (called IOCs) –Each IOC shares its knowledge across IR-2 local ethernet in virtual EPICS environment –Users can connect with client displays, alarm handlers, strip charts, UNIX tasks, etc. Similar to web and browser with “links” MCC has been using EPICS longer than BaBar and they’re better at it

37 Starting EPICS To start EPICS –Open a window on You should be logged into the console as babarpep Close all copies of Netscape –Type “startepics” –Ignore warning messages and type “BaBarMain” You’ll get the windows shown here

38 EPICS & Mice EPICS uses THREE mouse buttons –LEFT click (  ) Activates buttons, switches, allows typing Calls FIRST menu option by default –CENTER click Calls dmChan on the EPICS channel, returning a window with the name of the channel and options for details Useful for learning name of a channel to add to a strip chart –RIGHT click (  ) Reveals pull-down menu of sub-panels Critical to realize this (you might never find ½ the panels in EPICS because you never RIGHT clicked and dragged!)

39 EPICS & Colors & Buttons

40 Vital EPICS Panels Always start from BaBarMain SVT  Background –Expert Controls  Calibrate ABORT Thresholds –Expert Controls  AUTO Fast History –Expert Controls  1-minute Diode Dose Rates –Details  Trip History –History Plots  Monitoring Diode Dose Rates BKG –  SVT Diodes Abort Status FCT PEP  Inj Cntrl LUM –  Details

41 Starting & Using Stripcharts StripTool is an application bundled w/ EPICS Allows scrolling time series plot of up to 10 EPICS channels simultaneously –History starts accumulating when you START the tool Key features –LEFT clicking on side legend changes displayed y- axis scale to that channel –RIGHT clicking and dragging in plotspace gives menu. Controls Dialog is very useful. Controls Dialog allows changing of scales, color of plotted curve, and duration of time displayed –Amount of scroll-back history stored is mostly random (depends on EPICS release)

42 StripTool Windows

43 Liaison Alarm Handler Automated method of alerting user when EPICS channels exceed predefined thresholds –Beeps and flashes Launched from BKG panel in EPICS –Click on Liaison ALH button launches a new session Liaison ALH is specific to Liaison –Channels in it are relevant LEFT click on main window gives detailed tree/hierarchy

44 Alarm Handler Tips LEFT click on flashing square acknowledges and silences Can always File  Close to dismiss tree window –This will NOT close ALH top window Good thing to have Occupy-All-Workspaces Detailed breakdown Tree hierarchy Click G for help Click P for EPICS panel Click Arrow for sub-tree

45 Clipping a Picture into the E-log From babarpep on –Open an xterm window –Type “xv &” –LEFT click in the new xv window that opens –Move the windows around so that they don’t block your view of what you want to grab –LEFT click “Grab” in the xv controls window –Move the cross cursor to the corner of the area you want to grab; LEFT click and drag open a selection rectangle. When it is the right size, RELEASE. –Hit “Save” in the xv controls window. –Enter a filename and save it. –Using the e-log running in Netscape on the NCDterm, start a new entry, and click “More Lines” and then “Browse.” –Use the menus to select the file you just saved. You may need to edit the filter or type the filename in by hand, as the directory contents are not refreshed immediately. –You’re DONE!

46 xv Windows

47 Our Tutorial We’ll use IR-2 workstations to invoke EPICS just as in MCC We’ll practice –Finding panels –Interpreting signals –Answering questions (a quiz?!)

48 Tutorial Tasks SSH into as babarpep Start EPICS –Are we taking data? –What is the BaBar livetime percentage for this shift? –How much logged lumi to date (by BaBar)? –What is EMC_TOTL’s current value? –Put BaBarMain in all workspaces (make the BaBarMain window appear in all desktop workspaces) –Is PEP-II “respecting” BaBar? (find the BaBar State Machine page for help) –What is the LER beam lifetime right now? Start the Liaison Alarm Handler –Is there an alarm currently? If so, why? –How many conditions contribute to a PEP-II alarm? Start the 3 Liaison Stripcharts –What is the DCH total current pain threshold? –What is the current L1 trigger rate? SVT Diodes –When were the abort thresholds last calibrated? –What are the current SVT diode thresholds? –When was the last dose calibration? –By how much did the last dose rate calibration adjust the FE:TOP diode’s (stripchart) pedestal? –Find the 5-minute timer / SoftAbort panel For the advanced –When will the HER beam current cause a YELLOW alarm? –Add a TOP/BTM SVT diode to a stripchart –What is the FE:TOP diode’s Total Integrated Dose? –What is the BW:MID diodes’s pedestal current?

49 More Tutorial Questions The DRC scalers turn RED and the Alarm Handler beeps. What do you do? You look up and see that the beam currents are zero. What do you do? PEP-II trips a magnet supply and tells you that it will 20 minutes to bring it back up. What do you do?

50 References stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/ stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/

51