LHC Beam Operation WorkshopM. ZerlauthDecember 2011 Thanks to : rMPP and MPP members, K.Fuchsberger, M.Galetzka, B.Goddard, R.Schmidt J.Wenninger, et al 1v0 Machine Protection System response MPS response in 2011 Dependability of MPS backbone Issues and areas of upcoming improvements Commissioning and validation procedures Change management (incl. SW release)

CERN LHC beam commissioning Workshop - Evian Review of Protection dumps beam dumps were cleanly executed during 2011 (-10% wrt to 2010 ) No beam induced quench with >100MJ 3.5TeV in 2011 (including all ‘quench’ tests) No equipment damage observed (apart from kicker erratic causing damage in SDD calibration of ALICE) MPS response of all dumps from 3.5TeV meticulously analyzed and validated – Initiating system always identified, but sometimes not fully clear why it triggered (‘spurious’ triggers, SEUs,…) 2010 Nota bene: All statistics only counting fills > injection 2011 Beam dumped without beam losses and orbit changes, 107 Beam dumped without beam losses and orbit changes, 208

CERN LHC beam commissioning Workshop - Evian Review of Protection dumps 3 During 2011 run we counted 40% more successful ramps to 3.5TeV Relatively speaking less MPS tests (commissioning, ramp-up after TS, loss-maps,…) ~ Factor of 3 less dumps by beam monitoring (beam losses, orbit changes,…) -> 2010/11 improvements do their job A small relative increase of false triggers of MPS system Beam dumped without beam losses and orbit changes, 30% Beam dumped without beam losses and orbit changes, 41%

CERN LHC beam commissioning Workshop - Evian Dependability of MPS 4 False triggers of most MPS backbone systems remained (surprisingly) constant 95% of false dumps in 2011 above injection energy 4-fold increase of QPS dumps (half of which confirmed or possible SEUs) -> See R.Denz confirmed SEUs + ‘environment’ related triggers 5 possible SEUs

CERN LHC beam commissioning Workshop - Evian False Beam Dumps from MPS - LBDS Occurrence of false dumps of some systems clearly related to increasing beam intensities Time MKD generator and attenuation on BIS-LBDS loops BETS, dilution kicker, vacuum pressure in dump line

CERN LHC beam commissioning Workshop - Evian False Beam Dumps from MPS - BLM Occurrence of false dumps of some systems clearly related to increasing beam intensities 3 dumps due to HV interlock after local losses + 1 communication problem Time

CERN LHC beam commissioning Workshop - Evian False Beam Dumps from MPS - Interlocks Occurrence of false dumps of some systems clearly related to increasing beam intensities PIC: 3 consecutive events in UJ14/16/ following in UJ16 and UJ56 Time SIS: Loss of communication with RF – HOM interlock BIS: Glitch on BPL loop B2-A

CERN LHC beam commissioning Workshop - Evian False Beam Dumps from MPS - QPS Occurrence of false dumps of some systems clearly related to increasing beam intensities Clear effect on QPS! HW faults in Q10.L2 with ions High local activation Time

CERN LHC beam commissioning Workshop - Evian Beam dumps by beam monitoring 9 EVENT_TIMESTAMPENERGYMPS_DUMP_CAUSEMPS_DETECTIONCOMMENT 21-NOV Orbit FeedbackBLMUnstable OFB in squeeze with gain increase by a factor of NOV Orbit FeedbackOP dumpWrong OFB Parameter ORBIT_REF CHANGING_TIME (1s instead of 680s) 03-NOV Beam LossBLMBeam Loss during aperture measurements 17-OCT Transv. beam instabilitySISHigh Losses due to bad tune signal (end of ramp and squeeze without QFB) 28-SEP Transv. beam instabilitySISOFB problems during ramp 28-SEP Transv. beam instabilitySISOFB not sending correct trims 20-SEP Beam LossBLMQFB dragged tune of B2H on resonance 11-SEP Beam LossBLMVacuum spike 11-SEP Beam LossBLMVacuum spike 25-JUL Beam LossBLMOrbit oscillations and consequent losses during optimisation 24-JUL Beam LossBLMVacuum spike 23-JUL Beam LossBLMVacuum spike 09-MAY Transv. beam instabilityBLMBeam instability after quench test (ADT picking up wrong tune) 08-MAY Beam LossBLMBeam instability in B2V at end of coupled bunch instability MD 28-APR Beam LossBLMVacuum spike 25-APR Beam LossBLMVacuum spike 20-APR Transv. beam instabilityBLMHorizontal beam instability with 2-3 s risetime 16-APR FB 2SISOrbit reference change applied in 1s instead of 385s, excursion in B2H 14-APR Transv. beam instabilityBPM6Beam instability during IP scan Some 40 dumps by beam monitoring indicate possible improvements of redundant active detection Mostly slow losses, caused by Vacuum activities, Feedback issues and beam instabilities Very well protected by BLMs (and QPS), but E>, additional interlocks for orbit corrector current and DIDT Nota bene: Not showing dumps of BPM in IR6 following too low bunch intensity (x4) & UFO dumps (x17 - see T.Baer)

CERN LHC beam commissioning Workshop - Evian DIDT 10 Beam Current Change Monitor was vital part of MPS systems for e.g. HERA Proposed for use in LHC MPS in 2005 (EDMS Doc ) With HERA like system, changes of < 0.1% of total beam current could be captured in 10 turns BI started development (with DESY consultancy) mid 2010, for deployment end run First system installed and data recorded, but showing not understood effects of bunch length,…? Important to validate and finish soon to eventually re-design during LS1? First system now installed in IR4 First measurements during proton runs BCT DIDT Courtesy of M.Pfauwadel, D.Belohrad

CERN LHC beam commissioning Workshop - Evian Work ongoing: Novel PC Interlock System 11 Status: Server and GUI application in place. Tracking of LHC status (beam process, time within beam process) works. Acquisition & Decision-taking works. Configuration-management (storing tolerances in LSA + reusing GUI from YASP): in preparation. Currently Commissioning and Testing a) in first phase to provide warnings b) after gaining experience, possibly to connect it to the BIS Courtesy of K.Fuchsberger In addition to existing SIS interlocks at injection and SB, new SW interlock system monitoring PC currents to protect against operations- and feedback- failures Redundant to SIS for arcs, adds protection for LSS 1/2/5/8 due to capability of tracking bump shape amplitude/variations Key interest for all other (non-COD) converters where currently no current racking is done

CERN LHC beam commissioning Workshop - Evian Fill 1717: Bump >2mm during ramp in IR7 12 B1, V Corr nameInj/Rmp/SqzPhysics MCBCV.9L7.B13016 MCBCV.7L7.B13016 Eventually dumped by SIS in physics. Tolerances in µrad: Only two kicks > e.g. 15 µ rad: Distribution of RT max kick difference to BP reference during ramp  Could have been caught earlier Courtesy of K.Fuchsberger

CERN LHC beam commissioning Workshop - Evian Fill 1614: Pi-Bump (~1mm) in arc at injection 13 B2, V E.g. kick > 10 µrad: Would have given also 1 false trigger Distribution of RT max kick difference to BP reference during ramp  Bump of this size is at the limit for simple detection by ref functions and tolerances. 1 bad BPM Courtesy of K.Fuchsberger

CERN LHC beam commissioning Workshop - Evian Other MPS changes/improvements for 2012 ● BLM HV checks implemented at HW level ● New (shorter) BLMs in injection lines, providing > dynamic range ● New LBDS TSU firmware (including recommendations of external audit) Increase re-trigger delay to avoid asynchronous dumps following internal TSU failures ● Started work on redundant BIS-LBDS trigger interface ● SMP-GMT hardware cross-checker ● First draft of operational procedure in case of non-working dump ready ● Operational procedure for abort gap cleaning ● Improvements of Roman Pot Controls ● … 14

CERN LHC beam commissioning Workshop - Evian Commissioning and validation procedures 15 ● MPS commissioning and validation well established and performed according to EDMS procedures (tracking done in SharePoint site) ● Profit from HWC re-factoring to introduce more formal approach and tools including representation of dependencies and sequencing as well for beam commissioning Courtesy of M.Galetzka, K.Fuchsberger

CERN LHC beam commissioning Workshop - Evian Machine Developments 16 ● Machine Developments per definition explore new machine and machine protection territory! ● MD requestors demonstrated responsibility in proactively providing the required MP documents ● More than 20 documents approved through EDMS for MD periods in 2011 ● Preparation phase has proven very useful for MD and Machine Protection teams and often increased the efficiency of the MDs To be made better for 2012: ● rMPP classification of requests + preparations of documents tbd more timely before the MD ● Certain flexibility wrt to agreed procedure was useful to increase efficiency (BFPP test, ion quench tests,…) but must not become the default

CERN LHC beam commissioning Workshop - Evian Change Management 17 ● Some improvements to track changes in MPS systems during 2011 run ● > 90 HW and MPS SW changes tracked in MPS commissioning SharePoint Site, however… ● Requires corresponding culture and level of responsibility of equipment experts ● Even more so for SW deliverables -> Culture of best practice ● Good design ● Unit Testing ● Integration Testing ● Functional Testing ● Replaying (BI) data from the Logging DB would allow testing on test-beds without beam (e.g. during LS1) ● Tools for intelligent rollback ● Tools to centralize information on deployments/ server reboots,… ● RBAC like protection against changes on operational machines

CERN LHC beam commissioning Workshop - Evian Summary 18 LHC Machine Protection Systems have been working well during 2011 run thanks to a lot of loving care and rigor of operation crews and MPS experts Most failures are captured before effects on beam are seen (no losses or orbit movements) Still no quenches with circulating beam (with ~ 100MJ per beam and 10mJ for quenching a magnet) No evidence of major loopholes or uncovered risks, additional active protection will provide further redundancy Maintaining present good level of orbit stability is a primary importance Still we have to remain vigilant to maintain current level of safety of MPS systems while increasing efforts on increasing MPS availability

CERN LHC beam commissioning Workshop - Evian 19 Thanks a lot for your attention