1. MPS Commissioning and Status
Jörg Wenninger
AB Department
Operations group
Acknowledgements to all the colleagues of the LHC MPP (Machine Protection Panel) and in particular to :
B. Puccio, M. Zerlauth, B. Goddard, J. Uythoven, E. Carlier, V. Kain,
S. Redaelli, B. Dehning. M. Sapinski, R. Schmidt
LHC MAC November 2008

2. LHC Machine Protection System
Devices
LHC
Devices
LHC
Devices
Movable Devices
BCM
Beam Loss
Experimental Magnets
Collimator
Positions
Environmental
parameters
BTV screens
Mirrors
Timing
SMP
Software
Interlocks
SEQ
CCC
Operator Buttons
Experiments
Transverse Feedback
Beam
Aperture
Kickers
Collimation
System
FBCM
Lifetime
BTV
MKI
Beam
Dumping System
Beam Interlock System
Safe
Beam
Flag
Injection BIS
PIC essential
+ auxiliary
circuits
WIC
FMCM RF
System
BLM
BPM in IR6
Access
System
Vacuum
System
Timing System
(PM)
Magnets
Power Converters
Doors
EIS
Vacuum
valves
Access
Safety
Blocks
RF Stoppers
Monitors
aperture limits
(some 100)
Monitors in arcs
(several 1000)
QPS
(several 1000)
Power
Converters
~1500
AUG
UPS
Cryo OK
= systems that will be discussed

3. Commissioning Procedures
Need for appropriate and approved commissioning procedures for MPS components.
Procedures were prepared for the run, but not all of them were finished on time.
Many open points of the procedures were clarified during this first commissioning campaign.
Procedure review and completion started. To be finished early Spring
Procedures:
Powering interlock system
Fast magnet current change monitors
Warm magnet interlock system
Beam interlock system
Collimators
Beam loss monitors
Injection
Beam dumping system
Vacuum system

4. Beam Interlock Controller
Beam Interlock System
Unique Hw solution for connecting any user system (= interlock) via a copper cable.
Fiber optic variant for long links (>1.2km)
BIC (Beam Interlock Controller) boards embedded in VME chassis.
Beam Permit Loops with Frequency signals connect the BICs with the corresponding kicker system (extraction, injection, dump).
In operation at the SPS and the SPS/LHC transfer lines since 2006.
Inputs are:
maskable (with safe beam)
unmaskable
User Interfaces
User
Permit #1
#14 #2
Beam Interlock Controller
copper cables
User System #1
User System #2
User System #14
front
rear
Beam
Loops
(F.O.)

5. LHC Beam Interlock System
BICs are distributed over the LHC
Ring BICs are connected by REDUNDANT beam permit loops, two for each beam.
Connected to the beam dumping system in IR6.
Dedicated injection BICs are connected to the injection and SPS extraction kickers.

6. BIS status Sept. 10th 186 clients • • • 186 User Systems 1
CCR #
Inj1
Inj2 1
Collimation (Environmental par.)
o o 22 • 2
Collimation (Motor positions) 20 3
Vacuum system (“sector valves”)
• • 24
Vacuum system (“X valves”) 6 4
PIC for essential circuits 16
PIC for auxiliary circuits o 5
BLM at aperture limitations* 8
BLM in arcs
Fast Magnet Current Change Mon.
ooo 12 7
Warm Magnets Interlock
Screens 9
RF & Transverse Damper 10
Beam excursion (BPM) 11
LHC Beam Dumping system
Operator Buttons 13
Programmed Beam Dump (via Timing) 14
Safe Machine Parameters 15
Fast Beam current Change Monitor (BCT)
Beam Aperture Kicker 17
Injection Kicker 18
TCDQ Position
o •
• o 19
LHC Access Safety System
ATLAS (movable device) 21
ATLAS (Detector part)
ATLAS Experiment Magnet 23
ALICE (Detector part)
ALICE Experiment Magnet 25
ALICE-ZDC (movable device) 26
CMS Experiment Magnet 27
CMS (Detector part) 28
TOTEM (movable device) 29
TOTEM (Detector part) 30
LHCb (Detector part) 31
LHCb (movable device) 32
LHCb Experiment Magnet 33
LHCF (Detector part) 34
MSI Convertor Sum Fault
• Not Maskable /
o Maskable input
Total of connections
186
BIS status Sept. 10th
186 clients

7. BIS Commissioning
Phase 1: connection of clients to the interface box and verification of link.
Phase 2: validation of beam permit loops (endurance test) and optical fiber quality checks (client inputs are deactivated).
Phase 3: activation of the client signals at the level of each BIC module.
Phase 4: connection of the beam permit loops to the beam dumping system (September 2008).
With the exception of a few missing client inputs (not ready),the BIS was fully commissioned.

8. Safe Machine Parameter System
Generation of critical parameters (for example masking flags)
+ Distribution over the Timing network
+ Reception & Checks
Safe Machine Parameters Controller
Timing
Generator
Data
sources
Timing Network
Clients
The LHC Parameters are:
Beam Energy
Safe_Beam_Flags (Beam-1, Beam-2)
Beam_Presence_Flag
Stable_Beam_Flag
Movable_Device_Allowed_in_Flag
Interfaces with:
- Beam Energy Tracking syst.
Slow Beam Current Transf.
Fast Beam Current Transf.
Sequencer
Management of Critical Settings
Clients:
- BLM
Injection Kickers
Experiments
Extraction BIS
For the LHC: only ‘Beam Energy’ was fully commissioned
The SPS Parameters are :
Safe_Beam_Flag
Probe_Beam_Flag
LHC_Beam_Flag
CNGS_Beam_Flag
Interfaces with:
- Beam Energy Tracking syst.
Beam Current Transformers
Clients:
- Extraction-BIS
For the SPS: fully commissioned

9. SMP System Status
Overall status:
First operational version of the system (without full redundancy).
No verification of the distributed information.
No connection to BIS (in case of failures).
SPS part: essential for safe extraction and injection.
All flags implemented and tested.
LHC part:
Only the energy distribution could be fully tested.
There was no time to test other flags with beam.
>> Missing functionality will be ready for 2009.

10. Powering Interlocks for SC circuits (PIC)
36 PLCs for protection of the super-conducting circuits.
PLC cycle time is 1 ms, beam dump in 50 msec (fast parallel path).
Each PLC holds a configuration defining, for each connected circuit, if the circuit is :
Essential : connection to non-maskable BIC input.
Auxilary: connection to maskable BIC input.
Not interlocked with beam.
Power
Converter
QPS
PC_DISCHARGE_
REQUEST
PC_PERMIT
PC_FAST_ABORT
POWERING_FAILURE
CIRCUIT_QUENCH
DISCHARGE_
Profibus
PLC in non-radiation area
Remote I/O
(close to clients)
AUG
UPS
ABORT
User Permit
BIS
Powering
Interlock
Controller
I/O boards

11. PIC Commissioning
The central PIC system functionality that is related to circuit protection was fully commissioned during the HWC phase.
For beam operation the link to the BIS had to be tested:
Connection PIC-BIC.
Configuration data of each PIC (critical PCs, uncritical PCs).
Since the configuration is complex and will change with time ( repetition of tests) it was decided to aim immediately for automated testing:
Automated test sequences to trigger a powering failure of each PC.
Verification of consistency of PIC and BIC signals with DB reference, storage of results in DB.
>> The test sequences were implemented, tested and all interlocks were checked using those sequences in preparation for beam operation.

12. FMCMs
FMCMs (Fast Magnet Current change Monitor) provide protection against fast magnet current changes after powering failures.
Hw based on DESY version.
12 installed in the LHC, 14 in the SPS-LHC transfer lines.
All FMCMs are connected to maskable inputs – lower priority.
All transfer line FMCMs tested.
LHC FMCMs testing started late, not ready for start of beam. Testing completed in the meantime.
VME
Crate
+ CTRP (or TG8)
CPU
VIPC626
Fast
Magnet
Current
change
Monitor
RS422 link
Power Converter
BIS interface
Voltage Divider
& Isolation Amplifier
resistive
magnet

13. FMCM Test Example Zoom around step time Transfer line dipole PC:
>> Steep step programmed into the PC reference to simulate failure
FMCM interlock trigger time:
DI < 0.1 A
DI/I < 0.01% - specification : 0.1%

14. WIC : Warm Magnet Interlock System
Based on Safety* PLC
Collects inputs from thermo-switches, flow meters, red buttons, …
Gives the Power Permit for the converter.
Warm magnet
Interlock
Controller
Beam Permit
Power Converter
Status info
Power Permit
BIS interface
Several 60°C
Thermoswitches
Water Flow
Red button…
Magnet 1
Magnet 2

15. WIC Commissioning
The WIC magnet protection functionality was fully commissioned during the HWC phase.
For beam operation the link to the BIS had be tested:
Connection WIC-BIC.
Configuration data of each WIC.
Response time.
>> The WIC system was fully tested, but on a manual basis (no automated sequences).

16. Collimators The collimation system provides interlocks on :
Jaw positions
Environmental parameters (temperature)
Position interlock tests sequence :
Automated test sequences were implemented in the collimator software.
Sequence hits 12 interlock limits (inner and outer 6 LVDTs).
Sequence takes less than 5 minutes/collimator.
In total 2 x 468 checks (inner/outer)

17. Collimators : in practice
Special thanks for support by the CO/DM section.
Collimators : in practice
All information recorded in logging DB.
Scripts to build ad-hoc plots.
For the moment comparison with BIS logging is still manual  to be automated.
....
CIB.UJ33.U3.B1,False False,17 Sep :26:10 (563255), , ,COLL#MOT-b1 (A): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :26:10 (563149), , ,COLL#MOT-b1 (B): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :26:03 (293007), , ,COLL#MOT-b1 (B): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :26:03 (291536), , ,COLL#MOT-b1 (A): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :25:48 (992761), , ,COLL#MOT-b1 (A): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :25:48 (992661), , ,COLL#MOT-b1 (B): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :25:39 (202643), , ,COLL#MOT-b1 (B): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :25:39 (201178), , ,COLL#MOT-b1 (A): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :25:29 (992484), , ,COLL#MOT-b1 (A): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :25:29 (992382), , ,COLL#MOT-b1 (B): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :25:20 (142231), , ,COLL#MOT-b1 (B): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :25:20 (140904), , ,COLL#MOT-b1 (A): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :25:05 (492104), , ,COLL#MOT-b1 (A): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :25:05 (491996), , ,COLL#MOT-b1 (B): FALSE -> TRUE
CIB.UJ33.U3.B1,False False,17 Sep :24:56 (101951), , ,COLL#MOT-b1 (B): TRUE -> FALSE
CIB.UJ33.U3.B1,False False,17 Sep :24:56 (100520), , ,COLL#MOT-b1 (A): TRUE -> FALSE

18. Collimator Commissioning
All collimator interlocks were tested.
The position interlock sequences were launched and analyzed manually.
The temperature interlocks were tested by forcing sensor readings above temperature interlock threshold (~ 400 tests).
Isolated problems were found. They were (are planned to be) fixed.
Overall the test were successful and the collimation system was ready for beam on September 10th.

19. Other Movable Devices
Movable devices under machine control:
All vacuum valves tested.
Personnel protection beam stopper tested.
All beam profile screen interlocks tested.
Experiments movable devices were either not ready (roman pots, locked out) or not tested by September 19th (LHCb).
>> All tests were done manually – more automation will be helpful…

20. BLM System Status
During HWC each of the ~4000 monitors of the BLM system was tested with a radiative source to verify the entire chain from detector to the data logging:
Very usefull – allow to spot isolated channel errors.
Test to be improved with more robust/accurate source positioning
>> This test will be repeated next year for all monitors
The BLM interlocks were only activated for the start of beam operation and all channels were set as maskable for the initial low intensity beam period:
Initially noise issues on some channels (cable & grounding) – fixed.
The management of the complex BLM thresholds (with 12 time windows and 26 energy steps) was put in place, but was not fully operational for start of beam.
The BLMs were used for aperture studies and as help for beam threading.
Not enough time for detailled BLM interlock tests with beam.

21. BLM System : Aperture Measurements
BLMs performed well and were used for aperture studies during the injection test.
Online display of BLM
difference signals
Special thanks for support from the ABP aperture team.

22. First Quench with Beam : BLMs
In the early morning of August 9th during the first injection test, the first beam induced quench was provoqued during ‘aperture’ probing (beam1):
BLM signal
(beam2 side)
BLM signal
(beam1 side)
Bunch intensity of ~4109 p (½ nominal emittance) which is within the expected range.
>> reduced the commissioning beam intensity to ~2-3109 p.
In preparation of September 10th, another quench test revealed that it is possible to quench even with ~2109 protons – but very unlikely in normal operation (very large angle of incidence).
M. Sapinski
Preliminary results

23. First Quench with Beam : QPS Signal
Voltage (V) across magnet
Magnet recovers from the quench!
Heaters fired by QPS
QPS threshold
Beam impact
400 ms

24. BLM Signals - Quenches
Very different loss distributions (impact & magnet/cryostat layouts).
Analysis ongoing (BLM team).
Collimator
Quench
Quench
700m
450m

25. Beam Dumping System (LBDS)
The LBDS is the most complex and critical component of the LHC MPS.
Its commissioning started early in 2008, starting in a more or less standalone mode (simulated RF frequency, energy tracking simulator ...). The system was progressively completed and complexified all along the year, with quasi-continous controls ‚dry-runs‘.
To provide realistic conditions and decouple the LBDS from the BIS commissioning, a private beam permit loop was installed locally in IR6 with a single BIC. This proved essentially to decouple LBDS and BIS commissioning.
A lot of effort was invested into automated Internal Post Operational Checks (IPOC, based on LBDS data alone) and eXternal Post Operational Checks (XPOC, combining LBDS and beam instrumentation etc data).
A reliability run of the LBDS system was performed with ~ 20‘000 beam1 and ~ beam2 dumps that were automatically analysed (IPOC & XPOC).
The LBDS was finally connected to the BIS about one week before beam operation.

26. LBDS Reliability Run Statistics : Beam1 pulses
~ 20‘000 pulses for beam1
~ 1‘500 pulses for beam2
~5-6% 7 TeV pulses
No asynchronous beam dump
1.8 month effective running time for each system.
Beam1 pulses
(May 2008)
No critical failures were recorded.
4 kicker switch failures (one GTO disk in a stack of 10) were observed, 3 were expected. Detected at an early stage with XPOC analysis.
A breakdown of dilution kickers was observed. The cause was high vacuum pressure – vacuum interlock was not active at the time (fixed).
Very sucessful realibility test of the LBDS system, including the post operational checks which were essential for early detection of anomalies !!

27. LBDS Interlock Status I

28. LBDS Interlock Status II

29. LBDS Kickers with Beam Dump with RF off Inject, inject and dump mode
No problems seen so far
MKD kicker strength seems correct
Requires a better controlled orbit before detailed checks.
MKB dilution kicker sweep has been seen on BTVs
Debunched beam extracted in a few cases
Sweep checked – good agreement
Inject, inject and dump mode
Dump with RF off
Comparison between measured & calculated sweep with debunched beam

30. LBDS Test Status with Beam
Beam 1 (TD/UD68)
Beam 2 (TD/UD62)
Inject and dump setup (delay injection-dump >= 0 turns) OK
Circulate and dump setup
To do
Dump region aperture
Started (some phases)
Detailed kicker synchronisation
Extraction element strengths
Started (corrected MSD)
Beam instrumentation checks
Started
Interlocks (BPMSA, TCDQ, …)
Sweep waveform measurement
To do (parasitic looks OK)
Dump protection systems setup
Post Mortem and XPOC (eXternal Post Operational Check)
Tracking tests
Abort gap keeper (for injection kicker)

31. First Emergency Dump First “Emergency Dump” on Thurs 11th at 22:45:08
On 11th September 2008 during operation with circulating beam.
At 22:45:08, beam 2 was dumped by the LBDS triggered by the BIS.
The dump was caused by a water fault in the DC cables in the MQ circuit in sector 81.
This event allowed to address the performance of the interlock / machine protection systems at a very early state, as well as to understand the functionality of the post mortem (transient data) recording

32. First Dump Data from Beam Interlock System Beam Dump 561.523 ms
Beam Interlock Controller at IP6
received dump request
- 50 s later (anti clockwise signal)
- 180 s later (clockwise signal)
Beam Dump ms
Beam Interlock Controller at IP8
received dump request at ms
Data from Beam Interlock System

33. Main Commissioning Issues
Absence of dedicate machine check-out period (with all sectors ready):
Difficult / impossible to perform global tests or tests involving many sub-systems – example LBDS Beam Energy Tracking System.
Difficult to obtain appropriate test conditions.
Not possible to finalize many tests before beam.
>> ‚OK‘ because of very low intensity, but risky...
Automated test sequences proved essential – need more automated testing in the future.
The frequent cryogenics system stops (often due to primary services...) and the associated optimization of running with beam (before closed orbit established) implied frequent disconnection/reconnection of BIS inputs.
At the time of the S34 incident we were preparing for the first real MPS commissioning with beam (BLMs, collimators, LBDS...).

34. Outlook 2009 Completion and review of commissioning procedures.
Development of more automated test sequences, in particular for tests that need to be repeated for many components (like PCs).
Integration of systematic MPS tests (pre- or post operational checks) of BIS, BLMs with the main LHC sequencer.
Better online documentation of the tests: dedicated MPS elogbook, etc
We plan to repeat all MPS commissioning test:
push for automated sequences to test EACH component (large groups)
... rather perform than sample testing.
Next year we want sufficient time allocated BEFORE circulating beam to perform all necessary tests
We anticipate difficulties to re-commission BIS and LBDS due to the conflicting requirements – we are evaluating possible options.

35. Spares

36. Internal Post Operation Check
BIS IPOC
OK/
not OK
Post Mortem Analysis
(with BIS data)
Internal Post Operation Check
(BIS IPOC)
Beam Dumped….
Pre-Operation Checks
Creation of (set of) Operation checks:
Pre-Operation Checks:
In Stand-alone (several checks)
In involving connected systems
Internal Post Operation Check (IPOC)
Note: Beam Operation could re-start if BIS is obviously ready and armed. Re-arming shall be allowed ONLY if all checks are OK

37. PIC-BIS Connection

38. PIC-BIS Connection
To reduce the number of connections, all PIC crates in one IR are connected in series to 2 BIC modules.

39. BLM Signal for Injection
Example of a clean injection :
>> no loss detected along the machine (< 107 p)
Injection
Dump on TCT
left side IR5

40. BLM Interlock Signal Activation
Noisy channels had to be isolated and cured (long cables, ground loops).
noise