1. MP3 report : W27 to W33 (22 slides, 10’ ?) J.Ph Tock for the MP3 team: TE-MPE : N. Catalan, R. Denz, M. Koratzinos, A. Verweij, M. Zerlauth TE-MSC : B. Auchmann, S. Le Naour, M. Modena, A. Musso, J.Ph. Tock + K Dahlerup, E Nowak, A Rijlart, Z Charifouline JPhT – MP3 18th August 2010 - Technical issues - Organisation update 1

2. EICs have the right to and are doing the swaps / Also sending logging MP3 has to be informed of QPS issues (written in MP3 logbook or email or …) Will run on board B before next technical stop to identify problematic channels ; only communication is affected and not the protection A new firmware is under test in the lab so this will become transparent Implementation is foreseen during the Christmas break Circuits concerned (from MP3 logbook) on > 5 weeks: RCBXV2.R2 (10), RCBXH2.R2(2), RCO.A56B1&2 (5), RCBXV&H1.R2 (3), RQS.A12B2 (5), RQTL11.L5B2 (5), RQTF.A67B2 (2), RQTF.A81B1 (5), RQTF.A56B2 (2), RQTD.A56B2(1) + some more from OP logbook Out of 2400 boards Switches from board A to board B JPhT – MP3 18th August 20102

3. 15/7 : RB.A45 : Quench heater power supply of B17.R4 discharges Replacement postponed to the day after with monitoring of the 3 other power supplies 17/7 : RB.A56 : Quench heater power supply of C34R5 discharges 20/7 : RB.A12 : Quench heater power supply of C34L2 discharges 20/7 : RB.A56 : Quench heater power supply of B11L6 discharges 23/7 : RB.A45 : Quench heater power supply of C18.R4 discharges 28/7 : RB.A45 : Quench heater power supply of B8.L5 discharges 17/8 : RB.A12 : Quench heater power supply of B24.L2 discharges Discharge of Quench Heater Power Supply JPhT – MP3 18th August 2010 The procedures were reviewed and agreed. We should not start a fill without redundancy but we may allow to loose redundancy during the fill. So, if an HDS is lost when there is beam in the machine, nothing will happen. However the injection permit is lost and no new beam will be injected … except if QPS_OK masked Was not applied yesterday … QPS_OK was masked (no mention in logbook) for injection so relying on the operator to check if no more QHPS discharges. Software will check this automatically and initiate (after EIC validation?) a slow power abort if 2 dipoles or 1 quadrupole QHPS is discharging. Implementation planned for October TS 7 cases in > 5 weeks – MTBF is 10 6 hours - > 6000 QHPS in the LHC Fault identified ; some replacement will take place. The availability should be improved by a factor of at least 10 3

4. Spurious trigger signal not related to any real event which is self curing (FILLING BUFFER --> FILLING BUFFER NOT OK --> FLT TIMEOUT --> LOGGING --> QPS-OK back) The origin of the temporary problem is related to the Global Busbar Detector, not filling the buffer, is related to a time-out and will last for 2 to 5 minutes. It has an automatic reset (6/8, 9/8,..)  In case of loss of QPS_OK, waiting 5 to 10 minutes can solve the problem 15/8: Checking QPS signal in a stable beams phase, it was noticed that QPS_OK was lost for a communication problem. Protection is still ensured. Intervention for a reset of the local main BB detector in the tunnel was planned that solved the issue  Personnal computer to be rebooted locally (access) ; will be solved during Chritmas break  Check allowed to plan the intervention and minimize its impact There is a plan to remove it during the Christmas break as nQPS is taking care of this. The information on the total voltage across the BB being a useful information, this will be kept but only in monitoring and not in the interlock Global BB detector : mains JPhT – MP3 18th August 20104

5. Request functions have been smoothened – 8 RQTD.B2 tripped ; feedback functions (13/7) – Various RQTF/D; real time trims (15/7 & 5/8) – Various RQS; new ramping-up parameters (23/7) – Various RQTF/D due to tune feedback re-enabled by mistake (1/8) – RCBV16.R4B2 (3/8) – RSF2.A23B2 and RSF.A67B2; squeeze functions(12/8) – RCBXH1.R2: Trips due to mismatch (30 mA) of the current before the start of squeeze Partial watch dog is implemented (OK for preloaded functions) Limits increased: – RQX.L/R2 & 8 : During setting for the 10A/s ramp, problem encountered with these 4 inner triplet circuits (slightly exceeded the initial commissioning value of 6A/s - needing something like 6.05 A/s). There is no limitation for a slight increase from the converter point of view so the limit was increased to 6.2A/s. Values have been changed in LSA and synchronised into the FGC. Approved by MP3 Trips due to too fast request JPhT – MP3 18th August 20105

6. Status regularly checked to plan for a reset, minimizing operation disturbance No perturbation in the last 6 weeks Autonomy > 10 days Undulators JPhT – MP3 18th August 2010 Triplets circuits Many trips due to PC when ramping down Triggers MP3 verifications of the heaters firing EPC issue under investigation to assess if solution is possible 6

7. B18R5 old QPS; Fip com lost; access required; incorreclty assembled connector (8/7) Segments 8R4 & 15 R4 of the mains: WorldFip experts accessed tunnel (10/7) RB.A12 : No communication with current leads; Access to tunnel to restart main BB detector (12/7) ; will not be used any more next year S56 global BB detector (26/7): reboot from tunnel ; will not be used any more next year DQAMGS.B11L5 : needed MPE piquet access in tunnel (10/8) Communication problems with access JPhT – MP3 18th August 20107

8. Communication lost on DQAMC.21L7 S67: cured by resetting the gateway(8/7) RCBV10.R1B1 : PC communication (8/7) RQT13.L6B1, RQT13.L6B2, RCO.A56B1 and RCO.A56B2 : (Fast) Power cycled cured it (8/7 & 15/7 & 17/7) RQTF/D in 78 : Power cycle cured it (10/7) RCO/D.A56B1 : communication lost on crate; Power cycle cured it (12/7) RQRF.A67B2 LD1 com pb (13/7); Power cycle cured it Network in CCC; lost QPS_OK of 6kA&600A in34 not visible; masked as at injection level ; came back < 15 minutes later (15/7) S67 communication lost with gateway; cured by rebooting gateway (15/7) RQD/F.A12 : Lost communication with MQ.16L2 (DQAMC.16L2) while ramping up the current. By miracle, communication came back after 25 minutes (23/7) RQT13.L6B1&2, RCO.A56B1&2: Loss of com. with the crate ; cured by power cycle (4/8) S67 mains lost due to communication problem(8/8) Loss of QPS signals in S12 ; server had to be rebooted (12/8) Loss of Loss of communication with main BB detector of RQF/D.A67 cured by sending logging (EIC) (15/8) Communication problems without access JPhT – MP3 18th August 20108

9. 10/7 : S67&78 lost: origin RQ10(9).R6 & L8 – No PM of RQ9, as on same acquisition system as RQ10, will be solved after Christmas break – Long cables and specific settings (see further) 13/7 : RQ5.L2 – Trip du to noise on U_RES ; QH discharge is OK ; During a ramp ; Noisy board that was replaced External? electrical perturbation (1/2) JPhT – MP3 18th August 20109

10. 23/7 : RQ9.R2 – Trip ; heaters fired ; U_Res over threshold but origin of perturbation not identified External? electrical perturbation (2/2) JPhT – MP3 18th August 2010 27/7 : RQ6.R1 – Trip ; heaters fired ; U_Res over threshold but origin of perturbation not identified ; – typical analysis with heater discharge and recharge checked 10

11. MP3 is 1 st line for analysis Some examples: 16/7 : RQT12.R5B2 : Trip due to fan problem on PC 18/7 : RQ6.R5 : PC trip (EPC issue) 25/7 : RCS.A45B1 : Tripped 3 times; EPC changed a power module 1/8 : RB.A56 : beam dumped by RB trip due to water fault issue in the PC. EPC intervened 7/8 : RQ8.R6 : Trip first thought to be due to noisy QPS board but finally was due to a PC component that has been changed 8/8 : RCO/D.A56B1 : Trip (<1A) during the night ; indication that possible earth current fault ; EPC intervened 16/8 : RB.A56 : Cannot be rearmed due to water fault so intervention by CV 17/8 : RCBH30.L2B2 : Circuit tripped due to a VIN DC UNDER VOLTAGE voltage 17/8 : RQT13.L2B2 : Tripped while precycling due to the PC with an internal voltage reference fault JPhT – MP3 18th August 201011

12. RSS.A34B1 Self opening of the switch (#3) ; first event of this kind in the LHC The faulty switch Z of the EE system has been replaced on 19/07 Circuit not used in between RB.A81 Trip du to nQPS board B21R8 (was changed- Broken resistor on the board) 8 th of July JPhT – MP3 18th August 201012

13. RCBCV7.L2B2 and RCBCH7.L2B1 – Test following intervention on the PC side – Resistance measured during PNO.A1 is slightly larger than the tolerance. Already observed in the hardware commissioning but very stable. – To be followed by MP3 22 nd of July JPhT – MP3 18th August 201013

14.  Trip of RB.A78 Sector trip when switched ON at 08h18m18s. To get access in L8, the EiC has put the RB on OFF without first requesting STANDBY. The result was that the current in the RB was SLOWLY decaying towards 0 (see timber plot). Access was then given with current in the RB (about 500 A) although the EiC thought there was 0 A. After access was finished, the EiC put the RB on standby. This created a huge voltage in the circuit because there was still a current of about 200 A in the circuit at the standby command. Then the main busbar detector tripped because U_RES went above the threshold and the switches were opened. Sequence was not followed This is managed in another body Not really MP3 but safety issue : 23 rd of July : RB.A78 JPhT – MP3 18th August 2010 Access with 500 A in the RB Sector trip with 200 A 14

15. Beam dump triggered by QPS The trip was caused by the spontaneous openig of a circuit breaker powering the QPS detector in 18L3. Intervention was required 2 nd occurrence this year Monday 3th August JPhT – MP3 18th August 201015

16. RCBXH1.R2 U_res signal with a significant offset tripped during pre-cycle Required an access at 02:00 to be reset from the tunnel ; will be investigated at next TS Tuesday 27 July JPhT – MP3 18th August 201016

17. Special configuration applied to these magnets to measure higher range voltage – Poor resolution on the low voltage range – High sensitivity to noise of the triggering signal Restoring the nominal settings will imply removing two resistors for each card. – MP3 recommended to test in the lab and implement 1-2 examples during the next technical stop. – Activity on-going Q10-Q9 sensitivity (to perturbations) JPhT – MP3 18th August 201017

18. Measurement of bus-bar resistance for the 6kA circuits – Reference values for time evolution and nQPS upgrade Already 14 magnets measured in the dispersion suppressor – All values around 5 nOhms for 4 to 5 splices. – Measurements require special equipment displaced and connected during the technical stops. IPQs bus-bar resistance measurements JPhT – MP3 18th August 201018

19. Hardware procedures and test plan for next shutdown/break under revision New tools on preparation – New interlocks that may lock the circuits after automatic analysis of the data – 1 st version to be released for September 2010 BLM threshold adjustment tests – A dedicated MP3 meeting will be organized for this subject. Location was already identified but needs to be confirmed. Parallel activities JPhT – MP3 18th August 201019

20. IPQs: New system for Q10R2 (Maybe Q9 & 8 too) – Will require a full re-commissioning (Thursday night?) Change of switch on QHPS : – Beginning of a campaign ; will improve availability – Recharging will be monitored / no discharge test planned – Then campaign to be completed during Christmas break Snubber capacitors (S12 or 78) – Will be installed but not connected – Connection during the Christmas break – Intervention on critical safety devices – Dedicated tests have to be foreseen, with powering MPE interventions at next Tech Stop JPhT – MP3 18th August 201020

21. Two persons on call for LHC operation 1.7:00 to 19:00 on weekdays 2.19:00 to 7:00 on weekdays and 24/24 on weekends Attendance to the 8:30 and 9:00 meetings Close follow-up of the LHC operation Shifts will be organized after technical stops Same commitment and workload for the MP3 members but resource optimization New supervision mode from MP3 (Since 28 th of July 2010) JPhT – MP3 18th August 2010 x162266 21

22. Average: 2.1/day, stable (2.0 over the last 30 days) Technical Stops – Recovery from TS Switch to supervision mode MP3 Actions JPhT – MP3 18th August 201022