Ruben Carcagno, Sandor Feher, George Velev (team leader) MQXFS1 status meeting Summary of the MQXFS1 Investigation

Charge Dear George, Ruben and Sandor: As you may know already, testing of the MQXFS1 magnet was interrupted and the magnet is being warmed up. To investigate what and why happened and provide us with "lessons learned," I am appointing you as an investigating team with George as the team leader. The investigation team’s report should include your findings (What has happened? Were all procedures followed? …) and recommendations on i) how to proceed with repairs and ii) what needs to be done/improved to reduce probability of similar events in the future. Mike: A plan for troubleshooting, repairs and further testing shall be developed with the investigation team’s participation and can be implemented only after obtaining endorsement from the team. Thank you, Sergey __________________________________________________________ Sergey Belomestnykh Chief Technology Officer and Head of Technical Division MQXFS1 status meeting – Dec 9, 2015

Description of the event On Friday November , testing of the MQXFS1 magnet was interrupted due to what was initially characterized as an “open circuit” inside the test dewar. The “open circuit” was actually a highly resistive section (~2 MΩ) located at the negative superconducting lead section connecting the test stand to the magnet. MQXFS1 status meeting – Dec 9, 2015

Description of the event - details On November 12, early evening, there were two attempts to reach current level of 3,900 A to perform heater-induced quench studies, and in both attempts the quench detection system shutdown the power supply before reaching this current level. These quenches are performed at 20% of the Short Sample Limit (SSL) (~3900 A) as specified by LARP requirements. The first ramp reached 3,830 A before shutting down, and the second ramp reached 3,496 A. In both cases, the whole coil quench detection system shutdown the power supply because coil voltages exceeded the whole coil quench detection threshold On the next day (Nov 13 th ), it was found that the power supply interlocks could not be cleared due to the high resistance circuit, a decision to warmup the magnet was made, and this investigation started. MQXFS1 status meeting – Dec 9, 2015

Initial investigation The investigation team concluded that the high resistance section was caused by a section of the negative superconducting lead physically damaged due to overheating while ramping the magnet to perform heater-induced quench studies. This is the last step before the system (stand+magnet) be fully qualified for testing. MQXFS1 status meeting – Dec 9, 2015

Some pictures MQXFS1 status meeting – Dec 9, 2015

Investigation After several interviews and inspection of checkout and logged test data, the investigation team determined that the test stand quench protection system failed to protect the damaged superconducting lead section against a resistive condition. The superconductive lead had become resistive (quenched) because the test dewar liquid helium level inadvertently dropped, so current ramping was being conducted with uncovered current leads due to insufficient liquid helium level in the test dewar. The drop of the helium level was not observed on the computer monitors due to the failure of the liquid level monitoring system. Under these conditions, the temperature of the exposed superconducting lead increased and the lead quenched. Unfortunately, the quench detection system failed to detect this condition and to promptly take action to avoid any damage. Current kept flowing through this resistive section of the superconducting lead, resulting in overheating and physical damage. The quench propagated along the negative superconducting magnet lead towards the magnet, eventually reaching the section covered by the whole coil quench detection channel which ultimately shutdown the system before the target current of 3,900 A could be reached. MQXFS1 status meeting – Dec 9, 2015

Findings The failure of the quench detection system to protect the superconducting lead was traced to human error: a quench detection cable connected to the wrong receptacle, followed by missing identification of quench detection signal anomalies in the standard checkout data. It was found that one of the two quench detection cables was connected to the wrong receptacle, and as a result the quench detection system was monitoring the wrong voltages not only for the superconducting bus sections, but also for the copper section of the current leads. The other quench detection cable bringing the coil voltages was connected correctly. This incorrect connection was not identified during standard checkouts performed prior to ramping the magnet, although there were clear indications in the checkout data that there was something wrong with the leads quench detection signals. MQXFS1 status meeting – Dec 9, 2015

Conclusions The investigation team provided a list of recommendations to the TD Division Head in the three critical directions: technical, procedural, and organizational. Main recommendations: –The Test and Instrumentation (T&I) Department must ensure that a test stand and quench protection systems subject matter expert is available to perform critical initial tests of the quench protection system and to review and approve all checkout data before allowing the magnet test with current to proceed. –Add a second formal safety signoff addressing verification of the quench protection system and clearance to ramp the magnet to quench. This should be a joint signoff by the test facility Magnet Test Area Leader, representing the T&I department, and the test Principal Investigator, representing the Magnet Systems department. –Every time critical quench protection connectors are plugged, include a formal checkout procedure to verify that they were installed in the correct receptacles. MQXFS1 status meeting – Dec 9, 2015

Conclusions –cont. This is the first incident damaging a magnet for ~ 18 years of operation of the VMTF stand. The investigation team did not find any problems with the stand itself or the stand Quench Protection system. After the repair of the stand and magnet leads, and satisfying the short term committee recommendations, we believe that the magnet test can resume. MQXFS1 status meeting – Dec 9, 2015