GSI-CERN workshop on magnet – circuit protection and electrical quality assurance ELQA How to define test voltages Test hardware, procedures and test organisation

Outline Introduction ELQA - How to define testing voltages Test objectives ELQA overview When are tests required? ELQA - How to define testing voltages Test voltages for single circuits Test voltages taking into account adjacent circuits Concept for tests of a magnet in industry, on the test bench, in the tunnel, with / without instrumentation connected to the magnet string ELQA - Test hardware, procedures and test organisation How are the tests done? How are the tests organised? Conditions to perform a test ELQA hardware ELQA software framework Risk related to ELQA tests Findings of the tests

ELQA test objectives Tests are non invasive: low current, low power Tests are carried out at all stages of the accelerator construction and operation Early detection of problems ELQA is an important step to ensure safe operation

ELQA definition Electrical Quality Assurance is a set of tests: Insulation test (HV) Frequency transfer function measurement Polarity, continuity and resistance Magnets Current leads Instrumentation Quench heaters Entire circuits DFBs

Main ELQA tests Many additional applications and tests TP4 – Test Procedure 4 (TP3/ TP2 & TP1 were performed during machine assembly) HV and LV tests of circuits powered via DFBs 743 circuits DOC – Dipole Orbit Corrector check HV and LV tests of circuits powered locally 914 circuits MIC – Magnet Instrumentation Check HV and LV tests done locally on each magnet – it includes quench heater qualifications 1763 magnets. LS1 PAQ – Partial Qualification during Long Shutdown 1 HV and LV test following the consolidation work (SMACC) Up to 26 x 2 conductors in each sector; test is done on a daily basis. AIV – Arc Interconnection Verification LV test done after installation or replacement of magnets composing the continuous cryostat. Many additional applications and tests Details of these tests are described on EDMS: 788197, 1269114, 1269127

Simplest circuit tested with TP4 One DFB 2 current leads Magnet chain MANY other configurations need to be covered! Required checks: Electrical insulation of the circuit Continuity and resistance of the circuit Continuity and resistance of V-taps Check if V-taps are properly distributed along the CLs Verify Pt100 sensors and their specific connections ~50 x

Complete system overview A universal measurement system is needed to cover all configurations Measurements have to be automated Results must be stored in the central database Follow-up web page should be available as well

When? During the magnet manufacturing At the reception from industry Before the installation in the tunnel During the assembly of the machine After thermal cycles: if more than 100 K is reached In case of an earth fault detected by the power converter during operation After large interventions on the magnet chain or the instrumentation In case of a quench heater failure

ELQA schedule during LS1 Main part of ELQA activities was marked in black. Main part of ELQA activities was marked in orange. All possible ELQA scenarios are described in EDMS document 788197

Early detection is useful Faults will be detected during operation sooner or later Consequences of insulation faults Single fault: operation of the circuit is blocked Corrector circuits – minor issue Main circuits – major issue Voltage to ground redistribution Cascade of events Double fault: Possible serious damage Replacement of damaged components needed Detecting faults in operation translates directly into unforeseen delays of the beam for physics

Existing earth fault diagnostics Tests during commissioning, at 0 kA Reception tests (MB: 3100 V) ELQA tests (only cold part and instrumentation) At warm (RB: 700 V) During cool-down / warm-up (RB: 48 V) At cold (RB: 2100 V) Online detection during operation PC earth detection system 0 - 500 V QPS ‘voltage feelers’ (only RB, RQF and RQD circuits) Faults related to current level may remain undetected during ELQA and could pop-up during powering

Test voltage definition

HV test details Each circuit is tested vs. gnd and all other circuits Test voltage = 1.2 * MAX ((Vee + Vq), (Vee + Vee_neighbour)) At the DFB the maximum voltage will reach Vee The DFB and bus-bar system has to withstand the full test voltage to allow tests of the circuit Vee – Voltage during the energy extraction Vee_neighbour - Voltage during the energy extraction of the neighbouring circuit Vq - Voltage developed in the coil during quench

Procedure extract EDMS document: 788197 This is the max voltage seen at the DFB during operation This is the ELQA test voltage

Calculation explanation Voltage reached during energy extraction (VGPA): it is known that, in the case of a global power abort, all the circuits of a powering sub-sector are switched off almost simultaneously and energy extraction is activated in the entire subsector. The maximum voltage reached during the discharge between two circuits running along the same path is seen at the bus-bars and may be the sum of the individual voltages for both circuits VEE. Here it is assumed that the voltage rise during a magnet quench is not propagated through the bus-bar lines and it is limited to the magnet. This voltage can be calculated directly by the ultimate current and the value of the EE resistors. Voltage reached during a quench (VEEq): When a quench is detected, the QPS system will open the interlock loop and the energy extraction switches. The maximum voltage is then generated by dump resistors (VEE). This voltage seen in the circuit versus ground will be at the extremity of the electric chain VEE. If this particular magnet happens to be the magnet quenching, or inversely, if the fast energy extraction causes this magnet to quench, the maximum voltage seen by the coils would be the sum of VEE plus the voltage developed inside the magnet due to the quench Vquench. For circuits without energy extraction VEEq is equal to Vquench Voltage reached during a quench in a nested magnet (V2q): in the particular case of nested circuits, the fast current decay in one of the circuits often induces a quench in the neighbouring one. Again, the relative voltage between both circuits is the sum of both of them.

Paschen’s law 1 Torr = (101 325/760) Pa

Final ELQA values

Test voltages for quench heaters U = 1.2 x (450 + Vquench)

HV test limitations Weakness that would appear at a voltage just above the test voltage No precursors can be measured in a large system Any movement after the test may cause this fault to appear at a lower voltage: Sufficient margin with respect to the operating condition Regular testing In most cases internal (between powering lines) shorts are not detected Only for the RB each line can be tested separately For other circuits only the frequency Transfer Function Measurement might detect an internal short, but: The voltage is limited to 10 V The analysis is difficult

Voltage level general considerations Test voltage should always get lower at every production stage Better to keep some margin in case some changes are applied to the circuit/instrumentation or some new failure modes are detected (ex. VGPA) All components of the circuit need to be adapted to the test voltage: DFB, busbars, current leads etc. even if the max voltage occurs in the magnet coil.

Conditions to perform tests

Conditions to perform a test Cryogenic conditions that represent the worst case scenario for the electrical insulation Safety ensured to the personnel Lock-out of power converters No access during ELQA tests When tests are finished all circuits are grounded

ELQA framework: hardware and software

Complete system overview

Hardware The system is equipped with 84 inputs: 18 internal, 66 available on the front panel. Having 4 independent 1 of 84 multiplexers gives following possibilities: Measurements of any two voltages inside the system and on the device under test 4-wire resistance measurements Transfer function measurements using freely chosen reference impedance

ELQA software overview Firmware for µCs: written in C Communication between modules: I2C, RS-422 Communication with the host: USB VISA protocol (USB TMC or USB CDC), no need for extra drivers High level drivers for LabView Application for each test: LabView Migrating towards a uniform application model: queued state machine based on JKI state machine (https://indico.cern.ch/event/341422/) Data storage: MS Access – locally for each measurement system (device) Oracle - centrally hosted at CERN, multiple systems write results simultaneously edmsdb – production database Relational database structure, using LHCLAYOUT database as reference Follow-up webpage Data analysis tools Source code version control: CERN centrally hosted SVN, migration to gitlab possible

Debug console Apart from the standard measurement applications (TP4, DOC, MIC) there is a “debug console” Access to all the measurement instruments in the system Low level control of the multiplexers and the thermostats Usage of functionalities inaccessible for standard applications System self testing capability Advanced circuit diagnostics Designed to be used by experts only

Test contents: Software Database is the central part of the system. It stores: Reference data describing circuits Test content Validation conditions Configuration parameters Calibration parameters Results Oracle database, 100+ tables. Signal translation in low voltage test applications. Measurement results validation process.

Database structure: TP4 test Reference TP4 test results Test config Auxiliary

Main ELQA webpage Displayed information is automatically generated from the database Links to the main follow-up and management applications

Follow-up webpage

Data analysis example MIC QHR At warm after SMACC https://elqa-results.web.cern.ch/elqa-results/mic_stats/show_qhr_stat.php?campaign=157,163,165,164,158,141,160&ulimit=30&dlimit=1&nbins=80&show_dipoles=on&other_magnets=&QH= https://elqa-results.web.cern.ch/elqa-results/mic_stats/show_qhr_stat.php?campaign=140,142,148,150,144,134,146&ulimit=30&dlimit=1&nbins=80&show_dipoles=on&other_magnets=&QH=YT111 At cold after SMACC

CERN-made multichannel HV tester ELQA HV testers Max 2.5 kV Max 5 kV HV crate CERN-made multichannel HV tester Megger industrial HV tester

HV crate Max voltage = 2.4 kV The current is limited by hardware to 2 mA. 1 MΩ discharge resistor

HV test overview All circuits grounded One circuit hi-potted Leakage current precisely measured (nA precision) We are typically talking about the leakage current, as this is the value that we measure, but it will always depend on the applied voltage. Acquisition rate is about 1-2 Hz All parameters are defined in the procedure https://edms.cern.ch/document/788197/2.0 Test duration Voltage Ramp rate Cryo conditions All data is stored in a central DB and visible via the web interface http://cern.ch/elqa-results

HV test execution Multiple circuits in one location Ramp up in steps Fixed ramp rate Observation of the charging current Time on the final plateau Slow ramp may take more time than the plateau Observation of the leakage current Trip in case of exceeded current Software interlock: Current trip at flat-top Larger current trip during the ramp Independent protection by the power supply Controlled discharge using 1 MΩ resistor Possible analysis after the test Full history is stored

Example: RB.A12

RQSX3.L2

Main circuits overview included failed Each RB circuit is tested as two independent circuits: MBA and MBB

Risk related to ELQA tests

Risk for people High voltage source has a current limitation of 1.5 mA Energy stored in the capacitance of the circuit has no current limitation! Instrumentation is also under high voltage In case of a mutual coupling with other ungrounded circuits the voltage may appear in unexpected locations Access to the sector has to be blocked during the tests

Risk for equipment Too fast discharge (or a breakdown) Differential voltage in the magnet chain It is necessary to short-circuit sensitive components Bypass diodes TT sensors QPS electronics Breakdown – impossible to limit the energy dissipated into the arc Carbon trace Degradation If the voltage level is too close to the design value the insulation may degrade Ageing can be detected by the analysis of capacitance to ground Valid only if the ageing is not localised in a small section

Additional Risk Certain risk exists related to Mechanical manipulation of heavy DC cables Cryo pipes Instrumentation cables Temperature sensors

Manpower

Manpower Huge number of tests needs to be carried out CERN has established collaboration agreements with a number of institutes During the test campaigns ELQA team receives additional manpower that can reach 30 people in the hottest periods

Findings of the tests

Earth faults by type Faulty warm instrumentation cables Faulty electronic cards: Cryo temperature readout nQPS and iQPS DC warm cables Water on warm parts PCs, cables, current leads, EE switches, instrumentation Damaged capillary Metallic debris in diode containers Cold bus-bars Lyra -> MCS case – only at warm? Faulty insulation in DFB ‘pig-tails’ Spiders/routing in cryogenic lines in ARCs Spiders in connection cryostats Fast repair Warm part Repair: up to 3 months Cold part

Example: RB line breakdown

Examples of NCs

Examples of NCs Such tiny imperfections can stop the accelerator for months!

Examples of NCs: Instrumentation Resistance Check V-tap resistance too high ! Warm instrumentation cable and P10 connector to be revised. One V-tap found open at warm ! Current lead resistance: bolted connection on top of DCF to be redone

Examples of NCs: High Voltage Qualification Failure 5 groups can be identified within HV failure mode: Cable segment II is faulty. QPS controller or QPS cable are faulty. In the cold part of the circuit. Short circuit between two spools, Showed up and disappeared

Examples of NCs Quench heater electrical insulation problem, MP3 issue Insulation fault between the two circuits Circuits found open in the cold part. Circuits are condemned! Warm temp sensor of C.L. is swapped. Temporarily corrected. 60A corrector of Q31.R7 too resistive in the cold part. Circuit is not mandatory, and condemned. Magnet B30R7 has HV vs GND issue (1.6kV instead of 1.9kV). RCO.A78B2 is highly resistive; problem in the cold part. Circuit found open in the cold part. Circuit is condemned! RCO.A81B2 is open in the cold part.

Diode lead resistance 15 µΩ Why ELQA? Small probability X large number = large probability 15 µΩ Maximum safe limit for the measured device

Conclusions Electrical checks are necessary at each step of accelerator operation No shortcuts! Faults will come up if the equipment is not tested properly It is necessary to know possible failure scenarios so that the tests can be properly designed Tests carry certain risk: there is a balance between the ‘good’ and the ‘better’ Test systems need to be flexible to adapt quickly to new challenges Manpower is a non-negligible parameter

ELQA team in action