1. GSI-CERN workshop on magnet – circuit protection and electrical quality assurance
ELQA
How to define test voltages
Test hardware, procedures and test organisation

2. Outline Introduction ELQA - How to define testing voltages
Test objectives
ELQA overview
When are tests required?
ELQA - How to define testing voltages
Test voltages for single circuits
Test voltages taking into account adjacent circuits
Concept for tests of a magnet in industry, on the test bench, in the tunnel, with / without instrumentation connected to the magnet string
ELQA - Test hardware, procedures and test organisation
How are the tests done?
How are the tests organised?
Conditions to perform a test
ELQA hardware
ELQA software framework
Risk related to ELQA tests
Findings of the tests

3. ELQA test objectives Tests are non invasive: low current, low power
Tests are carried out at all stages of the accelerator construction and operation
Early detection of problems
ELQA is an important step to ensure safe operation

4. ELQA definition Electrical Quality Assurance is a set of tests:
Insulation test (HV)
Frequency transfer function measurement
Polarity, continuity and resistance
Magnets
Current leads
Instrumentation
Quench heaters
Entire circuits
DFBs

5. Main ELQA tests Many additional applications and tests
TP4 – Test Procedure 4 (TP3/ TP2 & TP1 were performed during machine assembly)
HV and LV tests of circuits powered via DFBs
743 circuits
DOC – Dipole Orbit Corrector check
HV and LV tests of circuits powered locally
914 circuits
MIC – Magnet Instrumentation Check
HV and LV tests done locally on each magnet – it includes quench heater qualifications
1763 magnets.
LS1 PAQ – Partial Qualification during Long Shutdown 1
HV and LV test following the consolidation work (SMACC)
Up to 26 x 2 conductors in each sector; test is done on a daily basis.
AIV – Arc Interconnection Verification
LV test done after installation or replacement of magnets
composing the continuous cryostat.
Many additional applications and tests
Details of these tests are described on EDMS: , ,

6. Simplest circuit tested with TP4
One DFB
2 current leads
Magnet chain
MANY other
configurations
need to be covered!
Required checks:
Electrical insulation of the circuit
Continuity and resistance of the circuit
Continuity and resistance of V-taps
Check if V-taps are properly distributed along the CLs
Verify Pt100 sensors and their specific connections
~50 x

7. Complete system overview
A universal measurement system is needed to cover all configurations
Measurements have to be automated
Results must be stored in the central database
Follow-up web page should be available as well

8. When? During the magnet manufacturing At the reception from industry
Before the installation in the tunnel
During the assembly of the machine
After thermal cycles: if more than 100 K is reached
In case of an earth fault detected by the power converter during operation
After large interventions on the magnet chain or the instrumentation
In case of a quench heater failure

9. ELQA schedule during LS1
Main part of ELQA activities was marked in black.
Main part of ELQA activities was marked in orange.
All possible ELQA scenarios are described in EDMS document

10. Early detection is useful
Faults will be detected during operation sooner or later
Consequences of insulation faults
Single fault: operation of the circuit is blocked
Corrector circuits – minor issue
Main circuits – major issue
Voltage to ground redistribution
Cascade of events
Double fault:
Possible serious damage
Replacement of damaged components needed
Detecting faults in operation translates directly into unforeseen delays of the beam for physics

11. Existing earth fault diagnostics
Tests during commissioning, at 0 kA
Reception tests (MB: 3100 V)
ELQA tests (only cold part and instrumentation)
At warm (RB: 700 V)
During cool-down / warm-up (RB: 48 V)
At cold (RB: 2100 V)
Online detection during operation
PC earth detection system V
QPS ‘voltage feelers’ (only RB, RQF and RQD circuits)
Faults related to current level may remain undetected during ELQA and could pop-up during powering

12. Test voltage definition

13. HV test details Each circuit is tested vs. gnd and all other circuits
Test voltage = 1.2 * MAX ((Vee + Vq), (Vee + Vee_neighbour))
At the DFB the maximum voltage will reach Vee
The DFB and bus-bar system has to withstand the full test voltage to allow tests of the circuit
Vee – Voltage during the energy extraction
Vee_neighbour - Voltage during the energy extraction of the neighbouring circuit
Vq - Voltage developed in the coil during quench

14. Procedure extract EDMS document: 788197 This is the max voltage
seen at the DFB during
operation
This is the ELQA test voltage

15. Calculation explanation
Voltage reached during energy extraction (VGPA): it is known that, in the case of a global power abort, all the circuits of a powering sub-sector are switched off almost simultaneously and energy extraction is activated in the entire subsector. The maximum voltage reached during the discharge between two circuits running along the same path is seen at the bus-bars and may be the sum of the individual voltages for both circuits VEE. Here it is assumed that the voltage rise during a magnet quench is not propagated through the bus-bar lines and it is limited to the magnet. This voltage can be calculated directly by the ultimate current and the value of the EE resistors.
Voltage reached during a quench (VEEq): When a quench is detected, the QPS system will open the interlock loop and the energy extraction switches. The maximum voltage is then generated by dump resistors (VEE). This voltage seen in the circuit versus ground will be at the extremity of the electric chain VEE. If this particular magnet happens to be the magnet quenching, or inversely, if the fast energy extraction causes this magnet to quench, the maximum voltage seen by the coils would be the sum of VEE plus the voltage developed inside the magnet due to the quench Vquench. For circuits without energy extraction VEEq is equal to Vquench
Voltage reached during a quench in a nested magnet (V2q): in the particular case of nested circuits, the fast current decay in one of the circuits often induces a quench in the neighbouring one. Again, the relative voltage between both circuits is the sum of both of them.

16. Paschen’s law
1 Torr = ( /760) Pa

17. Final ELQA values

18. Test voltages for quench heaters
U = 1.2 x (450 + Vquench)

19. HV test limitations
Weakness that would appear at a voltage just above the test voltage
No precursors can be measured in a large system
Any movement after the test may cause this fault to appear at a lower voltage:
Sufficient margin with respect to the operating condition
Regular testing
In most cases internal (between powering lines) shorts are not detected
Only for the RB each line can be tested separately
For other circuits only the frequency Transfer Function Measurement might detect an internal short, but:
The voltage is limited to 10 V
The analysis is difficult

20. Voltage level general considerations
Test voltage should always get lower at every production stage
Better to keep some margin in case some changes are applied to the circuit/instrumentation or some new failure modes are detected (ex. VGPA)
All components of the circuit need to be adapted to the test voltage: DFB, busbars, current leads etc. even if the max voltage occurs in the magnet coil.

21. Conditions to perform tests

22. Conditions to perform a test
Cryogenic conditions that represent the worst case scenario for the electrical insulation
Safety ensured to the personnel
Lock-out of power converters
No access during ELQA tests
When tests are finished all circuits are grounded

23. ELQA framework: hardware and software

24. Complete system overview

25. Hardware The system is equipped with 84 inputs:
18 internal,
66 available on the front panel.
Having 4 independent of 84 multiplexers gives following possibilities:
Measurements of any two voltages inside the system and on the device under test
4-wire resistance measurements
Transfer function measurements using freely chosen reference impedance

26. ELQA software overview
Firmware for µCs: written in C
Communication between modules: I2C, RS-422
Communication with the host: USB
VISA protocol (USB TMC or USB CDC), no need for extra drivers
High level drivers for LabView
Application for each test: LabView
Migrating towards a uniform application model: queued state machine based on JKI state machine (
Data storage:
MS Access – locally for each measurement system (device)
Oracle - centrally hosted at CERN, multiple systems write results simultaneously
edmsdb – production database
Relational database structure, using LHCLAYOUT database as reference
Follow-up webpage
Data analysis tools
Source code version control: CERN centrally hosted SVN, migration to gitlab possible

27. Debug console
Apart from the standard measurement applications (TP4, DOC, MIC) there is a “debug console”
Access to all the measurement instruments in the system
Low level control of the multiplexers and the thermostats
Usage of functionalities inaccessible for standard applications
System self testing capability
Advanced circuit diagnostics
Designed to be used by experts only

28. Test contents: Software
Database is the central part of the system. It stores:
Reference data describing circuits
Test content
Validation conditions
Configuration parameters
Calibration parameters
Results
Oracle database, 100+ tables.
Signal translation in low voltage test applications.
Measurement results validation process.

29. Database structure: TP4 test
Reference
TP4 test results
Test config
Auxiliary

30. Main ELQA webpage
Displayed information is automatically generated from the database
Links to the main follow-up and management applications

31. Follow-up webpage

32. Data analysis example MIC QHR
At warm after SMACC
At cold after SMACC

33. CERN-made multichannel HV tester
ELQA HV testers
Max 2.5 kV
Max 5 kV
HV crate
CERN-made multichannel HV tester
Megger
industrial HV tester

34. HV crate Max voltage = 2.4 kV
The current is limited by hardware to 2 mA.
1 MΩ discharge resistor

35. HV test overview All circuits grounded One circuit hi-potted
Leakage current precisely measured (nA precision)
We are typically talking about the leakage current, as this is the value that we measure, but it will always depend on the applied voltage.
Acquisition rate is about 1-2 Hz
All parameters are defined in the procedure
Test duration
Voltage
Ramp rate
Cryo conditions
All data is stored in a central DB and visible via the web interface

36. HV test execution Multiple circuits in one location Ramp up in steps
Fixed ramp rate
Observation of the charging current
Time on the final plateau
Slow ramp may take more time than the plateau
Observation of the leakage current
Trip in case of exceeded current
Software interlock:
Current trip at flat-top
Larger current trip during the ramp
Independent protection by the power supply
Controlled discharge using 1 MΩ resistor
Possible analysis after the test
Full history is stored

37. Example: RB.A12

38. RQSX3.L2

39. Main circuits overview included failed
Each RB circuit is tested as two independent circuits: MBA and MBB

40. Risk related to ELQA tests

41. Risk for people High voltage source has a current limitation of 1.5 mA
Energy stored in the capacitance of the circuit has no current limitation!
Instrumentation is also under high voltage
In case of a mutual coupling with other ungrounded circuits the voltage may appear in unexpected locations
Access to the sector has to be blocked during the tests

42. Risk for equipment Too fast discharge (or a breakdown)
Differential voltage in the magnet chain
It is necessary to short-circuit sensitive components
Bypass diodes
TT sensors
QPS electronics
Breakdown – impossible to limit the energy dissipated into the arc
Carbon trace
Degradation
If the voltage level is too close to the design value the insulation may degrade
Ageing can be detected by the analysis of capacitance to ground
Valid only if the ageing is not localised in a small section

43. Additional Risk Certain risk exists related to
Mechanical manipulation of heavy DC cables
Cryo pipes
Instrumentation cables
Temperature sensors

44. Manpower

45. Manpower Huge number of tests needs to be carried out
CERN has established collaboration agreements with a number of institutes
During the test campaigns ELQA team receives additional manpower that can reach 30 people in the hottest periods

46. Findings of the tests

47. Earth faults by type Faulty warm instrumentation cables
Faulty electronic cards:
Cryo temperature readout
nQPS and iQPS
DC warm cables
Water on warm parts
PCs, cables, current leads, EE switches, instrumentation
Damaged capillary
Metallic debris in diode containers
Cold bus-bars
Lyra -> MCS case – only at warm?
Faulty insulation in DFB ‘pig-tails’
Spiders/routing in cryogenic lines in ARCs
Spiders in connection cryostats
Fast repair
Warm part
Repair: up to 3 months
Cold part

48. Example: RB line breakdown

49. Examples of NCs

50. Examples of NCs
Such tiny imperfections can stop the accelerator for months!

51. Examples of NCs: Instrumentation Resistance Check
V-tap resistance too high !
Warm instrumentation cable and P10 connector to be revised.
One V-tap found open at warm !
Current lead resistance: bolted connection on top of DCF to be redone

52. Examples of NCs: High Voltage Qualification Failure
5 groups can be identified within HV failure mode:
Cable segment II is faulty.
QPS controller or QPS cable are faulty.
In the cold part of the circuit.
Short circuit between two spools,
Showed up and disappeared

53. Examples of NCs Quench heater electrical insulation problem, MP3 issue
Insulation fault between the two circuits
Circuits found open in the cold part. Circuits are condemned!
Warm temp sensor of C.L. is swapped. Temporarily corrected.
60A corrector of Q31.R7 too resistive in the cold part. Circuit is not mandatory, and condemned.
Magnet B30R7 has HV vs GND issue (1.6kV instead of 1.9kV).
RCO.A78B2 is highly resistive; problem in the cold part.
Circuit found open in the cold part. Circuit is condemned!
RCO.A81B2 is open in the cold part.

54. Diode lead resistance 15 µΩ Why ELQA?
Small probability X large number = large probability
15 µΩ
Maximum safe limit for the measured device

55. Conclusions
Electrical checks are necessary at each step of accelerator operation
No shortcuts!
Faults will come up if the equipment is not tested properly
It is necessary to know possible failure scenarios so that the tests can be properly designed
Tests carry certain risk: there is a balance between the ‘good’ and the ‘better’
Test systems need to be flexible to adapt quickly to new challenges
Manpower is a non-negligible parameter

56. ELQA team in action