Wiener crates; An installation and power supply analysis. Magnus Bjork Be/Co/Fe

Agenda How are most crates installed? How many failures? / What goes wrong? How should the crates be installed? Redundant crates. The “quick-and-dirty fix”. Conclusions and suggestions. Questions? References & Circuit breaker curves. Wiener’s report. 2

How are most crates installed? Incorrectly, due to various constraints. – Many crates are connected to power bars in the racks, this can bring several problems! – “Swiss” plugs are rated for 10 (13) Amperes, the crate is specified for maximum 16 Amperes. – At CERN “Swiss” outlets must be protected by 30 mA ground fault breakers. = Not a good place to connect high-powered industrial crates. – If any equipment on the power bar has a short circuit, all connected equipment loses power. 3

A maximum 16A crate (or several crates) connected to a “power bar” with a single 10A breaker… 4 cPCI +-12V 20A VME +-12V 10A

Three 16A crates in this one 10A “bar”… 5

Why is this working? The 16A maximum is the absolute maximum for the crate. The actual power supply need depends on the real power consumption. To get an estimate of the actual crate power used at this instant, add all the voltages x current values read (from either the front panel or) remotely via e.g. Diamon, Web, etc. divide the sum by 0,75 (power factor for heat losses in the power supply) and add ~30 Watts. (3x8W crate ventilators +Fan tray logic.) 6

Some mathematics to see clearer. 7 VME Maximum Power 1’070W / 0,75 = 1’426W + ~30W = 1’456W ~1’500W / 230V = 6,5A cPCI Maximum Power 1’310W / 0,75 = 1’746W + ~30W = 1’776W ~1’800W / 230V = 7,8A VoltageCurrentPower , VoltageCurrentPower ,

How many failures? Out of ~530 operational units the total number of PFC failures per year were; – 2009; 20 (~3,77%) (Protection diode modification done Jul-Oct 2009.) {Ref. 2} -2010; 7 (~1,32%) -2011; 11 (~2,07%) -2012, Jan -> June; 10 (~1,88%) (Total for ALL Wiener equipment; 0,96% failures in As a comparison, with CAEN the number is 3,38%) 8

What goes wrong? Quite simply; The PFC. – The PFC seems to have some weak points… – Wieslaw Iwanski, PH/ESE-BE, did some very extensive tests. {Ref. 1} – The power supplies now have years of use, normal end-of-life for certain components may be a natural consequence of aging, but for the moment no “usual increase in failures” can confirm e-o-l approaching. – MTBF for the ventilators is specified to be 6 years, the power supplies are said to “last longer”. – It should be noted that CERN specifically requested the Active PFC and the Softstart functionality. – To protect the PFC the diode upgrade was introduced. 9

10 This diode

D20 11

D14 12

How should the crates be installed? According to the OEM specifications. – Individual 16 Ampere type “B” or “C” curve circuit breakers. – Server grade power connections, preferably approved up to 16 Amperes and not needing any 30 mA ground fault circuit breakers. – Standard Burndy 10 Amperes max connector pins are acceptable, since the real maximum input current of our power supplies is either; 6,5 Amperes (VME) or 7,8 Amperes (cPCI). – In case of a short-circuit the 10 Ampere Burndy connector pins can handle the short-term current of over 16 Amperes, then the circuit breaker reacts. 13

A good power distribution 1 {Ref.3} 14

A good power distribution 2 {Ref.3} 15

16

Redundant crates Redundant crates should ideally be fed from two separate “Canalis”power distribution rails, both “Canalis” rails should be fed from UPS power supplies. The two power supplies of a redundant crate should be fed by two different phases from the two different “Canalis”. (R, S or T) The redundant crates should have automatic monitoring of the power supplies output, to be able to alarm and / or trigger automatic events in case of failure of one power supply. 17

The “quick-and-dirty fix”… Installing a small fuse in-line with the power feed to the crate power supply may mask a part of the problem but may cause other problems later. …But it has two “advantages”; it’s fast & cheap. For now. The crates have been working in the current setup for ~5 years and they still work fine. Revising the power feed can be costly and inconvenient, but for critical racks with Wiener crates installed it is a good idea, especially when considering the cost of unscheduled LHC downtime… 18

Conclusions and suggestions 1 For critical rack(s) with Wiener crate(s) a new power distribution should be done with an individual breaker box directly to the “Canalis”. Possibly without additional connectors or with server-grade hardware. The intensity and selectivity of the individual circuit breaker; 16 Ampere, “C” curve. A switch-on delay unit is not needed unless there has been previous problems powering up the whole group of high-powered racks. The main circuit breakers should be reviewed to guarantee the selectivity. 19

Conclusions and suggestions 2 The protection diode should be installed in the power supplies, if not already done. Future power supply failures can not be excluded but secondary consequences can be greatly reduced by simply following the specifications by Wiener and ensuring a proper electrical distribution. The (Active) PFC was requested by CERN since without a power factor correction the 3:rd and 9:th order harmonics might cause problems, especially if the neutral cable is not dimensioned 1-1 with the phase conductors. 20

Conclusions and suggestions 3 The softstart was also requested by CERN and is a good (and bad) reason why the installations have survived so far. The new ELMA crates can be seen as a solution to upgrade to new hardware. …But first we must install some crates in 24/7 operational use in some easy to access area, in order to validate them “in the real world”. The ELMA CPA500 power supplies also have a PFC, “Low inrush current” and softstart. 21

Questions? 22

References {Ref. 1} Wieslaw Iwanski, PH/ESE-BE – WienerTests_v40 {Ref. 2} Paul Harwood – Wiener Diode modification.pdf {Ref. 3} LHCb 23

24

25