1. Reliability Analysis of the LHC Beam Dumping System Taking Into Account the Operational Experience during LHC Run 1 Roberto Filippini CERN ATS Seminar, 4 th July 2013

2. Reliability Analysis of the LHC Beam Dumping System Taking Into Account the Operational Experience during LHC Run 1  Outline  Machine protection and LBDS  Background to present state  LBDS statistics 2010-2012  Validation of reliability predictions  Safety  Final recommendations and conclusions 4 July 2013R. Filippini CERN ATS Seminar2

3. The LHC Beam Dumping System  The LBDS is the final element of the protection chain, it performs the extraction of the beams on demand (dump requests) either at the end of machine fills or because of safety reasons. Two LBDS exist, one per beam. LBDS physical layout – point 6 Internal ILK Control References 10 MKB Interlocks and references Control and supervision Actuation 15 MSD 15 MKD External DR TDE Beam Line Functional layout 350 MJ destructive power 4 July 2013R. Filippini CERN ATS Seminar3

4. Machine Protection and LBDS  The LHC machine protection system MPS allows operation with the beams only if the LHC is cleared from faults/errors, and it supervises its functioning in order to prevent that a failure may develop into a critical accident. LHC State ILK Beam Operation LBDS SupervisionSafety logic Actuation Is it reliable, safe? 4 July 2013R. Filippini CERN ATS Seminar4

5. Machine Protection System 2003-2006  The reliability sub-working group of the machine protection system working group was charged to perform the analysis of safety and availability of the most critical systems of the MPS  The scope  All active devices, supervision and interlocking elements including the Beam Loss Monitors, Quench Protection System, Beam Interlocking Systems, Power Interlock System, LBDS. B. Todd, MP Workshop Annecy 2013 Most results confirmed, with a few exceptions 4 July 2013R. Filippini CERN ATS Seminar5 Reliability w.g. 2006

6. LBDS Reliability Analysis 2003-2006  The scope  Magnets, control and supervision, including the triggering system TSDS, the beam energy tracking BETS, the septa MSD, extraction kickers MKD and dilution kickers MKB. Passive protection elements not in the scope.  The assumptions for the analysis  Operation profile of 10 hours, 400 machine fills, 200 days of operation  Only random faults in system components  Post mortem diagnostics returns the system to an “as good as new” state  The goal of the analysis  Predict the probability of the LBDS to fail unsafely  Predict the number of false beam dumps 4 July 2013R. Filippini CERN ATS Seminar6

7. LBDS Reliability Analysis 2003-2006 (2)  Safety =1.8E-07 per year of operation = largely SIL4  Most critical component was the MKD with a 74% contribution  Number of false beam dumps was 8 +/- 2  The biggest contribution was from the MKD with 5 false beam dumps in total (61%) Magnets 4 July 2013R. Filippini CERN ATS Seminar7 Predictions from theoretical models! %

8. … 7 years later  3 years of LHC operation (2010-2012) with beam and a significant number of data collected among which system faults and repair/diagnostics interventions  The sources are LHC-OP logbook, and LHC-TE/ABT expert logbook  The scope  MKD, MKB and the related control and supervision electronics plus post mortem diagnostics and auxiliary systems on which the LBDS depends  The goals  Produce time series and general statistics  Estimate safety and number of false beam dumps  Validate the reliability prediction models  Discover unforeseen failure mechanisms and scenarios… 4 July 2013R. Filippini CERN ATS Seminar8

9. Time series 2010-2012 Jan 2010Dec 2012 Jan 2010 Dec 2012 1 2 3 4 July 2013R. Filippini CERN ATS Seminar9 Anomalies 1 Vacuum and BEM Anybus® 2 Vacuum and diagnostics 3 SCSS Asibus® Statistics per month

10. LBDS failure distribution vs. functions  139 failure events recorded of which 90 in the LBDS  Actuation (MKD, MKB) is the largest contributor (60%) 90 49 4 July 2013R. Filippini CERN ATS Seminar10

11. LBDS false dumps vs. machine phase  A total of 97 events during 2010-2012 triggered a false dump (with or without the beam) of which 66 from the LBDS, i.e. 73% of the total  The most important contributor is the actuation (MKD, MKB) 66 4 July 2013R. Filippini CERN ATS Seminar11

12. Actual availability  Assumptions  Only LBDS false beam dumps in the phases injection and stable beam are considered  No repetition of the same internal dump request, i.e. occurrence of the same event (e.g. inaccurate diagnostics) after a short interval => 5 false dumps not considered.  Results  The LBDS counted 29 false beam dumps, against the 24 (on average) foreseen.  The most important contributor is the actuation (15) then surveillance (12) and control (2) Decreasing trend 4 July 2013R. Filippini CERN ATS Seminar12 False beam dumps

13. The statistical analysis framework Raw data 2010-2012 IN PHASE 1 – Censoring data Predictions 2003-2006 Reliability models OUT PHASE 2 – Statistics and validation 4 July 2013R. Filippini CERN ATS Seminar13

14. Reliability prediction: Actuation function Failure mode and identifier # components Time to Failure Hypothesis test 4 July 2013R. Filippini CERN ATS Seminar14 Validation most conservative value is kept Time to recovery

15. Control and surveillance functions 4 July 2013R. Filippini CERN ATS Seminar15 Not validated

16. Failure on demand Average failure rate 4 July 2013R. Filippini CERN ATS Seminar16

17. Failure Dependency 4 July 2013R. Filippini CERN ATS Seminar17

18. Hypothesis test 4 July 2013R. Filippini CERN ATS Seminar18

19. Results at a glance  2518 LBDS components exposed to failures during 2010-2012 resulted in 90 failure events, distributed in 29 different failure modes…  …but 70 failure modes never occurred Hypothesis test always true with the exception of the PTM power supply that was expected to fail The most conservative TTF was taken 4 July 2013R. Filippini CERN ATS Seminar19

20. What about safety?  All beams were safely dumped at every beam dump request for LBDS during 2010-2012 => this is a necessary but not sufficient condition to state that the system is safe, e.g. SIL3 at least  How to get a sufficient condition for safety without a piece of evidence? 1. Deductive reasoning => as availability and safety were estimated from the same model, and availability agreed with observations, then also safety will. Based on intuition, no statistics proof 2. Inductive reasoning => the failures may have moved the system very close to the accident…  …How close? a metric for the safety distance is needed 4 July 2013R. Filippini CERN ATS Seminar20

21. UNSAFE Degraded NON acceptable Degraded acceptable Safety margins  From the initial nominal state the LBDS may tolerate a number of failures, some of which are detected and cause the anticipated beam dump  The safety principle => never operate at a single point of failure Nominal state Zero safety margin near miss/single point of failure 1 safety margin Min safety requirement 2 safety margin Min safety requirement 4 July 2013R. Filippini CERN ATS Seminar21

22. LBDS and safety by design  The behavior at failure of the LBDS is conceived in order to …  Tolerate faults by redundancy => fault masking  Prevent faults by surveillance => failsafe Dump request Primary Synchronized trigger TSU-a TSU-b TFO-a TFO-b RT-b RT-a Back-up a Asynchronous re-trigger TSDS (simplified) Back-up b Asynchronous re-trigger Beam dump 4 July 2013R. Filippini CERN ATS Seminar22

23. Example: TSDS and safety distance  Simplified state transition diagram of the TSDS  Some failure events may be detected and trigger a false dump UNSAFE TSU-b fails TSU-a fails Safety distance = number of failure events left 1 2 3 Initial state 4 July 2013R. Filippini CERN ATS Seminar23 TFO-a False dump TFO-b False dump TSU-a fails RT-b False dump

24. The safety gauge The nominal dump has to occur with the LBDS as good as new The false beam dump is justified, when the tolerance margins are exceeded  LBDS and the safety policy  The trade-off between safety and availability can be translated into an ideal safety distance at a dump request for the LBDS 4 July 2013R. Filippini CERN ATS Seminar24

25. Safety distance versus functions Control function (TSDS) is the closest to the safety margins Surveillance function is unbalanced against availability (unnecessary over-protection) 4 July 2013R. Filippini CERN ATS Seminar25 Remark: Every failure event is assigned a safety distance

26. LBDS safety gauge 2010-2012 2.77 2.13 3.39 Ideal behaviour Poor safety margins Over-protected at detriment of availability 4 July 2013R. Filippini CERN ATS Seminar26 Ideal behaviour

27. LBDS safety gauge 2010-2012 (2) 2.77 2.13 3.39  The trend unveils a loss of safety margins in 2011, and a recover in 2012, almost back to the initial level of 2010 4 July 2013R. Filippini CERN ATS Seminar27 Safety distance Most failures at safety margin 0

28. Actual safety (1)  Problem statement  Given the average safety distance at failures for each LBDS function, over the period 2010- 2012, the objective is to calculate the maximum component failure rate below which LBDS is SIL3, for 300 days per year (total = 21600 h) with an average machine fill of 10 hours  Data…  P E = probability of the initiating events (90/21600)  N = 1674 number of components at failures in the LBDS  s = safety distance  d = detection rate; 0.73 for LBDS, 0.6 (actuation), 0.87 (control) 0.96(surveillance) safety distance s Initiating event P E 4 July 2013R. Filippini CERN ATS Seminar28 d Unsafe Failsafe d

29. Actual safety (2) residual safety margin Initiating event SIL3 bounds Continuous Poisson CDF SIL3 = 1xE-07/h 4 July 2013R. Filippini CERN ATS Seminar29 Probability of exceeding the safety margin s The failure rate threshold

30. Actual safety (3)  Actuation, control, and surveillance functions meet the safety requirements individually and together as LBDS  Example: LBDS SIL3 bound is 2.5 E-05/h - the highest rate is from the TSDS VME crate power supply failure =1.9 E-05/h with all other components being more reliable. SIL3 bound highest failure rate 1.9 E-05/h close to SIL4 bound 4 July 2013R. Filippini CERN ATS Seminar30 SIL4 bound

31. Actual safety (3)  Extreme outcomes and singularities  failure events that moved the LBDS to a potentially unsafe state, or close to it (near miss) before this was discovered.  1 erratic trigger of 2 MKDs over three years, from 30 independent TFO outputs  The maximum failure rate threshold in order to be SIL3 at least is 7.2 E- 05/h which is met.  2 failure at zero safety margin (detected) in the actuation and control functions, in 3 years  The maximum failure rate threshold for the control is 7.8 E-05. and the one for the actuation is 1.1 E-03, which are both SIL3 at least. 4 July 2013R. Filippini CERN ATS Seminar31

32. Wrap up and conclusions  This presentation provided a complete and very diverse overview on the behavior of the LBDS at failures during years 2010-2012  The added values are…  The LBDS statistics  Design insights  The methodologies of analysis, validation of reliability prediction models and statistical inference of availability and safety  Summary of results and recommendations follow… 4 July 2013R. Filippini CERN ATS Seminar32

33. Results to retain  Component statistics  139 failure events in 3 years of which 90 in the LBDS, distributed in 29 different failure modes (3 out of scope) of which 19 confirmed and 7 new failure modes  Availability and safety  29 false beam dumps in 3 years, in agreement with the 24 predicted in 2006  Safety is SIL3 at least, in agreement with 2006 predictions  Remarks…  A positive trend along the years is discovered in the number of false beam dumps, and margin of safety. 4 July 2013R. Filippini CERN ATS Seminar33

34. Recommendations (1)  Further investigations on failure mechanisms  Common Cause Failure suspected in a few components such as the failure of three High Voltage power supplies in the MKD generators, two Triggering Units not responding, and the spurious firing of two Trigger Fan Out units. Further analysis on CCF and consequences on reliability is recommended.  Availability concerns  7 false beam dumps are from the vacuum  12 failures from post mortem and diagnostics => cause of delays in re-arming  Diagnostics was not always accurate, faults fixed after several interventions  Some functions might be over-protected, e.g. LBDS surveillance  Safety concerns  SIL3 is largely met for LBDS, SIL4 possible but further analysis is recommended  The control functions of the LBDS (TSDS) is estimated to have the smallest safety margin.  HW changes during LS1 in TSDS (controls) and powering.  A reliability analysis of the TSDS with the applied changes is under preparation. 4 July 2013R. Filippini CERN ATS Seminar34

35. Recommendations (2)  Data quality  Good and large quantity, but inconsistencies existed as well as non- homogeneities in the data reporting, time stamps, consequences from diagnostics and intervention  Improvements during the years should be consolidated by the definition of standard procedures of data reporting and tools for the automatic information retrieval  Product assurance  Several components did not meet the reliability specification because of design flaws, and were returned to the manufacturer (e.g. Asibus®, Power trigger power supply).  Other issues  Maintenance, and diagnostics had a relevant impact on operation  A number of faults/errors are procedural (human factor) and should be taken into account for a more detailed analysis 4 July 2013R. Filippini CERN ATS Seminar35

36. Conclusions  Where we were…  Reliability and safety prediction from theoretical models => good guesses but lack of evidence  Where we are now  The statistical analysis of the LBDS based on the 2010-2012 LHC operations proved that the reliability predictions are in good agreement with the observations.  Where we may go…  The positive trends over the years show that an infant mortality period is over, and statistics are improving => more stable system, safer and better exploitable  and the lesson learnt from this study…  A periodical review of reliability of machine protection elements is recommended and should be integrated in the system life-cycle  Tools must be developed in order to collect data and infer statistical trends  An excerpt from this work will be presented at the conference ICALEPCS 2013  R. Filippini, E. Carlier, N. Magnin, J. Uythoven “Reliability Analysis of the LHC Beam Dumping System Taking Into Account the Operational Experience During LHC Run 1”, 5-11/10 2013 4 July 2013R. Filippini CERN ATS Seminar36

37. Acknowledgements My special thanks to Jan Uythoven, Etienne Carlier and Nicolas Magnin for their support during the preparation of this work … and all friends and colleagues with whom I had chats and discussions, and who made my second experience at CERN particularly enjoyable and gratifying. 4 July 2013R. Filippini CERN ATS Seminar37

38. …question time Contact Roberto Filippini email: rob.filippini@tiscali.it 4 July 2013R. Filippini CERN ATS Seminar38

39. Safety: SIL3,SIL4 graphical tests 4 July 2013R. Filippini CERN ATS Seminar39 Failure rate/h Probability T=10 SIL4 bound SIL3 bound surveillance control actuation LBDS