Report from the LHC BLM System Audit1 B. Dehning CERN AB/BI 16.07.2008 LTC, B.Dehning

Content System Overview Auditors Scope Auditors report content Consequences of review to be treated: before start-up during first time with beam next shutdown after next shutdown Summary 16.07.2008 LTC, B.Dehning

System Overview Tunnel: Surface VME cards: Database and Software: Beam energy Beam energy detector CFC elec. Threshold comp. Combiner BIC LBDS Tunnel Surface Tunnel: Detector: ionisation chamber + Secondary emission monitor Signal digitalisation and transmission by CFC card Surface VME cards: Signal reception and threshold comparison Combination of beam permit signals and surveys tasks Database and Software: Detailed system information in MTF Detailed installation description in Layout Reference settings for hardware are in LSA 16.07.2008 LTC, B.Dehning

Software Overview, Management of Settings Safety given by: Comparison of settings at DB and front-end Safe transmission of settings 16.07.2008 LTC, B.Dehning

LHC Beam Loss Monitor System (BLM) Report on the Audit held in June 2008 Stefan Lüders (IT/CO) on behalf of the Auditors Miguel Anjo (IT/DM), Joachim Bächler (PH/DT), Philippe Farthouat (PH/ESE), Stefan Haas (PH/ESE), Stefan Lüders (IT/CO), Javier Serrano (AB/CO) 2008/7/1 2008/7/1 BLM Audit Report BLM Audit Report 5

Scope This audit is supposed to verify design & implementation of the BLM: Fundamental design decisions PCB schematics & layouts, FPGA programming Interface to the Beam Interlock Systems (BIS) Particular focus put on safety relevant aspects: Safe and efficient operation of the LHC Sufficiently high reliability and availability Management of threshold values Single points of failures AND failure modes leading to blind faults This audit did not cover In-depth verification of the FMECA analysis Placement of the ionization chambers System software running on PowerPC & high-level control systems 2008/7/1 2008/7/1 BLM Audit Report BLM Audit Report 6

Auditors Report Content Recommendations by the Auditors 5.1 Determination and Management of Thresholds 5.1.1 Simulations of Loss Signals and Response Functions (6 items) 5.1.2 Management and Storage of Threshold Values (10 items) 5.2 Monitors & Electronics 5.2.1 Ionisation Chambers (2 items) 5.2.2 PCBs and Choice of Components (9 items) 5.2.3 FPGA Programming (5 items) 5.3 Environmental Aspects (5 items) 5.3.1 Electromagnetic Compatibility (EMC) (3 items) 5.4 Commissioning, Testing, and Documentation (6 items) Next pages: In green first comments to the recommendations of the auditors. 16.07.2008 LTC, B.Dehning

Consequences of review treated before start-up Simulation 1. It has not been clear to which extend all possible beam loss scenarios were fully computed. The auditors would like to encourage the BLM team to summarize the results of the simulation studies and measurements done so far. The simulated beam loss scenarios need to be compared with the observed loss locations (fellow will be request). Summarizing simulation is ongoing (implementation of thresholds started). Management & Storage of Thresholds 7. Documentation must be produced for the procedures on how the initial values of the Master Table are defined, how the values can be altered, and how these changes are propagated. Ongoing. 8. An application should be deployed that provides means to minimize the introduction erroneous values to this table, e.g. through human errors. Will be done. 10. An application should be deployed to safely handle the “maskable” and “disable” flags in the Master Table. Will be done. Procedures 42. It is recommended to take benefit of this and start as soon as possible full scale test including the full BLM read-out chain. 43. It is encouraged to expand those tests as soon as possible including the BIS (e.g. in point 6 or 8). Large scale test with all monitors in database and all monitors logged is starting now. 16.07.2008 LTC, B.Dehning

LSA Data Base Structure Two layers entry layer (stage tables) validated layer (final tables) Concept of Master and Applied table – Comparison of Threshold values (Applied < Master) Master: less frequent changes Applied: change of thresholds possible with user interface 300 families 4000 channels 16.07.2008 LTC, B.Dehning

Consequences of review treated before start-up Simulation 1. It has not been clear to which extend all possible beam loss scenarios were fully computed. The auditors would like to encourage the BLM team to summarize the results of the simulation studies and measurements done so far. The simulated beam loss scenarios need to be compared with the observed loss locations (fellow will be request). Summarizing simulation is Ongoing (implementation of thresholds started). Management & Storage of Thresholds 7. Documentation must be produced for the procedures on how the initial values of the Master Table are defined, how the values can be altered, and how these changes are propagated. Ongoing. 8. An application should be deployed that provides means to minimize the introduction erroneous values to this table, e.g. through human errors. Will be done. 10. An application should be deployed to safely handle the “maskable” and “disable” flags in the Master Table. Will be done. Procedures 42. It is recommended to take benefit of this and start as soon as possible full scale test including the full BLM read-out chain. 43. It is encouraged to expand those tests as soon as possible including the BIS (e.g. in point 6 or 8). Large scale test with all monitors in database and all monitors logged is starting now. 16.07.2008 LTC, B.Dehning

Consequences of review treated during first time with beam Simulation 5. Errors between simulation and measurements of up to 50 % are observed, therefore, and due to the aforementioned points, the auditors share doubts that the monitors would guarantee a safe and efficient operation of the LHC without a re-adjustment of the thresholds. Will be possible if needed. 6. The initial threshold settings have to be sufficiently conservative in order not to damage the LHC magnets. During the initial runs of the LHC, they must then be iteratively adjusted. Dedicated (threshold) test procedure should be proposed by the BLM team. Sufficient time should be assigned to make those tests. For example, tests with provoked beam losses should be conducted in order to verify the proper detection of those beam losses. To be decided. 16.07.2008 LTC, B.Dehning

Consequences of review treated during next shutdown (I) Simulation 2. Saturation effects should be studied in more detail, and the limits of the current monitor design should be summarized. Saturation studies: measurements ongoing (CNGS, last week SPS MD), simulation studies to be decided. Management & Storage of Thresholds 15. Means should be investigated for merging and combining the “MTF” and “Layout” databases. Final analysis to be started (criteria: protection, possible GUIs, update speed, availability). 16. An SLA or MoU (service level agreement / memorandum of understanding) stating the responsibilities of the IT department in case of database failure is recommended. 17. A similar SLA/MoU should be set-up with the AB/CO/DM section. To be decided. PCP & Components 20. Currently, the BLM holds about 5% of spares for the major PCBs. It is suggested to increase this stock to at least 10% including spares Initiated. 16.07.2008 LTC, B.Dehning

Consequences of review treated during next shutdown (II) FPGA Code 31. In order to ensure complete testing of future changes in FPGA designs, a PASS/FAIL set of regression tests should be designed. Started, will need significant resources of design engineers during the next year. 32. In addition, a more complete review of the FPGA designs should be conducted once these have been finalized. follow-up of review planned beginning of next year. Procedure For bug-tracking, further development, and future upgrades, the auditors recommend setting up a “vertical slice”-test bench, which covers both types of monitors (ionization chambers and SEMs), the full chain of read-out electronics as well as a test database. LHC SPS system is existing, but differences to the LHC system will remain, therefore a LHC test system is needed. 16.07.2008 LTC, B.Dehning

Consequences of review treated after next shutdown EMC & Radiation 37. Independent tests should be conducted to verify the consequences of a total ionizing dose (TID), non ionizing energy loss (NIEL) and flux of particles able to produce single event effects (SEE). 39. It is recommend performing a more in depth analysis of the effects of potential SEUs on the behavior of the CFC and it’s FPGA. 40. SEEs in the power supplies of the arc and in the straight sections can lead to their complete failure. The failure rate should be determined and the power supplies should be verified to sustain the radiation. All to radiation exposed components have been tested in proton beams, some extensively some less. A more detailed understanding and the long term failure rate needs a better determination. 16.07.2008 LTC, B.Dehning

Summary Design and implementation of the BLM is sound, complete, straight-forward, and, conform to requirement on high inherent level of safety, reliability and availability (SIL3) . The auditors are convinced that the absolute precision will meet the BLM requirements being within a factor 5 (initially) and 2 (later). The data-driven approach is a good implementation choice. However: Iterative tests using real beams needed to finalize threshold values. Threshold management in DBs require better documented procedures. Additional tools needed for protecting Master table (e.g. access control, avoiding erroneous values, setting “maskable” and “disable” flags). The spare part contingency of 5% is too low. Set-up of a “vertical slice”-test bench is recommended. 2008/7/1 2008/7/1 BLM Audit Report BLM Audit Report 15

Remarks Audit documentation can be found at: http://cern.ch/blm Several recommendations require a higher level of maintenance efforts as for non safety system needed 16.07.2008 LTC, B.Dehning