1. Report from the LHC BLM System Audit1
B. Dehning
CERN AB/BI
LTC, B.Dehning

2. Content System Overview Auditors Scope Auditors report content
Consequences of review to be treated:
before start-up
during first time with beam
next shutdown
after next shutdown
Summary
LTC, B.Dehning

3. System Overview Tunnel: Surface VME cards: Database and Software:
Beam energy
Beam energy
detector
CFC elec.
Threshold comp.
Combiner
BIC
LBDS
Tunnel
Surface
Tunnel:
Detector: ionisation chamber + Secondary emission monitor
Signal digitalisation and transmission by CFC card
Surface VME cards:
Signal reception and threshold comparison
Combination of beam permit signals and surveys tasks
Database and Software:
Detailed system information in MTF
Detailed installation description in Layout
Reference settings for hardware are in LSA
LTC, B.Dehning

4. Software Overview, Management of Settings
Safety given by:
Comparison of settings at DB and front-end
Safe transmission of settings
LTC, B.Dehning

5. LHC Beam Loss Monitor System (BLM)
Report on the Audit held in June 2008
Stefan Lüders (IT/CO) on behalf of the Auditors
Miguel Anjo (IT/DM), Joachim Bächler (PH/DT), Philippe Farthouat (PH/ESE), Stefan Haas (PH/ESE), Stefan Lüders (IT/CO), Javier Serrano (AB/CO)
2008/7/1
2008/7/1
BLM Audit Report
BLM Audit Report 5

6. Scope
This audit is supposed to verify design & implementation of the BLM:
Fundamental design decisions
PCB schematics & layouts, FPGA programming
Interface to the Beam Interlock Systems (BIS)
Particular focus put on safety relevant aspects:
Safe and efficient operation of the LHC
Sufficiently high reliability and availability
Management of threshold values
Single points of failures AND failure modes leading to blind faults
This audit did not cover
In-depth verification of the FMECA analysis
Placement of the ionization chambers
System software running on PowerPC & high-level control systems
2008/7/1
2008/7/1
BLM Audit Report
BLM Audit Report 6

7. Auditors Report Content
Recommendations by the Auditors
5.1 Determination and Management of Thresholds
5.1.1 Simulations of Loss Signals and Response Functions (6 items)
5.1.2 Management and Storage of Threshold Values (10 items)
5.2 Monitors & Electronics
5.2.1 Ionisation Chambers (2 items)
5.2.2 PCBs and Choice of Components (9 items)
5.2.3 FPGA Programming (5 items)
5.3 Environmental Aspects (5 items)
5.3.1 Electromagnetic Compatibility (EMC) (3 items)
5.4 Commissioning, Testing, and Documentation (6 items)
Next pages: In green first comments to the recommendations of the auditors.
LTC, B.Dehning

8. Consequences of review treated before start-up
Simulation
1. It has not been clear to which extend all possible beam loss scenarios were fully computed. The auditors would like to encourage the BLM team to summarize the results of the simulation studies and measurements done so far. The simulated beam loss scenarios need to be compared with the observed loss locations (fellow will be request). Summarizing simulation is ongoing (implementation of thresholds started).
Management & Storage of Thresholds
7. Documentation must be produced for the procedures on how the initial values of the Master Table are defined, how the values can be altered, and how these changes are propagated. Ongoing.
8. An application should be deployed that provides means to minimize the introduction erroneous values to this table, e.g. through human errors. Will be done.
10. An application should be deployed to safely handle the “maskable” and “disable” flags in the Master Table. Will be done.
Procedures
42. It is recommended to take benefit of this and start as soon as possible full scale test including the full BLM read-out chain. 43. It is encouraged to expand those tests as soon as possible including the BIS (e.g. in point 6 or 8). Large scale test with all monitors in database and all monitors logged is starting now.
LTC, B.Dehning

9. LSA Data Base Structure
Two layers
entry layer (stage tables)
validated layer (final tables)
Concept of Master and Applied table – Comparison of Threshold values (Applied < Master)
Master: less frequent changes
Applied: change of thresholds possible with user interface
300 families
4000 channels
LTC, B.Dehning

10. Consequences of review treated before start-up
Simulation
1. It has not been clear to which extend all possible beam loss scenarios were fully computed. The auditors would like to encourage the BLM team to summarize the results of the simulation studies and measurements done so far. The simulated beam loss scenarios need to be compared with the observed loss locations (fellow will be request). Summarizing simulation is Ongoing (implementation of thresholds started).
Management & Storage of Thresholds
7. Documentation must be produced for the procedures on how the initial values of the Master Table are defined, how the values can be altered, and how these changes are propagated. Ongoing.
8. An application should be deployed that provides means to minimize the introduction erroneous values to this table, e.g. through human errors. Will be done.
10. An application should be deployed to safely handle the “maskable” and “disable” flags in the Master Table. Will be done.
Procedures
42. It is recommended to take benefit of this and start as soon as possible full scale test including the full BLM read-out chain. 43. It is encouraged to expand those tests as soon as possible including the BIS (e.g. in point 6 or 8). Large scale test with all monitors in database and all monitors logged is starting now.
LTC, B.Dehning

11. Consequences of review treated during first time with beam
Simulation
5. Errors between simulation and measurements of up to 50 % are observed, therefore, and due to the aforementioned points, the auditors share doubts that the monitors would guarantee a safe and efficient operation of the LHC without a re-adjustment of the thresholds. Will be possible if needed.
6. The initial threshold settings have to be sufficiently conservative in order not to damage the LHC magnets. During the initial runs of the LHC, they must then be iteratively adjusted. Dedicated (threshold) test procedure should be proposed by the BLM team. Sufficient time should be assigned to make those tests. For example, tests with provoked beam losses should be conducted in order to verify the proper detection of those beam losses. To be decided.
LTC, B.Dehning

12. Consequences of review treated during next shutdown (I)
Simulation
2. Saturation effects should be studied in more detail, and the limits of the current monitor design should be summarized. Saturation studies: measurements ongoing (CNGS, last week SPS MD), simulation studies to be decided.
Management & Storage of Thresholds
15. Means should be investigated for merging and combining the “MTF” and “Layout” databases. Final analysis to be started (criteria: protection, possible GUIs, update speed, availability).
16. An SLA or MoU (service level agreement / memorandum of understanding) stating the responsibilities of the IT department in case of database failure is recommended. 17. A similar SLA/MoU should be set-up with the AB/CO/DM section. To be decided.
PCP & Components
20. Currently, the BLM holds about 5% of spares for the major PCBs. It is suggested to increase this stock to at least 10% including spares Initiated.
LTC, B.Dehning

13. Consequences of review treated during next shutdown (II)
FPGA Code
31. In order to ensure complete testing of future changes in FPGA designs, a PASS/FAIL set of regression tests should be designed. Started, will need significant resources of design engineers during the next year.
32. In addition, a more complete review of the FPGA designs should be conducted once these have been finalized. follow-up of review planned beginning of next year.
Procedure
For bug-tracking, further development, and future upgrades, the auditors recommend setting up a “vertical slice”-test bench, which covers both types of monitors (ionization chambers and SEMs), the full chain of read-out electronics as well as a test database. LHC SPS system is existing, but differences to the LHC system will remain, therefore a LHC test system is needed.
LTC, B.Dehning

14. Consequences of review treated after next shutdown
EMC & Radiation
37. Independent tests should be conducted to verify the consequences of a total ionizing dose (TID), non ionizing energy loss (NIEL) and flux of particles able to produce single event effects (SEE) It is recommend performing a more in depth analysis of the effects of potential SEUs on the behavior of the CFC and it’s FPGA SEEs in the power supplies of the arc and in the straight sections can lead to their complete failure. The failure rate should be determined and the power supplies should be verified to sustain the radiation. All to radiation exposed components have been tested in proton beams, some extensively some less. A more detailed understanding and the long term failure rate needs a better determination.
LTC, B.Dehning

15. Summary
Design and implementation of the BLM is sound, complete, straight-forward, and, conform to requirement on high inherent level of safety, reliability and availability (SIL3) .
The auditors are convinced that the absolute precision will meet the BLM requirements being within a factor 5 (initially) and 2 (later).
The data-driven approach is a good implementation choice.
However:
Iterative tests using real beams needed to finalize threshold values.
Threshold management in DBs require better documented procedures.
Additional tools needed for protecting Master table (e.g. access control, avoiding erroneous values, setting “maskable” and “disable” flags).
The spare part contingency of 5% is too low.
Set-up of a “vertical slice”-test bench is recommended.
2008/7/1
2008/7/1
BLM Audit Report
BLM Audit Report 15

16. Remarks Audit documentation can be found at: http://cern.ch/blm
Several recommendations require a higher level of maintenance efforts as for non safety system needed
LTC, B.Dehning