1. Design of a dependable Interlock System for linear colliders TE-MPE Technical Meeting 1 Patrice NOUVEL

2. Summary Introduction – Context – Problematic – State of the art Requirements establishment – Operational context – Functional requirements – Performance requirements – Interfaces and constraints Design proposal – Functional analysis – Implementation proposal Design verification – Feasibility study – Hardware demonstrator Conclusion and future works 30/01/2014 2

3. Context - CLIC CLIC (Compact Linear Collider): – 3 TeV Collisions – Two beams acceleration scheme – 2012: Conceptual Design Report (CDR) – Cooperation with ILC (International Linear Collider) Future: – ILC : industrialization – CLIC : continue R&D based on CDR CLIC CDR Vol1 30/01/2014 3

4. Context - CLIC Power and energy: – Beams : Main Beam : 280 GJ, 40 nm 2 (x 10 000 pilot beam) Drive Beam : 1.4 MJ, 1 mm 2 (x 100 pilot beam) – Equipment : 580 MW site Pilot beam (Cu) : Energy deposit < 60 J/g M. Jonker et al. MACHINE PROTECTION ISSUES AND SOLUTIONS FOR LINEAR ACCELERATOR COMPLEXES. LINAC12 Beam operation – 50 Hz (100 Hz) – Charge density ramp Need to protect the machine 30/01/2014 4

5. CLIC and machine protection Machine Protection [1] : – Risk reduction => impact and occurrence of unwanted event – Impact : protect => e.g. collimators – Occurrence : prevent => e.g. interlock systems CLIC failures classifications and strategy : – Fast failures (< 1 µs) : e.g. deflected beam in RF cavity Passive protection – Inter-cycle failures (2 ms – 20 ms) : e.g. power converter Interlock system Safe by design principle – Slow failures (>20 ms) : e.g. beam orbit drift Interlock system 30/01/2014 5 [1] B.Todd et al. Machine protection of the Large Hadron Collider. 6th IET Conf, on System Safety - 2011

6. Interlock system Principle : – Stop the beam operation and/or extract the beam based on the machine state Initial requirements for the CLIC Interlock System: – Beam permit: VETO, PASS (binary information, unique and global) – Beam permit loop implementation – Post-pulse analysis: last pulse stability to estimate the next pulse stability – Hardware demonstrator 30/01/2014 6

7. Thesis problematic Design of a dependable interlock system for linear collider Work Positioning: How to answer the problematic: – Design: concepts -> pre-prototype – Integration dependability – Study post-pulse analysis and linear collider Starting points: – CLIC project – Initial requirements – State of the art on Interlock Systems 30/01/2014 7 System Life cycle - IEEE 1220 B. TODD, PhD thesis 2006. A Beam Interlock System for CERN High Energy Accelerator. P.NOUVEL, PhD thesis 2013 Design of a dependable interlock system for linear collider

8. State of the art Protect the machine: permit Reliability and availability Modular architecture Typical interfaces : – Data acquisition – Actuators – Control system – Timing system – Post mortem Cosylab: machine protection workshop 2012 30/01/2014 8

9. Selected protection systems LHC Interlock system – FPGA – Response time max: 100 µs – SIL 3 (100 y < MTBF < 1000 y) – 17 nodes, 140 interfaces LHC Safe Machine Parameters – Threshold comparison LCLS Interlock system – FPGA, gigabits link – Threshold comparison 30/01/2014 9 [3] S. Norum et al. The machine protection system for the Linac Coherent Ligth Source. PAC. 2009 [2] B.Todd. The Safe Machine Parameter – 2011 [1] R. Schmidt et al. Protection of the CERN Large Hadron Collider – New Journal of Physics. 2006 [1] [2] [3]

10. Methodology choice Needs: – Establish a balanced specifications – Basic, transferable to non-experts – Iteration – Set up the project basis (from specifications to prototype). Deal with project uncertainties – Special focus on the dependability Proposal: – IEEE 1220 : Standard for application and management of the system engineering process – Tailored version of IEC 61508 : Functional Safety of Electrical/Electronic/ Programmable Electronic Safety-related Systems 30/01/2014 10

11. IEEE 1220 Methodology Requirements establishment Design proposal System Engineering Process – Extract from IEEE 1220 30/01/2014 11 Adapted to the problematic

12. Requirements establishment Methodology: – Operational scenarios – System interfaces identification – Functional requirements – Performance requirements – Critical interfaces study Comments: – Only main requirements specified 30/01/2014 12 System Engineering Process – Extract from IEEE 1220

13. Requirements establishment - synthesis Main functional requirements (intent declaration) : – Critical: interlock the machine, post-pulse analysis – Non-critical : control, monitoring, test Main performance requirements: – Response time: 2 ms to interlock the machine, 6 ms to perform the post-pulse analysis – Dependability: Critical interfaces: – Technology, local interfaces, architecture 30/01/2014 13 Requirements for one node regarding the redundancy For more information: -MPE-TM (22.03.2012) -Dependability requirements and Design compliance for Interlock Systems. 2013 SYSTOL conference

14. Design proposal Functional analysis: – System behavior – Functional decomposition – Functional architecture Implementation proposal – Sub-functions – System – Modules 30/01/2014 14 System Engineering Process – Extract from IEEE 1220

15. Functional analysis: decomposition a)Sub-functions definition – Individual data analysis – Global analysis – Beam permit system – Control function b)Operational scenarios c)Time, data and control flow – Requirements assignments d)Failure modes and effects e)Safety and monitoring function – Functional risk reduction 30/01/2014 15

16. Functional analysis: architecture 30/01/2014 16

17. Implementation : sub-functions Beam permit system => Beam permit loop Individual Data Analysis => Threshold comparison Global analysis => Summarizers 30/01/2014 17

18. Implementation: system Implementation : – Beam permit loop for each linac – Front end used as slave node (beam permit loop) – Concentrators modules dedicated to post-pulse analysis – Master module delivering the final beam permit to actuators – 3 types of modules 30/01/2014 18

19. Implementation: modules Common part (control, monitoring, test) 30/01/2014 19

20. Design verification Concepts feasibility study: – Beam permit system, beam permit loop – Post-pulse analysis Hardware demonstrator: – Ability of the design to reach the requirements – Basis for prototype 30/01/2014 20 System Engineering Process – Extract from IEEE 1220

21. Feasibility: context CLIC Test Facility: CTF3 – Feasibility study: Drive Beam generation 2-beams acceleration – Protection system existing: Interlock Valve monitoring (software) Vacuum monitoring (software) Repetitive beam losses in CLEX (software) – Beam mostly harmless (~ 700 J, ~ 1 mm²) 30/01/2014 21 140 m

22. Feasibility: experiment Objectives: – Apply post-pulse analysis – Enhance beam operation Statement: – Recurrent vacuum leak (1.5% unavailability) Hypothesis: – Repetitive beam losses – Automatic beam operation Proposal: – Automatic process to restart the beam with safety considerations 30/01/2014 22

23. Feasibility: JAVA application Technical description: – Machine interlocked – Checking klystrons – Sending probe beams – Post-pulse analysis : BPM, radiation monitors – Based on threshold comparison – Logging: application and post- pulse analysis 30/01/2014 23

24. Feasibility: results and discussion Threshold management: – Initial definition (location, operating condition) – Dynamic (operating condition) Need of machine parameters: – Suggestion: integrate safe machine parameters Post-pulse analysis: – Based on fast equipment (120 s) – Computation (integration, averaging, extremum) 30/01/2014 24

25. Hardware demonstrator Technology choice [1] : VHDL Blocks : – Current ideal implementation: FPGA – VHDL blocks for sub-functions (transferable) – VHDL blocks for test bench (GTP, control, monitoring) Design to reach the requirements : – Response time: minimize the critical path – Dependability: functional specifications, simulation (unit testing, system integration, code coverage), hardware test 30/01/2014 25 SafetyResponse time FPGAHigh~µs PLChighest~ms Microcontrollerlow~ns [1] B. TODD, PhD thesis 2006. A Beam Interlock System for CERN High Energy Accelerator.

26. Demonstrator: modules Layout Blocks VHDL – Master Module 30/01/2014 26

27. Demonstrator: hardware used « SPEC » board: – SFP gigabit connector – Open hardware intiative – PCIe connector – FMC connector – Serial port FPGA : Xilinx Spartan 6 – Gigabits link (IP) – Enough slices available FMC (FPGA Mezzanine Carrier) : – Connectivity (Xilinx) – Debug (Xilinx) Control software: LabVIEW 30/01/2014 27

28. Demonstrator: test bench 30/01/2014 28 Emulating the CLIC acquisition infrastructure CLIC Interlock system pre-prototype

29. Measurement procedure Response time: – Definition of the chain of event (CLIC) – Measures (intern, extern), extrapolations, estimations Dependability : – Accelerated test: demand (acc factor x4000) and temperature (acc factor x8) – Limit : emulation 10 9 h > 3 years 30/01/2014 29

30. Results and discussion Response time – Interlock the machine: – 320 µs vs. 2 ms – 1.58 ms left for the acquisition infrastructure (and transmission) Response time – post-pulse analysis : – 125 µs vs. 6 ms – Left time available for more advanced computation Dependability: 30/01/2014 30 Requirements node Measurement results

31. Verification - Synthesis Suggestions : – Integration Safe Machine Parameters – Implementation of mechanism to manage dynamically thresholds Requirements produced: – Acquisition : 1.58 ms – Advanced computation : requirement at ~5 ms Improvements: – Gigabits link – Dedicated thermic test (board limit) – Radiation (SEU) test to consider Next step: – Prototype in a operational environment 30/01/2014 31

32. General conclusion Design of an Interlock System [1] – Requirements establishment – Design proposal – Design verification Dependability – Requirements definition – Verification Application to linear colliders – Increased knowledge of the post-pulse analysis Deliverables – Design proposal and its implementation – Pre-prototype 30/01/2014 32 [1] P. Nouvel, B. Puccio, H. Tap, M. Jonker. Design process of the interlock system for the Compact Linear Collider. Poster presented at International Particle Accelerator Conference, 2013

33. Future works proposed Short term: – Rigorous specification – JAVA application at CTF3 – Thermic test Long term : – Conception methodology (model simulation, model based design) – Prototype integration : PCIe, remote monitoring/control. – Design translation to other accelerators (ILC, ESS) – capitalization – SMP integration study Complementary research trails: – Definition of stability criteria for the post-pulse analysis – Interaction between the Interlock system and the beam operation sequencer – Extension to CLIC injectors (damping ring) 30/01/2014 33

34. Thanks for your attention Questions ? 30/01/2014 34

35. Slides annexes 18/12/2013 35

36. Annexe - Implémentation FPGA: maitre FPGA : Spartan 6 Horloge : 125 MHz Utilisation : – Registers: 2200 ~ 4% – LUTs: 27 300 ~ 8 % (1% mémoire, 7% logique) – Slices: 942 ~ 13 % – MUXCY (carry path and carry multiplexer): 692 ~ 5% – LUT flip-flop pairs (fully used): 1284 – IOB: 15 ~ 5% – Dual Port RAM 8kB: 1 ~ 1% – Dual Clock buffer: 2 ~ 6% – Global clock buffer: 5 ~ 31 % – DSP slices: 1 ~ 1% – GTP: 2 = 100 % – PLL : 2 = 50 % 18/12/2013 36

37. Annexe - IEEE 1220 SEP 18/12/2013 37

38. Annexe - definition IEEE 1233: – prototype: An experimental model, either functional or nonfunctional, of the system or part of the system. A prototype is used to get feedback from users for improving and specifying a complex human interface, for feasibility studies, or for identifying requirements. 18/12/2013 38

39. Annexe – le cycle en V From « Functional Virtual Prototyping” Design Flow and VHDL-AMS. Y.HERVE, P.DESGREYS 18/12/2013 39

40. Annexe – Model Based Design 1.Identification/modélisation du système 2.Analyse du contrôleur et synthèse 3.Simulation – Software in the loop – Hardware in the loop 4.Déploiement 18/12/2013 40

41. Annexe – Post Mortem data LHC 2011 18/12/2013 41

42. Annexe - complément interfaces critiques 18/12/2013 42

43. Annexe – Analyse post-faisceau CTF3 18/12/2013 43

44. Annexe - Machine protection 18/12/2013 44 [1] B.Todd et All. Machine protection of the Large Hadron collider. 6th IET Conf, on System Safety - 2011

45. Annexe – Faisceaux au CTF3 18/12/2013 45

46. Annexe – application JAVA 18/12/2013 46

47. Interface identification Critical: – Acquisition and control infrastructure – Target systems (actuators) Non-critical: – Technical Network – Human-system interface – Timing system – Data management system (configuration, logging data) 30/01/2014 47

48. Functional requirements RequirementsUse Exemples 30/01/2014 48 - Interlock the machine - Critical equipment failure - Low beam stability - Post-pulse analysis - Next pulse instability - Control function - Ability to trigger manually an interlock - Monitoring function - Knowledge of the component state of the system (maintainability) - Provide evidence of the interlocking signal - Test function - Trigger an interlock on given channel

49. Performance requirements Response times: – Interlock the machine : less than 2 ms (requirements) – Post-pulse analysis : 6 ms 30/01/2014 49

50. Performance requirements Dependability: use of a tailored version of the IEC 61508 M. Kwiatkowski – PhD thesis 2013 : Methods for the Application of Programmable Logic Devices in Electronic Protection Systems for High Energy Particle Accelerators From M. Kwiatkowski – PhD thesis 30/01/2014 50

51. Performance requirements 1)Machine requirements : – Tolerable catastrophic event rate: 1 / 10 000 years – Unavailability allocated to interlock system: [0.1 – 0.3 %] 2) Hazard chain and risk identification: 3)Risk analysis => impact and likelihood – Based on operational statistic of LHC (2011) – Verified by hypothesis on the CLIC beam availability 30/01/2014 51

52. Performance requirements 4 et 5) Risk reduction : through system failure rates – False PASS: machine safety – False VETO: machine availability 6) Determining dependability attribute 30/01/2014 52

53. Measurable requirements Statement: difficult to verify (without simulation) that the design proposal reaches the dependability requirement Proposal: transpose these requirements to a verifiable level (i.e. beam permit loop node) Model: Beam permit loop 30/01/2014 53

54. Measurable requirements Parameters: – Node failure rate – Loop redundancy Objectives of the simulation: Results: Simulation adapted from : S. Wagner et al. ARCHITECTURE FOR INTERLOCK SYSTEMS: RELIABILITY ANALYSIS WITH REGARD TO SAFETY AND AVAILABILITY. ICALEPCS 2011 30/01/2014 54 Requirements for one node regarding the redundancy

55. Critical interfaces study Acquisition infrastructure – 22 000 acquisition modules – Daisy-chain topology (400) – Concentrators in alcoves – Data delivered to dedicated 48 front-end Target systems– actuators – Main Beam: Damping rings kickers – Drive Beam: RF gun 30/01/2014 55

56. Critical interfaces study Machine safety: – Acquisition : data corruption – Actuators : machine interlocking failure Tolerable rate: – 4.6 x 10 -7 / pulse = 8.2 x 10 -2 / h (from dependability study) – Independent from the demand Machine availability: – Critical signals duplication 30/01/2014 56

57. Requirements verification Needs : design proposal verification Possible means: – Software simulation (e.g. JAVA) – Simulation and hybrid development (e.g. VHDL-AMS) – Hardware demonstrator Information: – Acquisition infrastructure: gigabits link (white rabbit) – Defined number of requirements 30/01/2014 57