1. LCLS MPS Upgrade Gasper Jansa, Luciano Piccoli, Jeff Olsen, Garth Brown, Sonya Hoobler, Stephen Norum, Stephanie Allison, Kim Kukhee 19.5.2015 Adding BSA and RR

2. 2 Overview MPS overview BSA overview MPS RR design MPS BSA IOC design

3. 3 MPS requirements Turn off or limit rate of electron beam when faults are detected to prevent damage to sensitive machine components. Undulator permanent magnets are the primary components requiring protection from electron beam. Critical that beam is shut off within one pulse at 120 Hz (i.e. 8.33 ms).

4. 4 MPS Design

5. 5 MPS Link Node

6. 6 MPS Link Processor

7. 7 MPS Inputs Inputs from obstructions - Vacuum valves - Profile monitors screens - Beam stoppers And beam loss monitors - Toroids - Protection Ion Chambers (PICs) - Beam Loss Monitors (BLMs) Currently monitoring ~1500 inputs

8. 8 MPS

9. 9 MPS Mitigation Devices Pockels Cell - Electro-optic fast shutter. - Can limit laser to any rate from 0 to 120 Hz. Mechanical Shutter Laser Heater Shutter - Slower mechanical device - Either blocks all light, or allows light at full rate. BYKIK Kicker Magnet - Continually runs at full beam rate (120 Hz). - Positioned half way down machine. - Can limit beam rate to any rate from 0 to 120 Hz.

10. 10 MPS Communication Link Processor sends Synchronization and permit messages at 360 Hz. Status messages returned from Link Nodes upon receipt of sync messages. Unlatch messages sent to Link Nodes upon receipt of status messages. Permit message sent to Link Nodes

11. 11 BSA - Beam Synchronous Acquisition Acquire all beam-dependent scalars across multiple IOCs on the same pulse over multiple pulses of a certain kind (not just x-pulses-in-a-row) up to 360Hz Acquire up to 2800 values per scalar in one acquisition request. Each value of the 2800 values can be an average of up to 1000 values.

12. 12 BSA – Data Gathering Data gathering part consists of the following actions: - EDEF (acquisition definition) setup and start request done on the EVG IOC. - 360hz checking on the EVG IOC with user notification when finished. - 360hz requests (acquisition control) sent by the EVG IOC to all EVR IOCs via fast fiber optic link. - Data checking, averaging, and array update per scalar record per request on the EVR IOCs. Implementation uses EPICS record processing. EVREVR IOCIOC EVGEVG PNETPNET IOCIOC BPM FEE Triggers Timing Crate BPM Crate Data CA Client EDEF Flags, Pattern, etc EDEF Setup CA Client BSA Data

13. 13 BSA – Data Acquisition Across IOCs Scalar data – X, Y, T, Bunch Length, Beam loss, BLM, PIC (new)

14. 14 BSA – Acquisition control Beam code describes project, 1 = LCLS (0=any beam code – good for testing) Turn “ON” when ready. EVG IOC 360Hz event task will turn “OFF” when finished. Turn back “ON” to flush and restart the acq. Define # in each average, # measurements, severity at or above which data is not included in average. Forever option used by system EDEFs. Set machine conditions – values acquired only on pulses where ALL inclusion conditions are true AND NO exclusion condition is true Push “Release EDEF” to free this EDEF number. Name and user will be blanked out. Push “Reset Data” to blank out all BSA data arrays.

15. 15 BSA – Acquisition sequence Timing Thread Data Source Driver Thread High Priority Callback Priority FLINK Evaluate EDEF scanIoRequest BSA failure due to the overloaded callback Data source PV BSA record FLINK Compress record time Data Receptor

16. 16 Why MPS BSA IOC? Adding beam loss values from BLMs and PICs to BSA to correlate with other beam dependent values BLM and PIC Link Node IOCs are easily overloaded (100% CPU) by bringing up more than one EDM display -> panels goes white

17. 17 MPS Design – adding BSA IOC Timing Data MPS BSA IOC

18. 18 MPS BSA IOC Communication BSA IOC sends ‘enable BSA’ message to each BLM and PIC link node Link Nodes extract the IP of sender from ‘enable BSA’ message and start sending the BSA messages to BSA IOC after status messages to Link Processor

19. 19 MPS BSA IOC Sequence diagram

20. 20 MPS BSA Data 6 BLM Link Nodes - 8 BLM (BSA channels) / Link node - Data @360Hz: Current loss 1s, 1/10s, 1/30s 1/60s integrated losses Baseline and test values 9 PIC Link Nodes - 8 – 24 PICs (BSA channels) / Link node - Data @360 Hz: Current loss 1s, 1/10s, 1/30s 1/60s integrated losses Sum: 160 BSA channels @ 360 Hz

21. 21 MPS BSA Data

22. 22 MPS BSA IOC Hardware Initially Motorola MVME 6100 - BSA very CPU intensive : could only handle ~24 BSA channels @ 360Hz - ~3,4M records processing / seconds for 160 BSA channels (160*20*360*2 + 160*20*120*3) Industrial PC - Intel Xeon, SMP, @1,9GHz - LinuxRT - 2 PC 1 PC rolled out to production to receive data from 6 BLM and 1PIC Link Node. The rest of the PIC link node will be updated in few weeks

23. 23 MPS – limitations and problems BSA implemented as record processing using high priority callback -> single threaded - project in place to rewrite the BSA to be multithreaded Can not run multiple IOC on PC due to not being able to share EVR

24. 24 MPS – limitations and problems EPICS 3.14.12.4 has problems running on SMP when slow periodic records added use CPU affinity to run on single core

25. 25 MPS BSA Link Node Simulator Simulating all 15 Link Nodes (6 BLM, 9 PIC) Developed as EPICS IOC (MVME 6100) with EVR In dev env. connected to the MPS private network Used to perform stress tests Used as reference while debugging real link nodes

26. 26 MPS Rate recovery - requirements Critical that beam is shut off within one pulse at 120 Hz (i.e. 8.33 ms) – i.e. in single step If a fault is configured for automatic recovery, the MPS shall recover beam rate step by step, with each step going one rate higher. After each step, the MPS shall wait for 1 second before considering a further increase The MPS shall ensure that when recovering beam rate, going one step higher will not produce losses that would limit the rate back to original rate (i.e. it will prevent the oscillation between two rates).

27. 27 MPS Rate recovery – implementation proposal Possible rates: 0Hz, 1Hz, 10Hz, 30Hz, 60Hz and 120Hz Link nodes integrate beam loss over 5 time intervals: 1s, 1/10s, 1/30s, 1/60s, 1/360s Link nodes have to be aware of the current beam rate For each time interval up and down threshold is applied 5 bits (safe/not safe) sent to the Link Processor Link Processor evaluates and set the rate to the most restrictive rate Link Processor increases rate only no “no safe” bits are present for 1 second