1. Commissioning the European XFEL cryogenic system and best practice
EPICS Barcelona
7. October 2017
Matthias Clausen
DESY Deutsches Elektronen Synchrotron

2. Commissioning IOCs before the I/O is installed
Everybody knows this situation: As soon as the last (or even the first?) cable is connected you get the question: „When can we switch on“?
A way to ‚getting closer‘ to the turnkey system is to simulate the I/O. In our case we simulate _all_ I/O connected to the Profibus. This makes:
The Profibus driver happy because it will see the complete bus
The IOC happy because all records get initialized properly
The engineer happy because he can troubleshoot his database
In the end you get close to a situation where the IOC is ready as soon as the ‚real‘ fieldbus is connected
name, DESY
title of presentation, 22-May-06

3. Simulating I/O
WinMOD
name, DESY
title of presentation, 22-May-06

4. The simulation tool WinMOD is a commercial tool Runs on a Windows PC
Special Profibus card
Configured by XML file generated by the same CSS-based I/O configurator which is used to configure the Profibus.
name, DESY
title of presentation, 22-May-06

5. Testing I/O before it gets installed
All cryogenic controls racks for the XFEL tunnel were fully equipped with all componets before they got installed in the tunnel
This gave us the chance to test the I/O with the final production EPICS database
Hardware and software errors could be removed before installation
Only a few (per mill) errors were left in the final tests in the XFEL tunnel
name, DESY
title of presentation, 22-May-06

6. Diagnostics, diagnostics, diagnostics
Good diagnostic tools help to get started faster during commissioning.
We installed diagnostic tools on each Profibus line to diagnose the state and status:
On each Profibus line to diagnose protocol/ cable errors
On each Profibus line to collect status information from the installed equipment for condition monitoring
name, DESY
title of presentation, 22-May-06

7. Diagnostic Tools name, DESY title of presentation, 22-May-06
Profibus Diagnostic
Condition Monitoring
name, DESY
title of presentation, 22-May-06

8. Separation of Concern IOC XFEL Tunnel Controls RF Compensation Purpose
Continuous controls
Calculate RF losses
Required availability
24/7 for 6 months
24/7 for one week
Reboot whenever
Cold compressors are down
RF is OFF
Communication
Profibus to all signals
Ethernet: TINE protocol
# of changes/ reboot
Only a few; controls are mature
May be many; compensation needs optimisation
IOC Running on
cPCI CPU/ VxWorks
Virtual machine/ Linux
name, DESY
title of presentation, 22-May-06

9. Record Structure
name, DESY
title of presentation, 22-May-06

10. Trust is good control is better …
Testing PLC programs before they get installed (Specially from 3rd parties)
What can go wrong?
Wrong version (not the new one)
New version based not on current but an older version
Structure of communication protocol changed without notice
New functions are in conflict with EPICS IOC
What to do?
Prepare PLC / IOC test bench with is always available to test new PLC versions – but also changes on the IOC side …
name, DESY
title of presentation, 22-May-06

11. Spare Records Specially when IOCs have to run 24/7 for a long time:
Create spare records in each IOC. In our case:
A few AI ~ 10-20
Some PID ~ 20
Many CALC records ~ 30
New functions/ feature can be implemented ‚on the fly‘ and do not have to wait until the next boot. (May be several months in the future)
name, DESY
title of presentation, 22-May-06

12. IOC Boot
What is the current version running in the IOC? What is the next version? (Available – but not active) What was the previous version? (The way back should always be possible)
name, DESY
title of presentation, 22-May-06

13. Redundancy File Server (NetApp) Network UPS IOC
Fieldbus (ring structure)
name, DESY
title of presentation, 22-May-06

14. Summary
Test as much as possible before it is installed in the field Do not save on diagnostics Create you own standards on EPICS/ IOC versioning (or copy a good one from somebody else) Redundancy wherever you can effort it!
name, DESY
title of presentation, 22-May-06