Commissioning the European XFEL cryogenic system and best practice EPICS Meeting @ Barcelona 7. October 2017 Matthias Clausen DESY Deutsches Elektronen Synchrotron

Commissioning IOCs before the I/O is installed Everybody knows this situation: As soon as the last (or even the first?) cable is connected you get the question: „When can we switch on“? A way to ‚getting closer‘ to the turnkey system is to simulate the I/O. In our case we simulate _all_ I/O connected to the Profibus. This makes: The Profibus driver happy because it will see the complete bus The IOC happy because all records get initialized properly The engineer happy because he can troubleshoot his database In the end you get close to a situation where the IOC is ready as soon as the ‚real‘ fieldbus is connected name, DESY title of presentation, 22-May-06

Simulating I/O WinMOD name, DESY title of presentation, 22-May-06

The simulation tool WinMOD is a commercial tool Runs on a Windows PC Special Profibus card Configured by XML file generated by the same CSS-based I/O configurator which is used to configure the Profibus. name, DESY title of presentation, 22-May-06

Testing I/O before it gets installed All cryogenic controls racks for the XFEL tunnel were fully equipped with all componets before they got installed in the tunnel This gave us the chance to test the I/O with the final production EPICS database Hardware and software errors could be removed before installation Only a few (per mill) errors were left in the final tests in the XFEL tunnel name, DESY title of presentation, 22-May-06

Diagnostics, diagnostics, diagnostics Good diagnostic tools help to get started faster during commissioning. We installed diagnostic tools on each Profibus line to diagnose the state and status: On each Profibus line to diagnose protocol/ cable errors On each Profibus line to collect status information from the installed equipment for condition monitoring name, DESY title of presentation, 22-May-06

Diagnostic Tools name, DESY title of presentation, 22-May-06 Profibus Diagnostic Condition Monitoring name, DESY title of presentation, 22-May-06

Separation of Concern IOC XFEL Tunnel Controls RF Compensation Purpose Continuous controls Calculate RF losses Required availability 24/7 for 6 months 24/7 for one week Reboot whenever Cold compressors are down RF is OFF Communication Profibus to all signals Ethernet: TINE protocol # of changes/ reboot Only a few; controls are mature May be many; compensation needs optimisation IOC Running on cPCI CPU/ VxWorks Virtual machine/ Linux name, DESY title of presentation, 22-May-06

Record Structure name, DESY title of presentation, 22-May-06

Trust is good control is better … Testing PLC programs before they get installed (Specially from 3rd parties) What can go wrong? Wrong version (not the new one) New version based not on current but an older version Structure of communication protocol changed without notice New functions are in conflict with EPICS IOC What to do? Prepare PLC / IOC test bench with is always available to test new PLC versions – but also changes on the IOC side … name, DESY title of presentation, 22-May-06

Spare Records Specially when IOCs have to run 24/7 for a long time: Create spare records in each IOC. In our case: A few AI ~ 10-20 Some PID ~ 20 Many CALC records ~ 30 New functions/ feature can be implemented ‚on the fly‘ and do not have to wait until the next boot. (May be several months in the future) name, DESY title of presentation, 22-May-06

IOC Boot What is the current version running in the IOC? What is the next version? (Available – but not active) What was the previous version? (The way back should always be possible) name, DESY title of presentation, 22-May-06

Redundancy File Server (NetApp) Network UPS IOC Fieldbus (ring structure) name, DESY title of presentation, 22-May-06

Summary Test as much as possible before it is installed in the field Do not save on diagnostics Create you own standards on EPICS/ IOC versioning (or copy a good one from somebody else) Redundancy wherever you can effort it! name, DESY title of presentation, 22-May-06