Hardware Platforms for ESS ICS Timo Korhonen ICS Nov.. 5, 2014

Overview 2 Scope of the hardware platform decision(s) Goals Development strategies Platforms High-speed digital platform Middle range I/O Industrial I/O Conclusions

Scope of the decision(s) 3 Control systems have a lot of interfaces Need to cater for a spectrum of I/O requirements Fast, real-time signal processing State-of-the-art technology, evolving fast FPGA-based processing MHz to GHz range of signal acquisition Middle-range I/O requires synchronization, kHz range I/O Non-real time industrial I/O Typically PLC-based Off-the-shelf devices Serial, Ethernet or other fieldbus devices No single platform can cover this cost-efficiently!

Goals 4 Sufficient performance for the task at hand Data processing & transport capabilities Time synchronization Cost-efficiency Looking at the total cost, not only component cost Software (and firmware) dominates cost Roadmap for evolution Beware of obsoleteness traps Project timespan – prepare for change Efficiency of development ICS provides a platform for system development Division of work – development and integration

Development strategies 5 Use standardized equipment whenever possible Select platforms to support and promote them (via a harmonization committee: E2H2C) Benefit from experience in the community Prefer standards that are used in the community Benefit from collaborations Open sharing of work – direct peer-to-peer Use proven solutions whenever possible Sometimes new solutions are required, though Use modular systems Parts can be replaced without redoing the whole system Interfaces need to be clearly defined

Platforms 6 It is not reasonable to do everything with a single platform In the past, VME and CAMAC have been such ones There will be many industrial-type systems We need to be flexible (one size does not fit all) High-end, digital processing platform For beam instrumentation and LLRF applications Acquisition in MHz and above range Response times in microseconds Middle range I/O Often forgotten but important Not highest speed but real-time capable and flexible Industrial I/O PLC is the standard way in industry

High-speed Digital Platform 7 Performance requirements (very shortly) Support 14 Hz operation Good integration with timing Acquisition (ADC sampling) at 88 MSPS or faster 2.86 ms long pulses: a lot of data to be handled Local pre-processing of data Interfaces to machine protection/interlock Post-mortem abilities In fact, any (packaging/bus) platform could provide this! The real questions are elsewhere: Maintainability Development and long-term sustainability Collaborations and support for IKC partners

High-speed Digital Platform – the box 8 Today, there is no clear single choice cPCI, VME, custom (aka “pizza”) box, MTCA.4,...? Serial links have overtaken parallel buses Intelligence is in the interfaces and protocols The “box” supplies infrastructure (power, cooling, monitoring, connectivity, some convenience features) Take a proven solution or go for something new? Several labs are looking at MTCA.4 DESY (main promoter), SLAC, many others Packaging standard supports a modular approach Different components can be mixed (interoperability!) Serviceability, component upgrades, etc. Potentially a long lifecycle (if enough deployment)

High-speed Digital Platform 9 What to take, then? Decide on one (until now, cPCI and MTCA) (quite) some work has been done on MTCA Using a single card (SIS8300) + RTMs MTCA.4 native timing receiver in preparation Seems prudent to continue on the MTCA path Decided on a platform – fine. What then? Platform alone does not provide any functionality Systems and processes have to be built around Here lies the major investment! We have some components developed but Deployment will bring up many issues Systems need to be maintained for a long time

High-speed Digital Platform – and beyond 10 Lifetime issues ESS project timespan > 10 years Obsolecense issues will arise during the project Need to be ready for change! Biggest cost factor is in integration Software, firmware, maintenance processes How could that be modularized? Development of a full lifecycle system is costly Has somebody addressed the deployment issues? How to do gradual upgrades? How to secure maintainability? Deciding the packaging standard is not enough A more comprehensive framework is needed

High-speed Digital Platform – blocks 11 Processing in FPGA is the standard way today Monolithic designs have been common Reuse of components is hard Needs special skills FMC (FPGA Mezzanine Card), VITA-57 standard addresses these issues Separate analog (application-specific) and digital parts with a defined interface One digital board can be used for several purposes Reduce hardware stock complexity Re-use of firmware libraries, software, integration Modularization at the level of the major investment Firmware, software, operational procedures Separation of work (defined interfaces)

High-speed Digital Platform - framework 12 Set up a collaboration with PSI (ESS partner lab) In the last 3 years, PSI invested a lot into development of a framework that addresses the issues of: Performance and flexibility Maintainability and integration Lifecycle management and modularity Done with industry (small company) collaboration We could benefit from that Even if the form factor is different – majority of effort is in the infrastructure for applications Over 10 person-years worth of work Used in production (PSI LLRF, diagnostics, etc.) Interest also from other labs (Daresbury, DESY..)

Framework proposal - development 13 Port the existing framework to MTCA.4 hardware Hardware implementation by the company IOxOS SA Reuse existing designs, with upgrades Rapid hardware development Soft- and firmware can be retained close to 100% Open collaboration with PSI and IOxOS Full documentation and source code sharing Most tricky problems have already been solved Interrupts, DMA handling, etc Collaboration in the first line, can develop into IKC In the mid-term, port previous work to this platform All previous work can be used in the short term Minimize distruptive sudden changes

Middle-range I/O 14 High-end platform is expensive and centralized Use only where needed Some applications still need time synchronization Especially in a pulsed machine! PLCs are not ideal for this (asynchronous cycle times) EtherCAT is a good solution for this range Uses Ethernet cabling, special protocol Can run on a PC or a “hard” IOC Loop times of several kHz possible Distributed I/O (reduce cabling) Cost effective Many I/O modules, several manufacturers Gaining popularity in the EPICS community

Industrial I/O 15 PLC is the standard choice for most industrial-type I/O High reliability Widely known and deployed in partner labs and industries A huge selection of I/O modules available Ideally, select one manufacturer Reduce development and maintenance costs One skillset, reduced stock of spares Select a manufacturer with domain expertise Local support for IKC partners Procure via an open call for tender Not quite straightforward, though – but in progress

“Other” I/O 16 There is still a large number of devices not covered yet Serial devices Use MOXA – seems to be almost de-facto standard Ethernet-based devices Ethernet as a fieldbus, use StreamDevice & ASYN Cameras GigE, 10GBE, CameraLink(HS) Very much provider-dependent – take what the company gives EPICS AreaDetector supports a large device base Motion control Working together with our NSS colleagues

Summary 17 A lot of discussion revolves around the high-end platform But that is only a part of the story work-intensive and critical part, but still Other areas tend to get too little attention try to set up a comprehensive set of solutions We have selected MTCA.4 for the high-end Knowing that there will be issues to solve But they can be solved We try to establish and leverage collaborations Mutual help between institutes Other areas are equally, if not even more urgent Need to keep going there, too Start prototyping (in our new lab)

18 Questions?

19 Reserve slides

System interfaces (PSI board for IS&LEBT) 20 FPGA FMC Event Receiver PCIe switch PCIe switch CPU (signal interfaces) Timing distribution Ethernet (EPICS) PCI express for internal (and external) communication Timing (Event Receiver) on- board via PCI EPICS running locally Ethernet to the outside world

PSI board: a picture tells more than 1kWords 21 CPU PowerPC P GHz dual core runs real-time Linux / EPICS Boots over LAN, uSD card, or on-board FLASH FPGA Virtex-6 LX130T TOSCA-II PCI-express Network on-chip IP connects: 512 MB shared memory (DDR3) connetcs: VME, user logic, FMC sites, VME_P2 User FPGA code PCI-express GEN2 switch central interconnect between CPU / FPGA / XMC / VME_P0 contains Non-Transparent (NT) function