1. ICS platforms for beam instrumentation
Farina Simone Integrated Control System Division
Embedded Systems Engineer

2. Outline System implementation strategy by performance requirements
Standardization procedure
Basic components and system setup strategy
Overview of Timing system features
MTCA AMC digital platforms
AMC platforms FGPA framework
During this discussion we will cover the implementation strategy based on system performance requirements, this is done to optimize the cost/efficiency. We will then see the standardization procedure for the 3 categories, the relevant documentation and the revision process set in place to update the components list will also be described. Next point will be an overview of the most significant components in a MTCA based system deployment. We will then shift the focus on describing the main platforms for data acquisition, both the versatile FMC carrier that can be customized to fit different applications and the DAQ cards supporting signal conditioning chains on the RTM side will be described. The discussion will then be concluded by an overview of the FPGA frameworks for these cards and a side by side comparison will be provided.

3. Three layer strategy - control systems at ESS
ICS has adopted a three layer strategy for implementing the control system based on signal speed
A custom made platform based on microTCA for applications with data acquisition exceeding few kHz
The CPU will run EPICS and the FPGA will have a complete application development environment
For slower signals, EtherCAT will be used as a real-time fieldbus with good price/performance ratio
Synchronization and event information are key for applications where a full custom platform solution would be too costly
Low speed signals are handled with commercially available PLC systems
This is a cost-effective solution that addresses ESS reliability and maintainability requirements
The PLCs will be connected to EPICS for further integration into the control system
A custom made microTCA board with a powerful FPGA. A FPGA framework constitutes the development environment. The CPU board runs EPICS
Signal speed
10 MHz
1 MHz
100 kHz
Digital controls platform
10 kHz
1 kHz
EtherCAT I/O modules that are connected to commercial or open-source EtherCAT master controllers which are in turn connected to the EPICS based control system
100 Hz
EtherCAT
10 Hz
The three main categories of components that will be deployed throughout the facility are PLCs, EtherCAT connected modules and MTCA systems. The PLCs will address the needs of slow speed highly reliable systems, where they provide both a cost-effective and highly reliable solution that has been well proven by many industrial applications. The intermediate category is based on EtherCAT devices controlled by commercially available components and/or open-source drivers. These devices offer a good compromise in term of price vs performance. An example of this category would be the control system for motors like the slow tuning of the RF cavities. For high demanding applications, like fast digitizers electronics, ESS has decided to rely on MTCA equipment and take benefit, where possible, of the portfolio of commercially available components. All these system will of course provide an interface to EPICS to allow integration into the control system.
Slower signals are handled by industrial automation (PLC) for reliability and cost reasons. The standardized platform for ESS applications comes from Siemens
1 Hz
0.1 Hz
Industrial automation (PLC)

4. Standardization Process: PLCs
Comprehensive list of components for:
Conventional systems
Safety systems
Standardization process procedure :
Submission of request for new component
Monthly board decides on:
component introduction in the standard list
updates of old components to the new releases
Introduction of new components in the library
In the strive to achieve a consistent and comprehensive list of components that will cover most of the use cases without compromising ESS ability to have and maintain a local stock of replacement parts readily available on site, a standardization process has been set in place resulting in the creation of the “ESS Standardized PLC equipment” document. This document addresses both the conventional and safety system components. In order to maintain an up to date list and to address the requests for new devices coming from the system designers a procedure has been set in place. The designer can request for a new component to be made available in the library and monthly there will be a working group meeting that will decide on the introduction of these requested items into the aforementioned document eventually proposing other devices suitable for the task. The group will also be responsible of keeping the part numbers up to date when old components gets discontinued and/or replaced by newer models.

5. Standardization Process: MTCA
Comprehensive list of equipment including:
Infrastructure components:
Chassis
Power modules
MicroTCA Carrier Hubs
Chassis replacement parts
Specialized AMC cards:
High performance CPU
Data Acquisition
Timing synchronization
Versatile AMC cards:
FMC carrier
RTM support card
RTM cards
FMC modules
A similar document is also available for MTCA. This document covers the basic MTCA infrastructure that is made up of Chassis, Power modules and Carrier Hub which is the core component taking care of the management and providing the main interconnection switching feature between the AMC cards. A comprehensive list of replacement parts for the pieces that are subject to wear in the chassis, like the cooling units (fans), is also provided.
The next sections of this document lists the AMC cards: high performance CPUs based on Intel i7 architecture, Timing synchronization cards with the compatible mezzanine modules proving front panel interfaces and data acquisition cards with analogue interfaces on the zone 3 connector that can be paired with signal conditioning RTM cards. FMC carrier cards are also covered in this document and so is the set of FMC cards for which firmware and software support is available or foreseen. The list of foreseen rear transition modules is also included.

6. MicroTCA Chassis (1)
Two different chassis types for production deployment:
12 slots 9U for systems requiring more than 6 AMCs and/or more than 4 RTMs or redundant low noise power modules;
6 slots 3U to reduce system space in racks.
Front to rear cooling scheme
Tradeoffs:
As said in the previous slide the standardadized list contains a set of chassis to be used at ESS. For the final installation two options has been discussed and selected in agreement with the relevant stakeholders. Both these options comply to the defined cooling strategy inside the rack islands in the Klystron gallery. The first is a 9U chassis providing support for up to 12 AMC cards and RTMs, it can be populated with up to 4 power modules, that allows the implementation of load sharing as well as redundant architectures, and 2 MCHs. This chassis has 2 cooling units located at the bottom and at the top of the AMC section that push and pull the air to and from the AMCs, MCHs and PMs. The second chassis that has been selected is a 3U variant where the boards are aligned horizontally and that can support up to 6 AMCs, four of which can also be connected to a compatible rear transition module, one MCH and 2 PM depending on their size. On this chassis the cooling units are both located on the back and pull the air out of the system so filling all unused slots with filler plates is mandatory to achieve good cooling performance. The first chassis will address those use cases where the amount of cards installed would not fit on the smaller variant or where space for future upgrades or redundacy is desired.
Supported Components 9U 3U
MCH(s) 2 1
PM(s)
4 (3+1)
2 (single width)
CU(s)
2 (push-pull)
2 (pull)
AMC(s) 12 6
RTM(s) 4

7. MicroTCA Chassis (2) 9U backplane interconnections:
2 GbE links (1 towards each MCH)
2 high speed AMC to AMC links (BP specific)
Redundant PCIe 4x
4x LVDS type AMC to AMC links (BP specific)
This slide depicts the backplane connections available in the 9U chassis. Bla bla bla
8x MLVDS lanes
4x AMC Clock input lines (2 for each MCH)
1x FAT Pipe Reference CLK

8. MicroTCA Chassis (3) 3U backplane interconnections: 2 GbE links
2 high speed AMC to AMC link
PCIe 8x to all AMCs
4x LVDS type AMC to AMC links
8x MLVDS lanes span all AMCs
2x AMC Clock lines
This is the backplane layout for the 3U chassis. Main differences compared to the previous chassis are the routing of both GbE interfaces towards the same MCH, the grouping of the Fat Pipe and Extended Fat Pipe into a single x8 link routed to the only available MCH.
1x FAT Pipe Reference CLK

9. MicroTCA Carrier Hubs MicroTCA Carrier Hub (standard deployment):
PCIe gen.3 48 port switch;
Up to 6 Virtual PCIe clusters;
GbE switch;
Clock input/output on backplane CLK1 and CLK2 plus front panel SMA;
MicroTCA Carrier Hub (specialized deployment):
Regarding the MTCA carrier hubs two variant are foreseen but only one is currently scheduled for deployment. It offers a 48 port PCIe gen 3 switch capable of handling up to 6 virtual cluster, basically dividing the system into up to 6 independent sections. It also provides GbE switching capabilities and the ability of distributing clock over the backplane either generated from an AMC or externally and conveyed on the available SMA front panel connectors. The newer model is included in the list to cope with possible future scenarios where more than one chassis could be interconnected, effectively extending the PCIe tree over more that one crate through the optical uplinks. This MCH is also MTCA4.1 ready but ESS does not foresee to adopt the latest revision of the standard and the defined LLRF backplane.
Extended functionalities over the standard deployment
PCIe gen.3 80 port switch;
PCIe Uplinks (x8 or x16) to extend bus on multiple chassis;
Support to up to 4 eRTMs and 2 rear PM (MTCA.4.1)

10. MCH script setup
Many different configurations need to be performed on a chassis through the MCH:
PCIe tree definition;
Clock distribution;
Ethernet switch setup;
Basic accessibility configurations;
Backplane power configuration;
Etc.
Most of these settings could be done using the web interface, exceptions are clock distribution and power configuration, but it would be cumbersome.

11. MCH script setup
Configuration using script is supported and allows to:
Setup PCIe virtual switch defining upstream and downstream AMCs
# Syntax: pcie_vs_cfg = vs_id, up_amc, nt_up_amc, ds_amc_list pcie_vs_cfg = VS_0, AMC1_4, NONE, AMC3_4, AMC7_4
# creates PCIe virtual switch with AMC1 as RC # and AMC3 and AMC7 as EP
Clock distribution for TCLKA and TCLKB
# Syntax: clk_phys_out = dst, src
clk_phys_out = 2, 21
# AMC TCLKA for slot 2 will originate from TCLKB slot 5
Ethernet switch configurations are also fully available
Script loaded using CLI upload_cfg command. Backplane power configuration for the setup of redundant power supply schemes or load sharing between PM requires backplane FRU EEPROM information update and will be carried out in the MCH console interface.

12. MicroTCA PM & CPU Power Modules:
Wiener (1KW)
N.A.T. (600W)
Both will be used. Choice according to:
Noise requirements;
Redundancy;
Concurrent Technologies AM900:
Intel i7 x86-64 architecture
Up to 16GB DDR3 memory
PCIe upstream

13. MicroTCA AMC Timing, Event Receiver
Timing system EVR:
clocks from/to TCLKA/TCLKB
driving/receiving differential triggers AMC RX/TX ports 17 to 20 (MLVDS)
front panel 4 x TTL outputs, 2 x TTL inputs
front panel 2 Universal I/O modules
Delay compensation and feedback
Timing system UIO mezzanine modules:
2 optical fiber variants
7 coax cable variants (NIM, TTL/LVTTL, LVPECL)

14. MicroTCA AMC Timing, Event Receiver
Provide the base frequency of ~14 Hz (derived from MHz)
Provide timestamping
Provide trigger and clock signals (using ports 17 to 20 and TCLKA/TCLKB)
Reacts on frames sent by Master Event Generator :
contains an 8-bit event code and alternating distributed bus/data byte (8 bits).
Distributed bus (DBus) broadcasts eight binary signals which can be output from each Event Receiver. Clocks, status bits, etc.
Event Generator can broadcast arbitrary data (up to 2kB blocks)
Frame

15. MicroTCA AMC FMC carrier
IFC1410:
in-kind contribution PSI
Processing System:
NXP T2081 PPC running at 1.8GHz
2GB DDR3L (upgrade path to 4GB with ECC)
64 MB SPI Flash memory
512MB NAND Flash
PCIe x4 gen.3 connection to Programmable Logic (no direct connection to backplane)
Programmable Logic:
Xilinx Kintex Ultrascale KU040 (vertical migration KU060)
2x 512MB DDR3L
3 PCIe End Points ( x4 gen.3 to backplane )
2 HPC FMC support
Support for non-standard form factor mezzanine card (ie FBIS connection) on AMC side 1 or 2
Zone3 connector (RTM support) compliant to class D1.4

16. MicroTCA AMC DAQ IFC1420 IFC1411 + DAQ1430: Processing System:
in-kind contribution PSI
Processing System:
Same characteristics as IFC1410
Programmable Logic:
Xilinx Kintex Ultrascale KU040 (vertical migration KU060)
2x 512MB DDR3L
2 PCIe End Points ( x4 gen.3 to backplane )
10 ch ADC 250MSPS 16-bit (DAQ1430)
4 ch DAC 2.5GSPS 16-bit (DAQ1430)
Internal, Front Panel and Backplane Clock Sources
1 HPC FMC support
Support for non-standard form factor mezzanine card (ie FBIS connection) on AMC side 1 or 2
Zone3 connector (RTM support) compliant to class A1.1CO

17. MicroTCA AMC DAQ IFC1420 characterization
Test setup:
ADC clock source provided by TCLKA (backplane) from MCH front panel;
Filtered sinusoidal input source at 21.4 MHz;
SIS8900 RTM signal conditioning board with Single Ended to Differential circuitry;
Average Analog Performance of the 10ch:
ENOB
11.56 Bits
SFDR
84.17 dBFS
SNR
70.33 dBc
Comparison with Analog Performance of ADC3110 ( 8ch 250MSPS 16-bit ADC FMC card ) :
ENOB
11.84 Bits
SFDR
87.54 dBFS
SNR
72.04 dBc

18. MicroTCA AMC DAQ IFC1420 characterization
Channel to channel crosstalk results (dBc):

19. MicroTCA AMC Data AQuisition
SIS8300-KU:
COTS AMC card
Programmable Logic:
Xilinx Kintex Ultrascale KU040
2 GB DDR4
x4 gen.3 PCIe End Point to backplane
10 ch ADC 125MSPS 16-bit
2 ch DAC 16-bit
Internal, Front Panel, RTM and Backplane Clock Sources
Support for up to 2 SFP+ modules for high speed system interconnect
2 front panel RJ45 for external I/Os (ie FBIS connection)
Zone3 connector compliant to class A1.1CO

20. TOSCA Framework NoC Overview
IFC14xx Firmware infrastructure:
Separate Data and Control interconnect infrastructure;
Data:
Full mesh switch;
Every agent SW has two master and slave interfaces for request and response communications;
Communication based on PCIe TLPs
IO Control:
Shared parallel bus;
Master and slave interfaces for each agent switch;
Round Robin arbiter to assign bus mastership

21. TOSCA Framework AgentSwitch details
Independent communication paths for Request and Response
Each master port owns a dedicated path with each slave
Buffers are built with 32 Bytes Header section and up to 512 Bytes Data payload
Interconnects between the Agent_SW Master and the Agent_SW Slave are implemented with
dedicated 64+8-bit path operating at 250 MHz

22. TOSCA Framework Files Relations

23. TOSCA Framework Features
Communication based on proprietary interfaces (some are similar to AXI4-Lite) and TLP like frames on the NoC
Customized IPs for many peripherals, including DDR Controller. Vivado IPs for PCIe, BRAM, FIFOs, etc.
Two DDR memory blocks accessible from NoC (flow control) and/or specialized interfaces from/to XUser AP Specific block
Conventional Shared Bus Architecture on TCSR registers (Tosca Control and Status Registers)
Board configuration using FPGA based implementation of microcode
Network on Chip implementation, every Agent SW has two master and slave interfaces for request and response communications and an independent path towards any other Agent SW. Improves throughput when two masters want to talk to two different slaves at the same time.

24. ESS Framework (BI) Overview

25. ESS Framework (BI) Configuration Controller

26. ESS Framework (BI) Files Relations

27. ESS Framework (BI) Features
Communication based on standard AXI4 interfaces (incl. Lite & Stream variants)
Uses Xilinx Vivado IPs whenever possible (Microblaze, DDR Controller, SPI, I2C, GPIO, etc.) but some proprietary code for DMA and PCIe transfers
Arbitrated access to DDR memory using AXI Interconnect IP (either in Shared Access Mode or in Crossbar Mode)
Conventional Shared Bus Architecture on Register Bank AXI4-Lite bus
Board configuration using Soft Processor (Microblaze) and C code acting on peripheral interfaces IPs

28. Framework Comparison Side to side comparison TOSCA – ESS BI
TOSCA Framework
ESS BI Framework
Custom (Specialized) interfaces and use of TLP-like frames on the NoC
Standard AXI4 interconnections (may excl. Application specific internal busses)
Peripherals use mostly customized IPs , including custom DDR Controller. Internals use Xilinx Vivado IPs
Extensive use of Xilinx Vivado IPs but still some customized code for PCIe and DMA transfers to retain compatibility with proprietary driver
Two DDR memory blocks accessible from NoC (flow control) and/or specialized interfaces from/to XUser AP Specific block
Arbitrated access to DDR memory using AXI Interconnect IP
Conventional Shared Bus Architecture on TCSR registers
Conventional Shared Bus Architecture for Register Bank access
Board configuration using FPGA based implementation of microcode
Board configuration using Soft Processor and C/C++ code
Network on Chip implementation for primary data transfer
Many AXI4 busses connecting the DDR to the other modules

29. Summary Identification of 3 major device types for deployment
List of standard electronic components available in CHESS
Chassis selection for deployment and MCH configuration using script and Command Line Interface
Timing system equipment and frame characteristics
Description of in-kind contributed platforms and COTS alternative for DAQ systems using RTMs
FPGA Framework overview for both platforms

30. Glossary
MicroTCA : Micro Telecommunication Computing Architecture; MCH : MicroTCA Carrier Hub (manages the other boards, provides interconnection switch capabilities, clock distribution, etc.); PM : Power Module; CU : Cooling Unit; AMC : Advanced Mezzanine Card (basic constituent of the MTCA system, can provide support for a RTM); RTM : Rear Transition Module (provides real estate area for additional electronics); MMC : Module Management Controller (monitors the status of the AMC); MCMC : MicroTCA Carrier Management Controller; FRU : Field Replaceble Unit (basically any component of a MTCA system) AXI4 : Advanced eXtensible Interface (fourth generation of AMBA interface); AMBA : Advanced Microcontroller Bus Architecture; TLP : Transaction Layer Packet (PCIe);

31. Glossary
RC : Root Complex EP : End Point CLI : Command Line Interface LVDS : Low Voltage Differential Signaling MLVDS : Multipoint LVDS ENOB : Effective Number Of Bits (ENOB= 𝑆𝐼𝑁𝐴𝐷 − ); dBFS : decibel Full Scale; dBc : decibel relative to Carrier; SFDR : Spurious Free Dynamic Range; SNR : Signal to Noise Ratio; SINAD : SIgnal to Noise And Distortion ratio (𝑆𝐼𝑁𝐴𝐷= Psignal + Pnoise + Pdistortion Pnoise + Pdistortion )

32. Backup slides
Backup Slides

33. MicroTCA Overview What is MicroTCA ? “MicroTCA is a modular standard.
By configuring highly diverse collections of AdvancedMCs in a MicroTCA Shelf, many different application architectures can be easily realized.
ESS has decided to adopt a multi layer strategy for deploying the required electronic equipment and MicroTCA has been selected as standard platform for those systems that require fast computation, response time or whose input signals are above few tens of KHz.
But then <why MicroTCA?>...<MicroTCA is a modular standard. By configuring highly diverse collections of AdvancedMCs in a MicroTCA Shelf, many different applications architectures can be easily realized. The common elements defined by MicroTCA are capable of interconnecting these AdvancedMCs in many interesting ways–powering and managing them, all at high efficiency and low cost.>
The key benefits of this standard are, leaving aside for the moment the interconnections on the backplane used for bulk data transfers:
High availability;
Redundancy;
Manageability;
Serviceability.
The common elements defined by MicroTCA are capable of interconnecting these AdvancedMCs in many interesting ways—powering and managing them, all at high efficiency and low cost.”
Main features:
High availability
Redundancy
Manageability
Serviceability

34. MicroTCA Overview (2) IPMI management on IPMB bus
This is an overview of a MicroTCA system with all the relevant components (missing AMC to AMC connections)
The orange sections, MCMC, EMMC and MMC are those responsible for the management of the system or the individual cards and they are connected using IPMB busses (either IPMB-0 or IPMB-L).
On this I2C like busses the informations about cards status and sensor readings are collected by the MCMC on the MicroTCA Carrier Hub and alarms can be raised if a monitored level is outside the user/vendor defined boundaries. The system is also capable of coping with the alarms according to predefined strategies.
IPMI management on IPMB bus
Board presence detect and enable signals

35. Describe classes for RTM

36. MicroTCA AMC DAQ IFC1420 characterization
Analog Performance of the 10 channels:

37. Software Support for IFC14xx
Low-level driver customized by ICS to optimize performance by use of interruptions to command the DMA transfers;
Userspace library provides a unified C interface to firmware resources;
EPICS integration:
EPICS 7 support TBD;
areaDetector support TBD;
EPICS asyn device support
ifcdaqdrv2
tsclib userspace library
tsc kernel module
PCIexpress
FPGA Firmware

38. Software Support for IFC14xx
ifcdaqdrv2
Firmware application
ADC implementation
EPICS asyn device support
FMC common
ifcdaqdrv_utils
ifcdaqdrv2
ifcdaqdrv2: stand-alone C library
Abstracts away the difference between the FMC cards and, therefore, the different firmware applications;
Generic data acquisition interface;
tsclib userspace library
tsc kernel module
PCIexpress
FPGA Firmware

39. MicroTCA development environment
NFS mount
Development environment
E3 Server
EEE Server
IFC1410:
T208x Freescale PPC architecture
Linux Kernel
Device Tree
Root File System
Kernel Modules
(MRF EVR,IFC1410, EtherCAT)
U-boot scripts
FPGA bitstreams
2GB DDR3 memory
Xilinx Kintex Ultrascale FPGA
NFS mount
2x HPC FMC support
D1.4 RTM support
TFTP
Two options exist to use the development environment
ICS can provide the development environment as a stand-alone system on a MTCA CPU card with local storage
User can set up any regular PC as a boot server for the MTCS system
All software modules listed will be provided by ICS using an on-line BitBucket/GitLab repository
Stakeholder has access to all updates (such as updates of kernel modules or new FPGA firmware versions)

40. System Example interconnections : LLRF
Timing:
Gets synchronization info from external device and provides:
clock to MCH
triggers to all AMCs
machine configuration settings
MCH:
Monitors AMCs status, distributes clocks and provides interconnections
CPU:
This slide covers another test stand currently under development at ESS and LTH, the Lund Technical University. Depicted is the scheme of the LLRF excluding the RF-LPS section.
This system is currently implemented using a COTS Struck SIS8300-L2 digitizer AMC, this board provides 10ch 16-bit analog to digital converter section running at 125MSPS and hosts a Virtex 6 FPGA where the control algorithm is implemented.
RTM carrier board, Local Oscillator RTM and Piezo Driver RTM are currently under development by In-Kind contributors from the Polish Electronic Group.
The remaining components provide similar functionalities to those described in the previous example.
Discussions are ongoing to implement the system using the IFC1420 digitizer card or the new revision of the SIS8300 that offer similar capabilities but include the new Ultrascale FPGAs.
Manages a subset of AMCs:
configures and collects data
runs EPICS IOCs
AMC & RTM:
Application specific. May provide interlocks and information to MPS