1. Acquisition system for the CLIC Module. Sébastien Vilalte

2. LCWS11 27-09-2011 Sébastien VILALTE A large number of modules have to be acquired between the rare accesses from the surface (~900m). With a large number of channels, the CLIC module needs a local acquisition the closest as possible to the module. The module is a construction “brick” and its number of signals can fit a crate acquisition. The acquisition crate should be a part of the module, one crate for each module. Because of specific signals processing, power consumption and costs, the acquisition electronics should be « custom ». In an other hand, It must support several sub-systems acquisitions developed by several teams and which can evolve: standardized connectivity & protocol. General specifications of a local acquisition system 2

3. Since 2008 we work on the CLIC module acquisition in collaboration with CERN (Lars Soby). → about 100 signals to be acquired every 2m for the 7 sub-systems. Most versatile solution: industrial crate with 1 service board: autonomous 12VDC power supplies performed from 230VAC line, network access and distribution on back-plane. Several standard instrumentation boards: simple architecture based on FPGA and mezzanines: different mezzanines developed for the different subsystems with a standard interface (FPGA high speed connectors). General specifications of a local acquisition system LCWS11 27-09-2011 Sébastien VILALTE 3

4. Network: has to guarantee the distribution of the timings; machine clock, synchronous in some cases; data transmission on long distances, an high speed data rate; radiation hardness… Every crate linked point-to-point to the surface: → limits hardware hard to access, no intermediate stage, isolates the crate if problem, better reliability. Link: Long distances, data rates, EMC and radiation environment: best choice is an optical link between crate and surface. Data collection: A the surface, a PCIe board should collect data. → possible high data rates. → compatible with off-the-shell devices connected to the Control network. Architecture: network LCWS11 27-09-2011 Sébastien VILALTE 4

5. Network: synchronous fiber optic network with a controlled frequency: GBT → high speed network developed by CERN microelectronics department for data acquisition, timing, trigger and experiment control. (cf. https://espace.cern.ch/GBT-Project/default.aspx) Features: → 4.8Gb/s optical link. → up to 40 local chip-to-chip links, ser-des (data concentration). → Radhard design (~100Mrad, 1.E15 n/cm²). → multiple synchronous clocks management. → final version will include SCA slow control features (ADCs, DACs, JTAG, I2C, SPI, alarm monitoring…)very useful for the crate monitoring. → HDL code developed for FPGA interfacing in the GBT collaboration. → future LHC front-end standard network. Each part of the GBT already tested with success (transceiver, SERDES, laser driver…). Architecture: network LCWS11 27-09-2011 Sébastien VILALTE 5

6. - Front part of GBT distribution (e-link) in backplane. - Two mezzanines / motherboard: best compromise between standard board sizes and possible front-panel connections. Architecture: local crate LCWS11 27-09-2011 Sébastien VILALTE 6

7. Standard instrumentation motherboard: A single FPGA (Altera ARRIA II) performs: → interface with the GBT e-link via the backplane. → interface with mezzanines dedicated to the different applications. → possible implementation of a dedicated code for sub-systems applications: processing, feedback controls (attenuators, calibrations…). Use of very high speed Samtec HSMC connectors for mezzanines-motherboards links: → 8 transceivers links up to 8,5Gbps. → 17 LVDS Tx, 17 LVDS Rx 1,2GBps. → 3 reserved links for clocks. A motherboard with high speed mezzanines links will keep high performance in the future: → will allow upgrades for sub-systems mezzanines or future local applications developments with a standard connectivity. i.e. future high speed ADCs with serial high speed links (transceivers)… Architecture: local crate LCWS11 27-09-2011 Sébastien VILALTE 7

8. Main issue for local acquisition: Radiations. T otal dose 15kGy=1,5Mrad? for 15 years (simulations) + neutron fluence(?). Radiation on FPGA: Specific FPGA Rad-hard technologies impossible to get (military) or Rad-tol industrial not adapted for this application (number of cells, Rad-tol, speed…). Technology and several techniques allow to limit radiation effects on industrial FPGAs: → small technologies are more resistant to TID and leakage currents anneal total dose effects. TID should be no more a problem for techno<90nm. SEL are limited by the size of the pnpn thyristor structure. Currently 40nm, next 28nm. → triple modular redundancy code techniques fix the problems of non-destructive SEU. → a final hardcopy version could improve the problems due to RAM susceptibility to neutron fluence: possible if no reconfigurable processing is performed in the FPGA. → shielding to be studied. → a lot of information with the working group in CERN (Jorgen Christiansen fpga-radtol@cern.ch). Architecture: local crate LCWS11 27-09-2011 Sébastien VILALTE 8

9. Main issue for local acquisition : Radiations. Radiation on analog parts and ADCs: → amplifiers, ADCs: not qualified but selected by experience and known to be much more resistant. → power supplies: CERN microelectronics group develops Rad-hard DC-DC converters. ~300MRad, 5.E15n/cm². All parts should be qualified. Architecture: local crate LCWS11 27-09-2011 Sébastien VILALTE 9

10. Second issue: power consumption. Low available power consumption for the whole crate: 30W(?). → power supplies must be switched off between two acquisition. Not obvious for digital. Also to be defined in the future with the collaboration: Mechanical crate standard: industrial, custom… and back-plane. And with CERN Control: Network protocol. Final collection of a section: server, stand-alone system, etc… Architecture: local crate LCWS11 27-09-2011 Sébastien VILALTE 10

11. Before crate prototyping, the development of an evaluation board to test architectures, GBT, mezzanine boards: Emulation of a crate with two instrumentation boards / emulation of four crates. Future platform for developments and tests of mezzanines for applications. A first step: Evaluation Board LCWS11 27-09-2011 Sébastien VILALTE 11

12. A first step: Evaluation Board LCWS11 27-09-2011 Sébastien VILALTE Results: HSMC connector tested with success beyond the specs (up to 10Gbps). All links between FPGAs OK. Clock management and synchronization OK on board. Clock management and synchronization OK between board and PCI via optical link. GBT not yet tested. Produced beginning 2011 240x240mm Hitachi FX-II 60µm class 18 layers 2000 components 900 differential pairs 1,2Gb/s 56 transceiver pairs 8,5Gbps 4 SFP 8,5Gb/s 3 FPGAs 1700 pins 4 USB links 4 mezzanines 12

13. In parallel of evaluation board: development of a mezzanine for DB stripline BPM in collaboration with CERN (Lars Soby, Alfonso Benot Morell). Development of a generic acquisition mezzanine board: Allows to test amplifiers and ADC chosen for the stripline BPM acquisition. Can be useful for all around acquisitions. Can be used to test first shaping circuits for BPMs or other sub-systems. BPMs need 4 channels: quad ADCs are welcome. → limit the dispersion (CMRR, synchronization…), even if we calibrate. → reduce numbers of signals: clocks, controls, power supplies… → reduce power consumption. Mezzanine developments LCWS11 27-09-2011 Sébastien VILALTE 13

14. Drive Beam Stripline BPM mezzanine: Specifications and study by Steve Smith (SLAC) and CERN. resolution 2µm, accuracy 20µm, time resolution<10ns. Acquisition in baseband: 4-35MHz. → shaping with input filters. → needs high dynamic range acquisition: 75dB SNR, at least 100Msps. → needs multi-gain/attenuation to reach resolution both in single pulse and train configurations. → needs accurate calibration. Currently several designs studied by CERN (Alfonso) and LAPP. Prototypes will be produced, tested and compared soon (end 2011). Beam tests foreseen next year in CTF3 DB & MB. Signal processing to develop to reconstruct the beam shape. Mezzanine developments LCWS11 27-09-2011 Sébastien VILALTE 14

15. Generic acquisition board: Produced last spring. → 2 synchronous quad ADCs (Linear tech LTC2174); 11,8 enob; up to 125Msps; serial outputs. → 8 channels. → possibility to implement gain (18dB) and switchable attenuator (-26dB). → jitter cleaner 300fs. → BW=50MHz. Mezzanine developments LCWS11 27-09-2011 Sébastien VILALTE 110x80mm 120 µm class 6 layers 15

16. Mezzanine developments Results: ADC: SNR=72dB, SINAD=71,2 so 11,5 enob @100Msps. Amplifier OK. HSMC connector OK. Jitter cleaning and channels synchronization OK. LCWS11 27-09-2011 Sébastien VILALTE 16

17. Surface data collection: X8 PCIe board. PCIe board already designed to recover the evaluation board data (produced spring 2011): → X8 Gen2 for high speed standard interface. → 4 SFP+ optical links so recovers 4 crates. → 2 MCX inputs for external trigger and clock: if external synchronization needed. → 2 HSMC connectors allowing local mezzanine (i.e. generic acquisition…). → 1 USB link for tests and stand alone use. X8 Not necessary in the future but allows to test the system at its maximum. Number of PCIe lines will be determined by the final needed rate; depends on subsystems. Surface data recovering. LCWS11 27-09-2011 Sébastien VILALTE 17

18. Results: All links tested up to 8,5Gbps. ADC synchronization tested with success. Works well currently with USB, PCIe IP to be implemented (end 2011). Surface data recovering. LCWS11 27-09-2011 Sébastien VILALTE 18

19. Possible global architecture LCWS11 27-09-2011 Sébastien VILALTE 19 Crate 1Crate 3Crate 2Crate 4 Crate n PCIe 1PCIe n GBT links Surface Device 1 PCIe 2 Device n 230VAC line NETWORK Tunnel Possible timings

20. Evaluation board: good results. Next step: implementation of GBT protocol in FPGAs, tests of back-plane GBT e-links. Will allow to test mezzanines with BPMs or others sub-systems developments. Sub-systems developers are welcome to discuss! Crate & motherboards: choice of the crate to be done, development of the service board. First prototypes 2012. Mezzanines for BPM: production and tests next months, tests in CTF3. Documentation to be done for developers, especially mezzanine connection features. In the future, ADCs sampling rates and dynamics will increase, architecture with mezzanines will allow to upgrade systems. LAPP funding: manpower 2,5FTE, ~15k€/year. Conclusion, milestones LCWS11 27-09-2011 Sébastien VILALTE 20