, a rad-tol FPGA-based WorldFIP agent E. Gousiou, P.Alvarez, E.van der Bij, G.Penacoba, J. Serrano | CERN TWEPP 2011 nan FIP

Outline o Project Introduction o Functionality & Features o Design Validation o Conclusions

Outline o Project Introduction o Functionality & Features o Design Validation o Conclusions

WorldFIP, microFIP & nanoFIP (I) o WorldFIP is a real-time fieldbus used at CERN's LHC for a variety of control systems: Cryogenics, Power Converters, Quench Protection, Beam Instrumentation, Radiation Monitoring, Survey o More than WorldFIP client nodes (agents) and 200 WorldFIP master nodes installed in the LHC o WorldFIP was selected because of the good performance of its agents under radiation 4 FIELDBUS … WFIP agent WorldFIP Master User logic WorldFIP Architecture sensor WFIP agent User logic actuator WFIP agent User logic sensor

WorldFIP, microFIP & nanoFIP (II) 5 o However, in 2009 Alstom decided to phase out WorldFIP support o Moreover, the latest batches of WorldFIP agents, the microFIP chipsets, were found less radiation tolerant o Finally, it was decided to in-source this technology at CERN o The first phase of in-sourcing concerns the most critical part, the WorldFIP agents is the replacement of microFIP. It implements a subset of microFIP’s functionality. It is a radiation tolerant FPGA-based chip that acts as an agent for the communication over the WorldFIP fieldbus. nan FIP o The development has been divided to work packages that have been distributed among CERN groups and industry o Until recently, Alstom was the main provider of WorldFIP technology

Open Hardware 6 o nanoFIP is part of the Open Hardware Repository o Public specification, design files and production files o Focus on peer reviews o Better hardware o See also “Open Hardware for CERN’s Accelerator Control Systems”, Erik van der Bij

Outline o Project Introduction o Functionality & Features o Design Validation o Conclusions

WorldFIP services : Data consumption & Broadcast data consumption (up to 124 bytes) Data production (up to 124 bytes) Communication in 3 speeds: 2.5 Mbps, 1 Mbps, Kbps Functionality & Features 8 Master User n consumption

Functionality & Features 9 WorldFIP services : Data consumption & Broadcast data consumption (up to 124 bytes) Data production (up to 124 bytes) Communication in 3 speeds: 2.5 Mbps, 1 Mbps, Kbps microFIP: 120 bytes for shared produced and consumed data Master User n production

Functionality & Features 10 Simple interface with the user: Data transfer over an integrated memory or User n WISHBONE MEMORY Master User n WorldFIP services : Data consumption & Broadcast data consumption (up to 124 bytes) Data production (up to 124 bytes) Communication in 3 speeds: 2.5 Mbps, 1 Mbps, Kbps microFIP: 120 bytes for shared produced and consumed data

Simple interface with the user: Data transfer over an integrated memory or User WISHBONE MEMORY Data transfer in 16 in, 16 out lines (no need for memory access) Functionality & Features 11 Master User n microFIP: proprietary interface WorldFIP services : Data consumption & Broadcast data consumption (up to 124 bytes) Data production (up to 124 bytes) Communication in 3 speeds: 2.5 Mbps, 1 Mbps, Kbps microFIP: 120 bytes for shared produced and consumed data n 16 bit DATA BUS

JTAG feature Efficient way to remotely reprogram the user FPGA 3’ for Actel A3P400; 1.5’ for Xilinx XC5VFX70T Functionality & Features 12 User TAPTAP Master User n Simple interface with the user: Data transfer over an integrated memory or Data transfer in 16 in, 16 out lines (no need for memory access) microFIP: proprietary interface TAPTAP JTAG microFIP: no reprogramming feature WorldFIP services : Data consumption & Broadcast data consumption (up to 124 bytes) Data production (up to 124 bytes) Communication in 3 speeds: 2.5 Mbps, 1 Mbps, Kbps microFIP: 120 bytes for shared produced and consumed data n WISHBONE MEMORY 16 bit DATA BUS n

Radiation tolerant design 13 Component Selection o Actel ProASIC3 family o Flash-based & reconfigurable o Configuration cells do not exhibit SEUs o Immune to SELs for the LHC o Measure TID > 300 Gy > 10 LHC years o Proven performance in radiation environments (ALICE, nQPS, NASA). Mitigation Techniques o Simplification of specifications

Radiation tolerant design 14 Component Selection o Actel ProASIC3 family o Flash-based & reconfigurable o Configuration cells do not exhibit SEUs o Immune to SELs for the LHC o Measure TID > 300 Gy > 10 LHC years o Proven performance in radiation environments (ALICE, nQPS, NASA). Mitigation Techniques o Simplification of specifications o TMR of the flip-flops & memories o Fail-safe state machines FPGA area usage

Radiation tolerant design 15 Component Selection o Actel ProASIC3 family o Flash-based & reconfigurable o Configuration cells do not exhibit SEUs o Immune to SELs for the LHC o Measure TID > 300 Gy > 10 LHC years o Proven performance in radiation environments (ALICE, nQPS, NASA). Mitigation Techniques o Simplification of specifications o TMR of the flip-flops & memories o Fail-safe state machines FPGA area usage

Radiation tolerant design 16 Component Selection o Actel ProASIC3 family o Flash-based & reconfigurable o Configuration cells do not exhibit SEUs o Immune to SELs for the LHC o Measure TID > 300 Gy > 10 LHC years o Proven performance in radiation environments (ALICE, nQPS, NASA). Mitigation Techniques o Simplification of specifications o TMR of the flip-flops & memories o Fail-safe state machines FPGA timing

Radiation tolerant design 17 Component Selection o Actel ProASIC3 family o Flash-based & reconfigurable o Configuration cells do not exhibit SEUs o Immune to SELs for the LHC o Measure TID > 300 Gy > 10 LHC years o Proven performance in radiation environments (ALICE, nQPS, NASA). Mitigation Techniques o Simplification of specifications o TMR of the flip-flops & memories o Fail-safe state machines FPGA timing

Radiation tolerant design 18 Component Selection o Actel ProASIC3 family o Flash-based & reconfigurable o Configuration cells do not exhibit SEUs o Immune to SELs for the LHC o Measure TID > 300 Gy > 10 LHC years o Proven performance in radiation environments (ALICE, nQPS, NASA). Mitigation Techniques o Simplification of specifications o TMR of the flip-flops & memories o Fail-safe state machines o Various reset possibilities User n PoR Rst In Rst Out Reset Frame

Outline o Project Introduction o Functionality & Features o Design Validation o Conclusions

Design Validation 20 Design validation with independent simulation test bench Test board for functionality & radiation testing developed by external company Extensive 5-day VHDL code review by 5 design experts from 2 different CERN groups 19 boards running continuously since May 2011

Design Validation – Test Board 21 User logic Field TR Fiel drive Master FIELDBUS nan FIP

Master nan FIP Design Validation – Test Board 22 User logic Field TR Fiel drive FIELDBUS Cons

Master Design Validation – Test Board 23 User logic Fiel drive FIELDBUS nan FIP Field TR Cons Prod

Control Room Design Validation – Test Board 24 User logic Field TR Fiel drive Master FIELDBUS nan FIP

Control Room Design Validation – Test Board 25 User logic Field TR Fiel drive nan FIP Master FIELDBUS

Control Room Design Validation – Test Board 26 User logic Field TR Fiel drive nan FIP Master FIELDBUS

Control Room Master Design Validation – Test Board 27 User logic Field TR Fiel drive 9V 50m FIELDBUS nan FIP

Control Room Master Design Validation – Test Board 28 User logic Field TR Fiel drive RS m 9V 50m FIELDBUS nan FIP

Radiation Testing Campaigns Large scale tests: Cross section estimation 29 Preliminary tests: qualification of test setup and first understanding of possible failures Extra tests: Irradiation of the nanoFIP, FIELDRIVE & the FIELDTR Study of the effects of high temperature while irradiating

30 o 2 nanoFIP chips o Use of the entire Produced and Consumed memory of nanoFIP o 5ms and 500ms macrocycles PSI facility 230 MeV p+ beam p+ beam Preliminary Radiation Tests at PSI April 2011 p+ beam

31 o Correct frames exchange until ~400 Gy; no SEE o At ~400 Gy no frame was being received by the Master; chips not responding o Soft reset/ power cycle not able to recover the functionality o Several hours later without radiation the chips had annealed and were fully functional Preliminary Radiation Tests at PSI – nanoFIP April 2011 “The instabilities were always accompanied by an increase of the current in the FPGA core from 1 to 33 mA.” Radiation-Tolerant ProASIC3 FPGAs Radiation Effects ACTEL report nanoFIP consumption DUT a [mA] nanoFIP consumption DUT b [mA]

FIELDRIVE consumption DUT a [mA] 32 o 1 set of FIELDRIVE/ FIELDTR o 5 ms macrocycle o Testing stopped at 400 Gy due to beam time expiration o No error appeared throughout the testing 10 % current consumption increase Preliminary Radiation Tests at PSI – FIELDRIVE April 2011

Large Scale Radiation Tests -Target Cross Section Scheduled for Nov 2011 PSI facility, p+ 230MeV Irradiation of 10 devices 33 2e9 p+/cm 2 / Gy 400 Gy ProASIC3 lifetime σ nanoFIP ~ 1e-13 cm 2 < 10 SEE / year 2000 nanoFIPs in the LHC LHC σ nanoFIP ~ 1e-13 cm 2

Outline o Project Introduction o Functionality & Features o Design Validation o Conclusions

Conclusions 35 Rigorous design process Exhaustive testing Wide collaboration across the organization and with the industry Expected demand at CERN > 2000 chips The public nature of the design has attracted a European company for train infrastructure

nanoFIP project status Extras

Considerations 37 o During the reprogramming process, the normal gateway tasks will be stopped o Reprogramming will only be used without LHC beam o ProASIC3 devices can be reprogrammed before the accumulation of ~100 Gy

o During the reprogramming process, the normal gateway tasks will be stopped o Reprogramming will only be used without LHC beam o ProASIC3 devices can be reprogrammed before the accumulation of ~100 Gy JTAG programmer SW 38.svf TMS TDI TDO TMS TDI Lib(X)SVF ISC Master

JTAG programmer SW 39.svf TMS TDI TDO TMS TDI … Lib(X)SVF ISC Master

JTAG programmer SW 40.svf TMS TDI TDO TMS TDI CRC FESCtrlFSS TMS TDI TDO Lib(X)SVF ISC TMS TDI … Master

FIELDBUS JTAG programmer SW 41.svf TMS TDI TDO TMS TDI … CRC FESCtrlFSS Lib(X)SVF ISC user Master

JTAG Controller HW 42 user Field TR Fiel drive FIELDBUS Master

JTAG Controller HW 43 user Field TR Fiel drive Master Cons FIELDBUS

JTAG Controller HW 44 user Field TR Fiel drive Master TCK TDO Cons Prod TDI TMS FIELDBUS TAPTAP

JTAG Controller HW 45 user Field TR Fiel drive Master Cons Prod TCK TMS TDI TDO FIELDBUS TAPTAP TDO

nanoFIP vs. microFIP 46 microFIP user logic sensor Signal Conditioner sensor Signal Conditioner is: is not: o Backwards compatible for the user sensor Signal Conditioner Master o Tailored to users’ needs o Common use of the chip and centralized support o rad-tol by design o nanoFIPs and microFIPs can co-exist under the same Master o Expected demand >2000 components nan FIP

nanoFIP vs. microFIP 47 nanoFIP user logic sensor Signal Conditioner sensor Signal Conditioner sensor Signal Conditioner Master is: is not: o Backwards compatible for the user o Tailored to users’ needs o Common use of the chip and centralized support o rad-tol by design o nanoFIPs and microFIPs can co-exist under the same Master o Expected demand >2000 components nan FIP

Project Organization & Some History ALSTOM-CERN contract with CERN purchasing ALSTOM’s design information. (2008) Concerns for the long-term availability of ALSTOM’s components; WorldFIP Taskforce set up. (2006) Project divided in different Work Packages: (2009) WP1: microFIP code preliminary interpretation (B. Todd, TE/MPE & E. van der Bij) WP2: project management documentation for the in-sourcing (E. van der Bij) WP3: functional specifications for microFIP’s replacement (E. van der Bij) WP4: rewrite & extend microFIP VHDL code WP5: write new code (P. Alvarez & E. Gousiou) WP6: test bench creation (G. Penacoba, TE/CRG ) WP7: design of a board for functional and radiation tests (HLP, France ) WP8: Radiation tests (CERN RadWG EN/STI & E. Gousiou) Taskforce conclusions: No technological alternative & in-sourcing of WorldFIP technology. (2007) 48

WorldFIP Frames Communication throughput for 1Mbps: FSS 2 bytes Ctrl 1 byte Id 2 bytes CRC 2 byte FES 1 byte 8 bytes * 8 bits* 1 us FSS 2 bytes Ctrl 1 byte Data 124 bytes CRC 2 byte FES 1 byte 130 bytes * 8 bits * 1us Master -> nanoFIP nanoFIP -> Master 1.1 ms for 124 data-bytes = 0.9 Mb/s Master -> nanoFIP nanoFIP -> Master turnaround time 10 us FSS 2 bytes Ctrl 1 byte Id 2 bytes CRC 2 byte FES 1 byte FSS 2 bytes Ctrl 1 byte Data 2 bytes CRC 2 byte FES 1 byte 10 us 138 us for 2 data-bytes = 0.1 Mb/s 49 turnaround time

Project Status Majority voter circuit: 50