BL-TMR and Mitigation Approaches for FPGAs Mike Wirthlin BYU #2: "BL-TMR and Mitigation Approaches for FPGAs" It is well known that active mitigation techniques such as TMR are needed to protect FPGA circuits when operating in harsh radiation environments. The BL-TMR tool was developed jointly by BYU and the Los Alamos National Laboratory (LANL) to automatically insert TMR structures within FPGA circuits. This tool has been used in a number of experiments both in spacecraft and in radiation testing. These results suggest that the BL-TMR tool, coupled with configuration scrubbing, significantly improves the reliability of FPGA circuits. This presentation will describe the objectives of the BL-TMR tool, the mitigation approach it applies, and reliability improving results that have been demonstrated by the tool. In addition, this presentation will summarize a number of other mitigation approaches that have been or could be applied to FPGA circuits.

1. TMR Overview

Triple Modular Redundancy (TMR) A form of N Modular Redundancy Triplicate hardware resources Majority Vote on hardware outputs Tolerates any single fault Tolerates many multiple fault combinations Mike Wirthlin, BYU

TMR Granularity System Level Device Level Module Level RTL Level process(clk_int_a) begin if clk_int_a'event and clk_int_a='1' then locked_d_a <= locked_a_int; if (all_locked_a = '0') then all_locked_a <= (locked_d_a and locked_d_b and locked_d_c); else all_locked_a <= tmr_voter( locked_d_a, locked_d_b, locked_d_c); end if; end process Module Level RTL Level Logic Level Mike Wirthlin, BYU

TMR Reliability TMR has lower reliability than non-redundant for long mission times Effective TMR almost always is coupled with “repair” Non-redundant TMR Mike Wirthlin, BYU

TMR + Repair = Very Reliable! Mike Wirthlin, BYU

FPGA Configuration “Repair” x Configuration Upset Mike Wirthlin, BYU

FPGA Configuration “Repair” x Configuration Upset Repaired Mike Wirthlin, BYU

TMR & Scrubbing Example Mike Wirthlin, BYU

Voters Before Flip Flops Mike Wirthlin, BYU

Voters After Flip-Flops Mike Wirthlin, BYU

More Frequent Voting Mike Wirthlin, BYU

TMR Synchronization Fault repair through scrubbing Fixes the cause of the error Does NOT fix the state of the circuit State of circuit must be synchronized to working circuits Mike Wirthlin, BYU

Synchronizing Voters Mike Wirthlin, BYU

Synchronizing Voters Mike Wirthlin, BYU

Clock Domain Crossing Mike Wirthlin, BYU

Partial TMR TMR may be applied selectively Failures in some circuit areas cause more harm than others Some circuit areas are protected by other SEE mitigation techniques (TMR not needed) Challenge: deciding where to apply TMR Circuits with feedback (state machines) Circuits with high “functional influence” Mike Wirthlin, BYU

Persistent vs. Non-persistent Upset Some upsets repaired through scrubbing Non-persistent upsets: repairable through scrubbing Persistent upsets: requires reconfiguration Non-Persistent Upset Persistent Upset Bitstream Repair Upset Bitstream Repair Upset = First type of functional error is non-persistent error = Example - DSP circuit output streams - correct operation -> difference is zero - incorrect operation -> difference is non-zero = This plot - non-zero for 64 time steps - Upset -> outputs diverge - @ 130 scrubbing repairs - functional errors disappear - circuit and functional data returned to normal *without* reset = We call this a non-persistent error or temporary interruption of service The first type of functional error is a non-persistent error. To illustrate this type of error, we collected output data streams from identical copies of a real DSP circuit. When the circuit is operating correctly, the difference is zero. When functional errors are present, the difference is non-zero. In this plot, the DUT and golden circuit are in sync for the first 64 time-steps. At time cycle 65, an SEU upsets a configuration bit in the DUT. Almost immediately, the outputs of the two circuits diverge due to functional errors and the difference is non-zero. At time cycle 130 scrubbing repairs the upset bit and shortly thereafter the functional errors disappear. The circuit and its functional data returned back to normal *without* the need for a system reset. Due to the upset bit, the system experienced a temporary interruption of service. = An SEU in the non-persistent cross section causes a temporary interruption of service (non-persistent error) = After scrubbing repairs configuration bitstream, output errors disappear error magnitude error magnitude Correct Output Incorrect Output time cycle time cycle

Persistent Circuit Structures Logic FF Logic FF Logic FF Logic FF Logic FF Non-Persistent Structure – Feed-forward Persistent Structures – Contribute to feedback Partial TMR – Priority given to persistent structures Non-persistent Circuit Structures Feed-forward logic Errors are flushed out of the circuit once scrubbing repairs the configuration Persistent Circuit Structures Those that contribute to feedback Feedback logic Logic that feeds into feedback logic Errors are trapped in the feedback loops (state of design) even after configuration is repaired by scrubbing A computer can be (relatively) easily programmed to find feedback in circuits Static analysis of circuit done at netlist (EDIF) level Same analysis for any circuit Some gains could be realized if customizing analysis to circuit type, but would be much more complicated. Remember, one of the benefits of TMR is its simplicity! Our Strategy for Partial Mitigation is: Focus on (give priority to) the persistent cross section of a design Still gain the desired improvements in availability/reliability (at lower cost than TMR) This only works for designs that can tolerate temporary data loss (not complete reliability like Full TMR) Mike Wirthlin, BYU

Full TMR TMR Reliable (when combined with configuration scrubbing) Simple to implement Expensive (in terms of area, power, etc.)

Partial TMR BYU and LANL have developed a tool to find the persistent components of a design and focus mitigation on them Applies Triple Modular Redundancy to these persistent sections of a design Mike Wirthlin, BYU

TMR Automation TMR is relatively easy to automate Analyze design Replicate resources Insert voters Verify resulting circuit Different Strategies for Automated TMR Netlist level HDL Level Selective/Partial Several tools available for Automatic TMR Mike Wirthlin, BYU

Automated TMR Tools BL-TMR (and other several other academic projects) Mike Wirthlin, BYU

2. BL-TMR

BL-TMR BYU-LANL TMR Tool BYU-LANL Triple Modular Redundancy Developed at BYU under the support of Los Alamos National Laboratory (Cibola Flight Experiment) Used to test TMR on many designs Fault injection, Radiation testing, in Orbit Testbed for experimenting with various TMR application techniques (used for research) Mike Wirthlin, BYU

Ongoing Development Based on the success of BL-TMR, additional funding has been provided to extend BL-TMR for additional devices, environments, and address new problems Commercial companies concerned about SER rates Cisco Systems High Energy Physics Brookhaven National Laboratory (BNL), CERN Space system developers SEAKR systems, Sandia, LANL, Lockheed Martin Interest in BL-TMR is growing Commercialization currently under consideration

BL-TMR (BYU/LANL TMR) EDIF data structure & API Available tools: Parse, represent, and manipulate EDIF Available tools: EDIF parser Half-latch removal SRL replacement Feedback cutset tool Full and partial TMR Detection circuitry insertion EDIF output Project size ~50 Java packages 350+ Java classes 478,401 lines of code Includes contributions from CHREC member LANL [brian@tiger:test] java -cp ~/jars/BLTmr.jar byucc.edif.tools.tmr.FlattenTMR ../no_tmr/synth/counters80.edf --removeHL --full_tmr --technology virtex -p xcv1000fg680 --log counters80.log BLTmr Tool version 0.2.3, 12 Oct 2006 Search for EDIF files in these directories: [.] Parsing file ../no_tmr/synth/counters80.edf Removing half-latches... Flattening Flattened circuit contains 3451 primitives, 3461 nets, and 13692 net connections Processing: ASUF 1.0 Forcing triplication of instance safeConstantCell_zero Analyzing design . . . Full TMR requested. Triplicating design . . . domainreport=BLTmr_domain_report.txt Added 1931 voters. 3431 instances out of 3451 cells triplicated (99% coverage) 6862 new instances added to design. 3431 nets triplicated (6862 new nets added). 0 ports triplicated. Tools and code available at: http://sourceforge.net/projects/byuediftools/ Mike Wirthlin, BYU

BL-TMR User Control Provides significant control to user Can be scripted for complex BL-TMR runs Usage: java byucc.edif.tools.tmr.FlattenTMR <input_file> [(-o|--output) <output_file>] [(-d|--dir) dir1,dir2,...,dirN ] [(-f|--file) file1,file2,...,fileN ] [--tmrSuffix suffix1,suffix2,...,suffixN ] [--full_tmr] [--tmr_inports] [--tmr_outports] [--no_tmr_p port1,port2,...,portN ] [--tmr_c cell_type1,cell_type2,...,cell_typeN ] [--tmr_i cell_instance1,cell_instance2,...,cell_instanceN ] [--no_tmr_c cell_type1,cell_type2,...,cell_typeN ] [--no_tmr_i cell_instance1,cell_instance2,...,cell_instanceN ] [--notmrFeedback] [--notmrInputToFeedback] [--notmrFeedBackOutput] [--notmrFeedForward] [--noInoutCheck] [--SCCSortType <{1|2|3}>] [--doSCCDecomposition] [--inputAdditionType <{1|2|3}>] [--outputAdditionType <{1|2|3}>] [--mergeFactor <mergeFactor>] [--optimizationFactor <optimizationFactor>] [--factorType <{DUF|UEF|ASUF}>] [--factorValue <factorValue>] [--low <low>] [--high <high>] [--inc <inc>] [--removeHL] [--hlConst <{0|1}>] [--hlUsePort <hlPortName>] [--technology <{virtex|virtex2}>] [(-p|--part) <part>] [--summary] [--log <logfile>] [--domainReport <domainReport>] [--writeConfig[:<config_file>]] [-h|--help] [-v|--version] For detailed usage, try `--help'

Sample Execution [brian@tiger:test] java -cp ~/jars/BLTmr.jar byucc.edif.tools.tmr.FlattenTMR ../no_tmr/synth/counters80.edf --removeHL --full_tmr --technology virtex -p xcv1000fg680 --log counters80.log BLTmr Tool version 0.2.3, 12 Oct 2006 Search for EDIF files in these directories: [.] Parsing file ../no_tmr/synth/counters80.edf Removing half-latches... Flattening Flattened circuit contains 3451 primitives, 3461 nets, and 13692 net connections Processing: ASUF 1.0 Forcing triplication of instance safeConstantCell_zero Analyzing design . . . Full TMR requested. Triplicating design . . . domainreport=BLTmr_domain_report.txt Added 1931 voters. 3431 instances out of 3451 cells triplicated (99% coverage) 6862 new instances added to design. 3431 nets triplicated (6862 new nets added). 0 ports triplicated.

% Increase in Critical Path Cost of TMR Size Increase Critical Path Before TMR After TMR % Increase in Critical Path blowfish 3.1X 28.3 ns 31.7 ns 12.0% des3 3.4X 11.1 ns 13.6 ns 22.5% qpsk 80.0 ns 83.9 ns 4.9% free6502 3.3X 29.6 ns 33.1 ns 11.8% T80 27.8 ns 33.7 ns 21.2% macfir 3.9X 14.4 ns 19.5 ns 35.4% serial_divide 4.1X 9.2 ns 12.2 ns 32.6% planet 10.9 ns 12.6 ns 15.6% s1488 9.9 ns 12.0 ns s1494 10.4 ns 17.3% s298 15.8 ns 19.1 ns 20.9% tbk 10.3 ns 12.9 ns 25.2% synthetic 4.0X 5.1% lfsrs 6.3X 9.0 ns 12.7 ns 41.1% ssra_core 3.5X 6.1 ns 7.2 ns 18.0% mean 3.6X 8.17 ns 12.08 ns 16.0% Mike Wirthlin, BYU

BL-TMR Incremental Results This graph shows the sensitivity and persistence of 200 partially-mitigated designs (from the same source design) Very fine-grain addition of redundant logic (in this case about 76 LUTs per increment, can do finer) No matter where your area constraint is, this fine-grained addition of logic gets very near the line Maximum reliability for the given area cost Notice the BL-TMR tool gives priority to the persistent sections of the design, eliminating persistence first Persistence eliminated after adding 61% redundancy (floor of ~670 bits) Sensitiity eliminated after adding 210% redundancy (floor of ~2,400 bits) Overhead Logic = (LUTs + FFs) above the amount the original design has 200 Incremental outputs of the BL-TMR tool 76 (LUTs + FFs) between design points 1262 sensitive bits between design points Mike Wirthlin, BYU

3. Design Flow

Design Flow RTL RTL Synthesis EDIF Netlist Signal List pTMR Parameters pTMR Tool pTMR Property Tags Modified Netlist Tagged EDIF Netlist Xilinx Map, Par, etc. FPGA bitfile

pTMR Steps Component Merging Design Flattening Graph Creation and Analysis IOB Analysis Clock Domain Analysis Instance Removal Feedback Analysis Illegal Crossing identification TMR Prioritization & Selection Voter Selection Netlist generation

11. Netlist Generation Circuit generated from pTMR rules Cells triplicated Voters inserted Netlist created for new circuit

3. Verifying BL-TMR

Fault Injection Configure user design onto two identical FPGAs Compare results of two designs using Comparator FPGA Insert configuration SEUs into design under test (FPGA2) and compare results If discrepancies between FPGAs are found, record configuration error FPGA 1 FPGA 2 Comparator Mike Wirthlin, BYU

SEU Insertion Example #1 Apply test vector to circuit input x FPGA 1 FPGA 2 x Insert configuration SEU into FPGA #2 FPGA1  FPGA2 x Compare circuit results Comparator Mike Wirthlin, BYU

Experimental Results – Design #2 Synthetic (LFSR/Mult) FPGA Editor Layout Sensitivity Map Persistence Map Unmitigated 3,005 slices (24%) 254,840 (4.39%) 46,368 (0.80%) top row -> Unmitigated Synthetic Design original: 24% of logic resources persistent is 5.5 times smaller than dynamic bottom row -> Synthetic design w/ TMR applied to as much of the design as possible (excluding clock, I/O) 97% utilization or roughly 370% more resources dynamic (pers & non-pers) is over 99% smaller than original Again, persistent x-section decreased by a factor of 100! (also sensitive) Caveats Inputs, Outputs and Clock were not triplicated Full TMR Applied 12,165 slices (99%) 2,395 (0.041%) 671 (0.005%) Mike Wirthlin, BYU

LANL Cibola Flight Experiment Los Alamos National Laboratory technology pathfinder validate FPGAs for high performance computing Investigate SEU behavior of Xilinx Virtex FPGAs Several BYU experiments validated in orbit TMR (including BL-TMR tool) Duplication with Compare DRAM controllers Cibola Flight Experiment 560 km, 35.4º inclination Mike Wirthlin, BYU

Sandia MISSE-8 BYU Experiments on ISS Under direction of Sandia National Laboratory BYU Experiments on ISS TMR PicoBlaze (Successful mitigation event!) Smart signal detection Reduced Precision Redundancy BRAM Scrubbing & BRAM ECC Endeavor (STS-134) May 16, 2012 Photo courtesy of NASA V4 FX60 V5QV (SIRF) Dr. Wirthlin to present Photo courtesy of Sandia National Labs Photo courtesy of NASA Mike Wirthlin, BYU

Radiation Testing Apply Ionizing Radiation to Design with TMR Verify accuracy of artificial simulator Identify upset in non-configuration state Identify other failure modes UC Davis, Crocker Nuclear Laboratory Medium-energy particle accelerator (76-inch cyclotron) 63 MeV proton source Flux: 1e7 particles/cm2/second: (~1 upset/second) 16 hour test (~25,000 upsets) Proton Beam FPGA Board Mike Wirthlin, BYU

5. TMR Summary Pros: Cons Significant improvements in reliability Easy to apply (limited design effort) Can be applied selectively Cons Requires significant hardware resources Negative impact on timing Difficult to verify Mike Wirthlin, BYU

Alternatives to TMR Exploit specific circuit structures/styles Memories, state machines, processors, etc. Arithmetic structures Detection+ Detecting a fault quickly opens up many lower cost mitigation strategies Temporal Redundancy Duplication with Compare Mike Wirthlin, BYU

Future Plans Clock domain aware TMR Timing aware TMR Improved support for clock and I/O resources Integrated Duplication with Compare (DWC) More frequent voting NMR (5-MR, 7-MR, etc.) Support for New FPGA Architectures Improved verification (formal verification) GUI support Improved partial TMR selection (Algorithmic pTMR)

Questions? Mike Wirthlin, BYU