1. Application Level Fault Tolerance and Detection
Principal Investigators:
C. Mani Krishna Israel Koren
Presented By:
Eric Ciocca
Architecture and Real-Time Systems (ARTS) Lab.
Department of Electrical and Computer Engineering
University of Massachusetts Amherst MA 01003

2. What is ALFTD? Application Level Fault Tolerance and Detection
ALFTD complements existing system or algorithm level fault tolerance by leveraging information available only at the application level
Using such application level semantic information significantly reduces the overall cost providing fault tolerance
ALFTD may be used alone or to supplement other fault detection schemes
Application Level Fault Tolerance and Detection

3. ALFTD Overview
Application Level Fault Tolerance and Detection allows for system survival of both data and system (instruction/hardware) faults.
System faults cause a process to eventually cease functioning
Data faults cause a process to continue running with incorrect results
ALFTD is scalable
The level of fault tolerance can be traded off with invested time overhead
Application Level Fault Tolerance and Detection

4. Principles of ALFTD P1 S4 P2 S1 P3 S2 P4 S3 Node 1 Node 2 Node 3
To provide system fault tolerance, every physical node runs its own work (P,primary) as well as a scaled-down copy of a neighboring node’s work (S,secondary)
If a fault should corrupt a process, the corresponding secondary of that task will still produce output, albeit at a lower (but acceptable) quality
Application Level Fault Tolerance and Detection

5. Principles of ALFTD The secondary processes can be scaled-down by
reducing the resolution of input data
reducing the precision of calculations
heuristically predicting results from previous iterations’ output
In some applications the secondary can be run optionally on an as-needed basis
If the corresponding primary is approaching a deadline miss
If the corresponding primary has been incapacitated
If the corresponding primary has produced faulty data
If faults are infrequent, an optional secondary will incur very little additional overhead
Application Level Fault Tolerance and Detection

6. ALFTD in OTIS
ALFTD was implemented into OTIS (Oribital Thermal Imaging Spectrometer) to test its viability as a fault tolerance and detection scheme
OTIS, part of the REE (Remote Exploration and Experimentation) program group from JPL, is intended to run on orbiting satellites
OTIS processes radiation data of a geographic area from a sensor array [input] and produces temperature and emissivity data [output]
Application Level Fault Tolerance and Detection

7. OTIS Structure OUTPUT M 1. MPI Starts MPI S 2. MPI Starts Slave and5 3 2 1
1. MPI Starts
MPI 4 S
2. MPI Starts Slave and
master processes S
3. Master sends tasks
4. Slave Calculations S
5. Slave Returns Results
Application Level Fault Tolerance and Detection

8. ALFTD in OTIS (cont’d) ALFTD is suited for remote applications,
As a software-based fault handling mechanism, it requires no extra hardware
The scaled secondaries require less power than full software redundancy
In OTIS, and other applications, ALFTD is passive, only requiring extra runtime in a fault case.
Application Level Fault Tolerance and Detection

9. ALFTD OTIS Structure ? OUTPUT M 1. MPI Starts MPI P15 3 2
1. MPI Starts 1 4
MPI P1
2. MPI Starts master and
slaves, primary and
secondary processes S2 P2 S3
3. Master sends tasks P3
4. Slave Calculations S1
5. Slave Returns Results
Application Level Fault Tolerance and Detection

10. Secondaries in OTIS
The secondary required for ALFTD is implemented to be functionally similar to the primary
Secondary scaling occurs through resolution reduction
OTIS’ “natural” temperature data input exhibits spatial locality
Points not directly calculated can be approximately estimated using interpolation between calculated points
Secondary processes have been tested at 20%-50% of the primary calculation overhead
While 50% affords better quality, 20% has less overhead
Application Level Fault Tolerance and Detection

11. Example of Secondary Resolution
100%
Secondary Resolution
50%
Secondary Resolution
33%
Secondary Resolution
25%
Secondary Resolution
(ALFTD Compensation for 10 rows in a sample dataset)
Application Level Fault Tolerance and Detection

12. Fault Detection
Output filters on the primary data determine when secondary validation is required
Output filters are created to check for application-specific trends in data
Aberrations from normal data characteristics can be considered to be the product of potentially faulty processes
OTIS relies on natural temperature characteristics to detect potentially faulty data
Spatial Locality: temperature changes gradually over small areas
Absolute Bounds: temperature should not exceed certain values
Application Level Fault Tolerance and Detection

13. Fault Detection (cont’d)
After the secondary has been run to validate a primary’s results, the “better” data is chosen according to the following logic grid:
Primary Results
Faultless
Ambiguous
Faulty
Primary
Secondary
Primary*
Secondary
Results
Application Level Fault Tolerance and Detection

14. Data Sets
Three data sets were chosen for their interesting characteristics
“Blob”
“Stripe”
“Spots”
Broad, unchanging areas with dark spots
Relatively undynamic except for one “stripe”
Turbulent spots may defy “spatial locality” predictions
Application Level Fault Tolerance and Detection

15. Fault Tolerance Results: “Spots”
Fault Tolerance with injected faults in “Spots”
Application Level Fault Tolerance and Detection

16. Fault Tolerance Results: “Spots” (cont’d)
Fault-Free Output
Faulty Output
25% ALFTD Computation Overhead
33% ALFTD Computation Overhead
50% ALFTD Computation Overhead
ALFTD-corrected faulty output
Application Level Fault Tolerance and Detection

17. Fault Tolerance Results: “Blob”
Fault Tolerance with injected faults in “Blob”
Application Level Fault Tolerance and Detection

18. Fault Tolerance Results: “Blob” (cont’d)
Fault-Free Output
Faulty Output
25% ALFTD Computation Overhead
33% ALFTD Computation Overhead
50% ALFTD Computation Overhead
ALFTD-corrected faulty output
Application Level Fault Tolerance and Detection

19. Fault Tolerance Results: “Stripe”
Difference Plots – faulty output versus faultless output
No ALFTD
25% ALFTD
Computation Overhead
33% ALFTD
Computation Overhead
50% ALFTD
Computation Overhead
No Error
Max Error
Application Level Fault Tolerance and Detection

20. Fault Tolerance Results: “Stripe”(cont’d)
Fault-Free Output
Faulty Output
25% ALFTD Computation Overhead
33% ALFTD Computation Overhead
50% ALFTD Computation Overhead
ALFTD-corrected faulty output
Application Level Fault Tolerance and Detection

21. Conclusion / Future Work
ALFTD has shown to be a cost-effective alternative to full redundancy
Improvements on the scheme will increase fault coverage and decrease secondary calculation overhead
OTIS has general application characteristics that will make its implementation a springboard to other, similar programs
ALFTD should continue to be effective in any programs that have predictable data characteristics
Application Level Fault Tolerance and Detection

22. Thank You! For additional information, please contact
Eric Ciocca
Israel Koren
C. Mani Krishna
Application Level Fault Tolerance and Detection