2/23/2019 A Practical Approach for Handling Soft Errors in Iterative Applications Jiaqi Liu and Gagan Agrawal Department of Computer Science and Engineering The Ohio State University 2/23/2019

Motivation Impact to Scientific Applications 2/23/2019 Motivation Soft Errors have become a threat in large scale systems Unpredictability Result from various factors Packaging material Radiation Voltage fluctuation Temperature Defective Hardware Transiency Silent Data Corruption (SDC) might not make application crash, but can result in erroneous output Impact to Scientific Applications Scientific applications expect accurate results – low tolerance to soft errors 2/23/2019

2/23/2019 Motivating Study Inspect Impact of Soft Error to iterative applications Inject bit flips in different bit of variable in different execution stage Observe how bit flip in different bits impacts the output Different execution stage (denoted as percentage of total iterations) impacts the output Mimics Single Event Upset (SEU) with only one bit flip in one execution By linearization method, we can get rid of the pointers. 2/23/2019

against the output from the normal execution. 2/23/2019 Impact from SEU to Iterative Applications By linearization method, we can get rid of the pointers. Impact of SEU to Sobel application: measured in Normalized Relative Difference against the output from the normal execution. 2/23/2019

Observation Significant errors occur in higher order bit flip 2/23/2019 Observation Significant errors occur in higher order bit flip Error from lower order bit flips are trivial and usually acceptable Errors occur in different iteration tends to affect the final output Early errors tend to be averaged by the iterative algorithm By linearization method, we can get rid of the pointers. 2/23/2019

Signature Based Detection 2/23/2019 Signature Based Detection Monitor convergence criteria (residual/signature) of the algorithm Normal execution leads to continues convergence Unexpected increase/decrease in convergence criteria is a signature of SDC Apply periodical checkpoint to recover in presence of SDC By linearization method, we can get rid of the pointers. 2/23/2019

Signature Based Detection 2/23/2019 Signature Based Detection Main Idea: Check for signature in each (or some) iteration Periodically take checkpoint If signature of soft error is detected, recover from the latest checkpoint By linearization method, we can get rid of the pointers. 2/23/2019

Partial Replication Identify critical session/iterations (CS) during the execution Replicate computation in critical sections Vote for correct result at the end of CS Avoid major impact of SDC 2/23/2019

Experiment Result Applications Datasets Evaluation Jacobi, Sobel, Conjugated Gradient (CG) Gauss Seidel (GS) Successive Over-Relaxation (SOR) Datasets Evaluation Effectiveness, Improvement from Partial Replication and Overhead 2/23/2019

Experiment Result - Effectiveness TP: True Positive FP: False Positive FN: False Negative F-Score over 90% is Considered as Effective Algorithm 2/23/2019

Experiment Result - Effectiveness 2/23/2019

Experiment Result - Overhead 2/23/2019 Experiment Result - Overhead Distribution of Execution Times: Signature Analysis + Checkpoint 2/23/2019

Experiment Result – Applying Partial Rep. 2/23/2019 Experiment Result – Applying Partial Rep. Results for Partial Replication and Partial Replication + Signature Analysis (including Checkpointing and Restart) on 32 nodes: Sobel and CG. Sobel replicates the last 40% of the execution while CG replicates the first 40%. 2/23/2019

Thanks 2/23/2019

Back up Slides I 2/23/2019

Motivation Traditional Checkpoint & Restart no longer satisfies the need of fault tolerance in large scale system. Huge amount of waste under low MTBF & large number of nodes Workload distribution on 100K nodes Need an alternative solution for larger system scale Decreasing MTBF Increasing checkpoint size due to increasing system size 2/23/2019