Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations Sasa Misailovic Joint work with Zeyuan Allen ZhuJonathan KelnerMartin Rinard MIT CSAIL

… … … … … Nodes represent computation Edges represent flow of data

… Functions – process individual data Reduction nodes – aggregate data … … … …

… min avg Functions – process individual data Reduction nodes – aggregate data … … … …

… min avg f2f2 f3f3 f1f1 … … … …

… min avg … … … …

Sampling inputs of reduction nodes Reductions consume fewer inputs … min avg … … … …

… min avg Sampling inputs of reduction nodes Reductions consume fewer inputs … … … …

Tradeoff Space Error Time

Tradeoff Space Error Time

Optimal Tradeoff Curve Error Time Using the tradeoff curve: Minimize time subject to error bound Minimize error subject to time bound

Our Result AnalysisOptimization Original program Error bound Optimized program Transformations

Outline Model of Computation Tradeoff Curve Construction Optimized Program Selection Related Work

Model of Computation … ff t uv w t uv w ffgg gg … … … … min avg

Model of Computation … ff t uv w t uv w ffgg gg … … … … min avg min 1 n n n f g t uv w avg mm

Structure of Computation Computation nodes DAGs of functions Functions: arbitrary code Process individual inputs Reduction nodes Aggregation functions Average, min, max, sum… Computation Tree Computation nodes and reduction nodes min 1 n n n f g t uv w avg mm

min Accuracy-Aware Transformations 1 n n n avg mm

min Accuracy-Aware Transformations 1 n n n avg mm …

min Accuracy-Aware Transformations 1 n n n avg mm …

min Accuracy-Aware Transformations 1 n n n avg mm …

min Program Configuration Vector Defines transformed program Functions: probability of executing each version Reductions: number of elements to sample 1 n n n avg mm

min Configuration Vector Specifies program version Functions: probability of executing each version Reductions: number of elements to sample 1 n n n avg mm Find optimal program = Find configuration vector that achieves optimal accuracy vs. performance tradeoff Find optimal program = Find configuration vector that achieves optimal accuracy vs. performance tradeoff

min 1 n n n avg mm Tradeoff Curve Construction: Algorithm

min Tradeoff Curve Construction: Algorithm 1 n n n avg mm

Tradeoff Curve Construction: Algorithm 1 mm n n n min avg

Tradeoff Curve Construction: Algorithm 1 mm n n n min avg

Tradeoff Curve Construction: Algorithm 1 n n n min avg m

Tradeoff Curve Construction: Algorithm 1 n n n min

Tradeoff Curve Construction: Algorithm 1 n n n min

Tradeoff Curve Construction: Algorithm 1 n n n min

Tradeoff Curve Construction: Algorithm 1 n min

Tradeoff Curve Construction: Algorithm

Computation Node Optimization (E 2,T 2 ) (E 0,T 0 )(E 1,T 1 ) Linear program

Computation Node Optimization (E 2,T 2 ) (E 0,T 0 )(E 1,T 1 ) Linear program

Computation Node Optimization (E 2,T 2 ) (E 0,T 0 )(E 1,T 1 ) Linear program

Computation Node Optimization (E 2,T 2 ) (E 0,T 0 )(E 1,T 1 ) Linear program

The Algorithm: Reduction Nodes m avg

The Algorithm: Reduction Nodes m avg

The Algorithm: Reduction Nodes m avg

The Algorithm: Reduction Nodes m avg

Approximate Tradeoff Curve Error Time Bidimensional Discretization

Approximating Tradeoff Curve Error Time Bidimensional Discretization

Approximating Tradeoff Curve Error Time

Approximating Tradeoff Curve Error Time

Approximating Tradeoff Curve Error Time

Approximating Tradeoff Curve Error Time

Properties of the Algorithm

Obtaining Optimized Programs

Related Work Accuracy-aware transformations Empirical justification: training/test input set [Rinard ICS ‘06, Rinard OOPSLA ’07, Ansel et al. PLDI ‘09, Misailovic et al. ICSE ’10, Baek & Chilimbi PLDI ‘10 Hoffmann et al. ASPLOS ‘11, Sidiroglou et al. FSE ‘11] Probabilistic accuracy analysis for loop perforation [Misailovic et al. SAS ‘11, Chaudhuri et al. FSE ‘11]

Related Work Accuracy-aware transformations Empirical justification: training/test input set [Rinard ICS ‘06, Rinard OOPSLA ’07, Ansel et al. PLDI ‘09, Misailovic et al. ICSE ’10, Baek & Chilimbi PLDI ‘10 Hoffmann et al. ASPLOS ‘11, Sidiroglou et al. FSE ‘11] Probabilistic accuracy analysis for loop perforation [Misailovic et al. SAS ‘11, Chaudhuri et al. FSE ‘11] Ensuring safety of transformed programs Separating critical and approximate parts of program [Carbin & Rinard ISSTA ’10, Sampson et al. PLDI ’11] Verifying relaxed semantics of programs [Carbin et al. CSAIL-TR ‘11] Analytic properties of programs [Majumdar & Saha RTSS ‘09, Chaudhuri et al. POPL’10, Ivancic et al. MEMOCODE ‘10, Reed & Pierce ICFP ’10, Chaudhuri & Solar-Lezama PLDI ’10, Chaudhuri et al. FSE ’11]

Summary Model of Computation Accuracy-aware program transformations Effects on overall accuracy and execution time Explore and Exploit Optimal Tradeoffs Approximate optimal tradeoff curve construction Polynomial, dynamic programming algorithm Randomized program configurations to achieve tradeoffs Envisioned Applications Image and video processing, numerical algorithms, queries on big data sets, machine learning, … Optimization, fault tolerance, dynamic adaptation