1. Mantis: Automatic Performance Prediction for Smartphone Applications Byung-Gon Chun Microsoft Yongin Kwon, Sangmin Lee, Hayoon Yi, Donghyun Kwon, Seungjun Yang, Ling Huang, Petros Maniatis, Mayur Naik, Yunheung Paik Seoul National University, UT Austin, Intel, Georgia Tech

2. Smartphone apps

3. Performance prediction problem Predict the execution time of a program on a given input before running it

4. Two kinds of approaches Differentiated by features chosen to model program’s performance Approach 1 : domain-specific program, automatically-extracted features Approach 2 : general-purpose program, manually-specified features

5. Performance predictor design dimensions Applicability Automation Accuracy Efficiency Approach 1 X O   Approach 2 O X   Mantis O O O O

6. Outline Motivation System overview Feature instrumentation Profiling Prediction modeling Predictor code generation Evaluation

7. Key insight of our approach Program execution runs often contain features that correlate with performance and are automatically computable efficiently

8. Automatically computable for (int i=0; i<n; ++i) { /* heavy computation */ }

9. Automatically computable for (int i=0; i<n; ++i) { if ( a[i] == true ) { /* heavy computation */ }

10. Cheaply computable for (int i=0; i<n; ++i) { if ( a[i] == true ) { /* heavy computation */ }

11. Key questions What are good program features for performance prediction? How do we model performance with relevant features? How do we compute features cheaply? How do we automatically generate predictor code?

12. System architecture Input data Profiler Feature values, perf. metrics Model generator Function over selected features Offline component Feature instrumentor Instrumented program Feature schemes Program No Predictor code Predictor code generator Predictor exe time <= TH Yes

13. System architecture Input data Profiler Feature values, perf. metrics Model generator Function over selected features Feature instrumentor Instrumented program Feature schemes Program No Predictor code Predictor code generator Predictor exe time <= TH Yes

14. Feature instrumentation Branch counts Loop counts Method call counts Variable values

15. Feature instrumentation Branch counts // original code if (flag) { lightweightCompute(); } else { heavyCompute(); } // instrumented code if (flag) { ++mantis_branch_cnt1; lightweightCompute(); } else { ++mantis_branch_cnt2; heavyCompute(); }

16. Feature instrumentation Loop counts // original code while(line=readLine()) { search(line); } // instrumented code while(line=readLine()) { ++mantis_loop_cnt; search(line); }

17. Feature instrumentation Method call counts // original code process(String arg) {... } // instrumented code process(String arg) { ++mantis_method_cnt;... }

18. Feature instrumentation Variable values // original code n=preprocess(); compute(n); // instrumented code n=preprocess(); mantis_n_sum += n; ++mantis_n_count; compute(n);

19. System architecture Input data Profiler Feature values, perf. metrics Model generator Function over selected features Feature instrumentor Instrumented program Feature schemes Program No Predictor code Predictor code generator Predictor exe time <= TH Yes

20. System architecture Input data Profiler Feature values, perf. metrics Model generator Function over selected features Feature instrumentor Instrumented program Feature schemes Program No Predictor code Predictor code generator Predictor exe time <= TH Yes

21. Performance modeling Expect a small set of features to explain performance among lots of features SPORE-FoBa (Huang et al. 2010)

22. Performance modeling SelectedFeatures = Prune_by_FoBa(Features) // FoBa (Zhang 2008) Terms = PolynomialExpansion(SelectedFeatures, d) // e.g., SelectedFeatures = {x1, x2} // (1 + x1 + x2) 2 => 1, x1, x2, x1 2, x2 2, x1x2 // ŷ = b 1 + b 2 x1 + b 3 x2 + b 4 x1 2 + b 5 x2 2 + b 6 x1x2 PerfModel = Choose_by_FoBa(Terms)

23. System architecture Input data Profiler Feature values, perf. metrics Model generator Function over selected features Feature instrumentor Instrumented program Feature schemes Program No Predictor code Predictor code generator Predictor exe time <= TH Yes

24. Predictor code generation: static program slicing A slice: a subprogram of the given program that yields the same value of variable v at program point p int x; if (b1) { x = 10; } else { if (b2) { x = 20; } else { x = 30; } x = 40; Print(x); program int x; x = 40; Print(x); slice

25. Predictor code generation: static program slicing Reader r = new Reader(file); String s; while((s = r.readLine()) != null) { mantis_loop_cnt++; // feature inst process(s); // expensive comp } program Reader r = new Reader(file); String s; while((s = r.readLine()) != null) { mantis_loop_cnt++; // feature inst } slice

26. Predictor code generation: static program slicer challenges Inter-procedural analysis Alias analysis Concurrency analysis Executable slices

27. Predictor code generation Intraprocedural: construct Program Dependency Graphs (PDGs) (HRB 1988) Interprocedural: construct a System Dependency Graph (SDG) (HRB 1988) Context sensitivity: augment the SDG with summary edges by running the SummaryEdge algorithm (RHS+ 1994) Run the two-pass reachability algorithm on the augmented SDG (HRB 1988) Translate intermediate code to final code

28. Outline Motivation System overview Feature instrumentation Profiling Prediction modeling Predictor code generation Evaluation

29. Mantis prototype Feature instrumentor Javassist Profiler Perl Script Model generator Octave Predictor code generator Joeq Datalog analysis & slicer Quad2Jasmin Jasmin bytecode quadcode jasmincode bytecode Android executable generator bytecode dx aapt DEX APK

30. Evaluation Prediction accuracy – Benefit of non-linear terms – Benefit of slicing Predictor execution time – Benefit of slicing Prediction on different hardware platforms Prediction under background load Mantis offline component processing time Prediction error = |ActualTime –PredictedExecutionTime| / ActualTime Prediction time = Predictor_Running_Time / ActualTime Prediction error = |ActualTime –PredictedExecutionTime| / ActualTime Prediction time = Predictor_Running_Time / ActualTime

31. Evaluation Prediction accuracy – Benefit of non-linear terms – Benefit of slicing Predictor execution time – Benefit of slicing Prediction on different hardware platforms Prediction under background load Mantis offline stage processing time

32. Experiment setup Applications: Encryptor, Path Routing, Spam Filter, Chess Engine, Ringtone Maker, and Face Detection Galaxy Nexus running Android 4.1.2 1000 randomly generated inputs for each application : 95-100% basic-block coverage 100 inputs for training 5% : predictor execution time threshold

33. Prediction error and time ApplicationPrediction error (%) Prediction time (%) Encryptor3.60.18 Path Routing4.21.34 Spam Filter2.80.51 Chess Engine11.91.03 Ringtone Maker2.20.20 Face Detection4.90.62 2.2-11.9% error by executing predictor costing at most 1.3% of app execution time

34. Prediction error and time ApplicationPrediction error (%) Prediction time (%) No. of detected features No. of chosen features Encryptor3.60.18282 Path Routing4.21.34681 Spam Filter2.80.51551 Chess Engine11.91.0310842 Ringtone Maker2.20.20741 Face Detection4.90.621072

35. Benefit of slicing Baselines: PE and BE Partial Execution (PE) : runs the instrumented program only until we obtain the chosen feature values Bounded Execution (BE) : runs the instrumented program for amount of time the Mantis predictor runs

36. Mantis vs. Partial Execution (PE) ApplicationMantis pred. time (%)PE pred. time (%) Encryptor0.20100.08 Path Routing1.3017.76 Spam Filter0.5099.39 Chess Engine1.0369.63 Ringtone Maker0.200.04 Face Detection0.610.17

37. Mantis vs. Bounded Execution (BE) ApplicationMantis pred. error (%)BE pred. error (%) Encryptor3.656.0 Path Routing4.264.0 Spam Filter2.836.2 Chess Engine11.926.1 Ringtone Maker2.2 Face Detection4.9

38. Related work Predicting performance or resource consumption in databases, cluster computing, networking, program optimization, etc. Non-trivial features: program complexity, hardware simulation specificity, cooperative bug finding Worst-case behavior prediction in embedded/real-time systems

39. Conclusion Mantis: a framework that automatically generates accurate and efficient program performance predictors – Extracts information from program executions – Models performance with machine learning – Generates predictors with program analysis – Uses even features that occur late in execution

40. Backup Slides

41. Selected features and generated models ApplicationSelected featuresGenerated model Encryptormatrix-key size(f 1 ) loop count of encryption (f 2 ) c 0 f 1 2 f 2 + c 1 f 1 2 + c 2 f 2 + c 3 Path Routingbuild map loop count (f 1 )c 0 f 1 2 + c 1 f 1 + c 2 Spam Filterinner loop count of sorting (f 1 ) c 0 f 1 + c 1 Chess Engineno. of second-level game- tree nodes (f 1 ), no. of chess pieces (f 2 ) c 0 f 1 3 + c 1 f 1 f 2 + c 2 f 2 2 + c 3 Ringtone Makercut interval length (f 1 )c 0 f 1 + c 1 Face Detectionwidth (f 1 ), height (f 2 )c 0 f 1 f 2 + c 1 f 2 2 + c 2

42. Prediction errors varying the number of input samples

43. Prediction error and time of Mantis running with Galaxy S2 and Galaxy S3 ApplicationGalaxy S2Galaxy S3 Prediction error (%) Prediction time (%) Prediction error (%) Prediction time (%) Encryptor4.60.353.40.08 Path Routing4.13.074.21.28 Spam Filter5.41.522.20.52 Chess Engine9.71.4213.21.38 Ringtone Maker3.70.514.80.20 Face Detection5.11.285.00.69

44. Prediction error under background CPU-intensive loads ApplicationMantis pred. error (%) for the x% background CPU load x=0x=50x=75x=99 Encryptor3.67.510.521.3 Path Routing4.25.35.86.7 Spam Filter2.84.75.25.8 Chess Engine11.913.515.315.8 Ringtone Maker2.22.33.03.1 Face Detection4.95.35.65.8

45. Predictor code generation: static program slicer challenges Inter-procedural analysis – Context-sensitive inter-procedural algorithm Alias analysis – Flow- and context-insensitive may-alias analysis with object allocation site heap abstraction Concurrency analysis – May-alias Executable slices – A set of rules we identified

46. Mantis offline stage processing time (in seconds) ApplicationProf.Model gen. SlicingTestTotalIter. Encryptor23731811739129003 Path Routing36328114145193 Spam Filter135106632142 Chess Engine6624102296016231424601183 Ringtone Maker2074194565266591 Face Detection143713641217980414