Lecture 9 Dynamic Compilation Motivation & Background Overview Compilation Policy Partial Method Compilation Partial Dead Code Elimination Escape Analysis Results “Partial Method Compilation Using Dynamic Profile Information”, John Whaley, OOPSLA 01 Advanced Compilers L9. Dynamic Compilation

I. Goals of This Lecture Beyond static compilation Example of a complete system Use of data flow techniques in a new context Experimental approach Advanced Compilers L9. Dynamic Compilation

Static/Dynamic Compiler: high-level  binary, static Interpreter: high-level, emulate, dynamic Dynamic compilation: high-level  binary, dynamic machine-independent, dynamic loading cross-module optimization Specialize program using runtime information (without profiling) Interpreter: matlab, Binary translator: Advanced Compilers L9. Dynamic Compilation

High-Level/Binary Binary translator: Binary-binary; mostly dynamic Run “as-is” Software migration (x86  alpha, sun, transmeta; 68000  powerPC  x86) Virtualization (make hardware virtualizable) Dynamic optimization (Dynamo Rio) Security (execute out of code in a cache that is “protected”) Interpreter: matlab, Binary translator: Advanced Compilers L9. Dynamic Compilation

Closed-world vs. Open-world Closed-world assumption (most static compilers) All code is available a priori for analysis and compilation. Open-world assumption (most dynamic compilers) Code is not available; arbitrary code can be loaded at run time. Open-world assumption precludes many optimization opportunities. Solution: Optimistically assume the best case, but provide a way out if necessary. Advanced Compilers L9. Dynamic Compilation

II. Overview in Dynamic Compilation Interpretation/Compilation policy decisions Choosing what and how to compile Collecting runtime information Instrumentation Sampling Exploiting runtime information frequently-executed code paths Advanced Compilers L9. Dynamic Compilation

Speculative Inlining Virtual call sites are deadly. Kill optimization opportunities Virtual dispatch is expensive on modern CPUs Very common in object-oriented code Speculatively inline the most likely call target based on class hierarchy or profile information. Many virtual call sites have only one target, so this technique is very effective in practice. Advanced Compilers L9. Dynamic Compilation

III. Compilation Policy ∆Ttotal = Tcompile – (nexecutions * Timprovement) If ∆Ttotal is negative, our compilation policy decision was effective. We can try to: Reduce Tcompile (faster compile times) Increase Timprovement (generate better code) Focus on large nexecutions (compile hot spots) 80/20 rule: Pareto Principle 20% of the work for 80% of the advantage Advanced Compilers L9. Dynamic Compilation

Latency vs. Throughput Tradeoff: startup speed vs. execution performance Startup speed Execution performance Interpreter Best Poor ‘Quick’ compiler Fair Optimizing compiler Advanced Compilers L9. Dynamic Compilation

Multi-Stage Dynamic Compilation System interpreted code Stage 1: when execution count = t1 (e.g. 2000) compiled code Execution count is the sum of method invocations & back edges executed. Stage 2: Our dynamic compilation system uses a three stage system. In Stage 1, all methods are interpreted. When the execution count for a method exceeds a certain threshold, t1, we transition to Stage 2, a compiled version. When the execution count hits another threshold, t2, we transition to Stage 3, a fully optimized version of the method. when execution count = t2 (e.g. 25000) fully optimized code Stage 3: Advanced Compilers L9. Dynamic Compilation

Granularity of Compilation Compilation takes time proportional to the amount of code being compiled. Many optimizations are not linear. Methods can be large, especially after inlining. Cutting inlining too much hurts performance considerably. Even “hot” methods typically contain some code that is rarely or never executed. All of the previous techniques compile a method-at-a-time. However, when we compile a method at a time, we really don’t have much choice in the matter. Methods can be large, especially after performing inlining, and cutting inlining too much can hurt performance considerably. Even when we are frugal about inlining, methods can still become very large. Advanced Compilers L9. Dynamic Compilation

Example: SpecJVM db Hot loop void read_db(String fn) { int n = 0, act = 0; byte buffer[] = null; try { FileInputStream sif = new FileInputStream(fn); buffer = new byte[n]; while ((b = sif.read(buffer, act, n-act))>0) { act = act + b; } sif.close(); if (act != n) { /* lots of error handling code, rare */ } catch (IOException ioe) { Hot loop Here’s an example from the SpecJVM db benchmark. This method contains a hot loop which contains a method call. However… Advanced Compilers L9. Dynamic Compilation

Example: SpecJVM db Lots of rare code! void read_db(String fn) { int n = 0, act = 0; byte buffer[] = null; try { FileInputStream sif = new FileInputStream(fn); buffer = new byte[n]; while ((b = sif.read(buffer, act, n-act))>0) { act = act + b; } sif.close(); if (act != n) { /* lots of error handling code, rare */ } catch (IOException ioe) { Lots of rare code! This method also contains lots of error handling code that is rarely or never executed. The “read” method in the inner loop also itself contains hot and not-hot portions. Advanced Compilers L9. Dynamic Compilation

Optimize hot “regions”, not methods Optimize only the most frequently executed segments within a method. Simple technique: any basic block executed during Stage 2 is said to be hot. Beneficial secondary effect of improving optimization opportunities on the common paths. The problem is that the regions that are important to compile have nothing to do with the method boundaries. By using a method granularity, the compiler wastes a lot of time compiling and optimizing large pieces of code that do not matter. Advanced Compilers L9. Dynamic Compilation

Method-at-a-time Strategy % of basic blocks compiled Check is an anomaly because it is a code just to check the correctness of the JVM – not indicative of real programs. Mtrt – raytracer – tight innermost loop Javac is real. Now let’s compare the performance of a method-at-a-time strategy with an optimal strategy. This graph shows the percentage of basic blocks that are compiled using a method-at-a-time strategy at different compilation threshold values. The vertical axis gives the percentage of basic blocks that are compiled, and the horizontal axis gives the execution thresholds. Notice that the horizontal axis is on a semi-log scale. At threshold one, all basic blocks in all methods that have ever been executed are compiled, so the value is one hundred percent. At threshold 5000, the percentage ranges from about 3% up to about 75%, depending on the application. execution threshold Advanced Compilers L9. Dynamic Compilation

Actual Basic Blocks Executed % of basic blocks Now let’s take a look at how many basic blocks are actually executed at different threshold values. The first thing to notice is that even at threshold one, on average only slightly more than half of the basic blocks are ever executed. As the threshold value increases, the percentage continues to drop. This shows that a large amount of code is rare, even in hot methods. Note that because the horizontal axis is a log scale, we see a rather gentle descent. If this were graphed on a linear scale, the dropoff would be much more sharp. execution threshold Advanced Compilers L9. Dynamic Compilation

Dynamic Code Transformations Compiling partial methods Partial dead code elimination Escape analysis Advanced Compilers L9. Dynamic Compilation

IV. Partial Method Compilation Based on profile data, determine the set of rare blocks. Use code coverage information from the first compiled version First, based on profile data, we determine the set of rare blocks. We use the code coverage information I described earlier from the Stage 2. Any basic block that is never executed in Stage 2 is said to be rare. Advanced Compilers L9. Dynamic Compilation

Partial Method Compilation Perform live variable analysis. Determine the set of live variables at rare block entry points. live: x,y,z Next, before we perform any code transformations or optimizations, we perform live variable analysis on the code. We determine the set of live variables at each of the rare block entry points. Advanced Compilers L9. Dynamic Compilation

Partial Method Compilation Redirect the control flow edges that targeted rare blocks, and remove the rare blocks. to interpreter… Third, we redirect the control flow edges that targeted the rare blocks, and then remove the rare blocks as unreachable code. Advanced Compilers L9. Dynamic Compilation

Partial Method Compilation Perform compilation normally. Analyses treat the interpreter transfer point as an unanalyzable method call. Next, we perform compilation normally. All of the analyses and optimizations simply treat the interpreter transfer point as an unanalyzable method call. One of the most attractive aspects of this technique is that it doesn’t require any changes to any of the existing optimizations and analyses. Advanced Compilers L9. Dynamic Compilation

Partial Method Compilation Record a map for each interpreter transfer point. In code generation, generate a map that specifies the location, in registers or memory, of each of the live variables. Maps are typically < 100 bytes live: x,y,z x: sp - 4 Next, in code generation, we record a map for each interpreter transfer point. This map specifies the location, in registers or memory, where the live variable can be found. These maps are typically less than 100 bytes each. y: R1 z: sp - 8 Advanced Compilers L9. Dynamic Compilation

V. Partial Dead Code Elimination Move computation that is only live on a rare path into the rare block, saving computation in the common case. The first optimization is partial dead code elimination. We made a slight modification to our existing dead code elimination algorithm to treat rare blocks specially. We move computation that is only live on a rare path into the rare block. This saves computation in the common case. Advanced Compilers L9. Dynamic Compilation

Partial Dead Code Example if (rare branch 1){ ... z = x + y; } if (rare branch 2){ a = x + z; if (rare branch 1) { x = 0; ... z = x + y; } if (rare branch 2) { a = x + z; Here is a really simple example. X is defined on the main path, but only used in rare paths. Therefore, the algorithm will copy the instruction to the rare blocks … Advanced Compilers L9. Dynamic Compilation

VI. Escape Analysis Escape analysis finds objects that do not escape a method or a thread. “Captured” by method: can be allocated on the stack or in registers. “Captured” by thread: can avoid synchronization operations. All Java objects are normally heap allocated, so this is a big win. We also modified our pointer and escape analysis to support rare block information. Now, treating an entrance to the rare path as a method call is a conservative assumption. The method call could perform arbitrary operations, much more than the original code. Now typically, this does not matter because there are no merges Advanced Compilers L9. Dynamic Compilation

Escape Analysis Stack allocate objects that don’t escape in the common blocks. Eliminate synchronization on objects that don’t escape the common blocks. If a branch to a rare block is taken: Copy stack-allocated objects to the heap and update pointers. Reapply eliminated synchronizations. Advanced Compilers L9. Dynamic Compilation

VII. Run Time Improvement This graph shows the run time improvement by using our technique. Each benchmark has three bars. The first bar is the original run time; these are all normalized to 100%. The second bar shows the run time using our partial method compilation technique. The third bar adds the two optimizations that I described earlier. The partial method compilation technique by itself improves run times by an average of 5%. Adding the two optimizations I described earlier improves First bar: original (Whole method opt) Second bar: Partial Method Comp (PMC) Third bar: PMC + opts Bottom bar: Execution time if code was compiled/opt. from the beginning Advanced Compilers L9. Dynamic Compilation