1. Speculative execution Landon Cox April 13, 2016

2. Making disk accesses tolerable Basic idea Remove disk accesses from critical path Transform disk latencies into memory latencies Requires predicting the future Caching Assume future accesses will look like past accesses Process D1D1 D1D1 D2D2 D2D2 Operating system Get D1 D1D1 D1D1 D1D1 D1D1 How long does the initial access take?~10ms

3. Making disk accesses tolerable Basic idea Remove disk accesses from critical path Transform disk latencies into memory latencies Requires predicting the future Caching Assume future accesses will look like past accesses Process D1D1 D1D1 D2D2 D2D2 Operating system Get D1 D1D1 D1D1 D1D1 D1D1 How long does the subsequent access take?~10ns (~million x faster)

4. Making disk accesses tolerable Basic idea Remove disk accesses from critical path Transform disk latencies into memory latencies Requires predicting the future Pre-fetching Guess future accesses based on past accesses Process D1D1 D1D1 D2D2 D2D2 Operating system Get D1 Get D1, D2 D1D1 D1D1 D1D1 D1D1 D2D2 D2D2

5. Making disk accesses tolerable Basic idea Remove disk accesses from critical path Transform disk latencies into memory latencies Requires predicting the future Pre-fetching Guess future accesses based on past accesses Process D1D1 D1D1 D2D2 D2D2 Operating system Get D2 D1D1 D1D1 D2D2 D2D2 D1D1 D1D1 D2D2 D2D2

6. Hiding disk latency Caching works when The recent past looks like the near future Amortize initial retrieval penalty over many other accesses Pre-fetching makes sense when it is Accurate: anticipate accesses before they occur Cheap: pre-fetching must not hinder other activities Metrics for better pre-fetching More accurate predictions Reasonable cost to obtain and act on those predictions

7. Approaches to pre-fetching Usually rely on heuristics Doesn’t require apps to be modified Sequential read-ahead or history-based However, heuristics Can be very bad for many classes of apps E.g., databases and small files May require extra resources to maintain history Or let the programmer issue hints. Positives and negatives? Programmer knows best (better accuracy) Must rewrite app, which could involve significant restructuring Source code may not be available

8. Speculative execution “Dynamic self-analysis” Allow app to run into the future on fake data Speculative execution reveals future requests Why is this approach appealing? Doesn’t require apps to be modified Hints not limited to specific access heuristics However … Speculation has to be correct despite fake data Speculation has to be lightweight

9. Spec. execution approach

10. Speculative execution Assign each process a speculative thread Run speculative thread when original stalls Speculative thread generates I/O hints Hints used by storage system to pre-fetch data In what address space should thread run? Needs access to process state But what shouldn’t spec. thread be able to do? Modify process state Should have no direct side-effects on original’s state

11. Speculative execution Ways that the spec. thread could cause side-effects Could modify data in address space Could modify data outside address space (e.g., files) Could generate signals (e.g., seg fault) How do would you prevent this? Use software-enforced copy-on-write Instrument loads/stores and point to copy versions But won’t instrumentation degrade performance? Make a copy of the program text Only instrument copy Original program can run un-instrumented

12. Speculative execution When speculation goes “off track” Speculative thread operates on fake data Fake data could affect program control flow Could cause speculative thread to execute differently than orig. Is there any way to prevent this? Not really If we knew content of real data would return it Have to detect off track speculation instead How should we detect off-track speculation? Maintain a log of hints Original thread checks if hints are correct If they differ, then speculation has gone off track

13. Speculative execution When speculative execution is effective Speculative thread can overlap with slow operation Speculative thread is much faster than slow operation Speculation rarely goes off track i.e., fake data must not meaningfully affect control flow i.e., result of slow operation is often known in advance Other slow operations besides disk reads? Distributed consistency protocols (two-phase commit) Synchronous meta-data writes Lots of interesting uses for speculative execution

14. Speculative execution Why are multi-threaded apps hard? Have to ensure that thread interleavings are same On a uni-processor possible most of the time On a multi-core machine much harder … Area of active research

15. Does it work? Runtime with spec + hintsRuntime with spec + no hints Hints are really useful Generating hints is cheap