1. What is the Cost of Determinism?
Cedomir Segulja, Tarek S. Abdelrahman
University of Toronto

2. Source: [Youtube]
Source: [Intel]

3. Non-Determinism Same program + same input ≠ same output
This is bad for …
Testing
Too many interleaving to test
Debugging
Hard to debug when behavior is not repeatable
Selling
CAD tools users expect each run to produce the same circuit

4. Deterministic Schedulers
Determinism
Deterministic Schedulers
Maximum Slowdown
DMP [Devietti et al. 2009]
1.7x
Kendo [Olszewski et al. 2009]
1.6x
Grace [Berger et al. 2009]
3.6x
CoreDet [Bergan et al. 2010]
10x
Calvin [Hower et al. 2011]
RCDC [Devietti et al. 2011]
Dthreads [Liu et al. 2011] 4x
Conversion [Merrifield and Eriksson 2013] 5x
Parrot [Cui et al. 2013]
3.8x
RFDet [Lu et al. 2014]
2.6x
Is good, but costly
What is the fundamental cost of determinism?
What is this cost across various execution environments?
“Determinism in the field” 1 2
Source: [Bergan et al. 2011] and the respective papers
*Only to show that determinism comes at a cost, and not to be used for a direct comparison (different features, benchmarks, # threads, etc.)

5. What is Determinism?
Property that requires observing the same output whenever program runs with the same input
SyncOrder determinism [Lu and Scott 11]
Require the same program result and same order of synchronization
More flexible than internal determinism
Still greatly eases testing [Cui et al. 13]
We assume data-race-freedom
Determinism during debugging is needed
But the cost of determinism matters the most in production
All data races are bugs [Boehm 2008, S. Adve 2010, Marino et al. 2010, Lucia et al. 2010, …]
Data races in general do not help performance [Boehm 12]
External
SyncOrder
Internal

6. What is the impact of enforcing a fixed synchronization order on program execution time?

7. Schedule-Record-Replay Framework1 2
application
application
schedule
thread1
thread2
scheduler
replayer
serial
hybrid
round-robin
perturber
idle
small perturbations
architectures
dynamic-A
dynamic-S
NUMA
background processes
recorder
DVFS

8. Replayer
Force threads to wait only when absolutely necessary under the schedule
And do so with as little overhead as possible
Non-deterministic execution vs. Non-deterministic execution with the replayer’s overhead

9. Deterministic Schedulers
Schedules
Deterministic Schedulers
Schedule
Grace [Berger et al. 2009]
serial
Dthreads [Liu et al. 2011]
round-robin
Conversion [Merrifield and Eriksson 2013]
Parrot [Cui et al. 2013]
Kendo [Olszewski et al. 2009]
dynamic
RCDC [Devietti et al. 2011]
RFDet [Lu et al. 2014]
DMP [Devietti et al. 2009]
hybrid
CoreDet [Bergan et al. 2010]
Calvin [Hower et al. 2011]
When does a thread pass its turn?
At the end – serial
After each synchronization operation – round-robin
After each instruction/store – dynamic-A/dynamic-S
After N instructions – hybrid
N = 100,000
No “reduced serial mode”

10. Platform
8-core Xeon E5-2660
24 SPLASH-2 and PARSEC benchmarks, 8 threads
Deterministic slowdown
𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑖𝑠𝑡𝑖𝑐 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒 𝑛𝑜𝑛 𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑖𝑠𝑡𝑖𝑐 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒
Data races in general do not help performance [Boehm 12]
15 benchmarks had races, performance degradation in only 3
barnes (11%), radiosity (5%), raytrace_parsec (8%)

11. Benchmarks serial round-robin dynamic-S dynamic-A hybrid
splash
barnes
1.10
0.98
0.95
0.96
0.99
cholesky
3.39
2.39
1.07
1.05
fft
4.36
1.02
1.01
fmm
6.34
1.33
1.16
1.13
1.19
lu_cb
1.00
lu_ncb
ocean_cp
ocean_ncp
radiosity
7.58
3.04
1.09
1.08
2.67
radix
raytrace
7.72
2.93
1.88
volrend
6.12
1.91
1.67
water_nsquared
water_spatial
parsec
blackscholes
bodytrack
5.87
1.04
dedup
5.04
1.77
1.63
1.34
facesim
6.19
ferret
3.19
1.58
1.23
1.25
fluidanimate
1.81
0.97
7.26
1.52
1.06
streamcluster
swaptions
vips
7.61
5.27
1.31
average slowdown
3.61
1.60
1.17
maximum slowdown

12. the fundamental cost of determinism is small.
For this set of benchmarks and our platform, and implementation overhead set aside,
the fundamental cost of determinism is small.

13. What is the performance cost of insisting on the same schedule across different environments?

14. Schedule-Record-Perturb-Replay Framework1 2
application
application
schedule
thread1
thread2
scheduler
replayer
serial
hybrid
round-robin
perturber
idle
small perturbations
architectures
dynamic-A
dynamic-S
NUMA
background processes
recorder
DVFS

15. Perturber
Small perturbations (context switches, thread migrations, page faults)
Simulate first order effects by inserting small delays (μs and ms)
Background processes
Spawn additional threads and control their work to sleep ratio
Dynamic voltage and frequency scaling (DVFS)
Use Linux’s cpufreq system to explore different DVFS policies
Non-uniform memory access (NUMA)
Spread threads over two NUMA nodes
Asymmetric architectures
Use DVFS to create asymmetry [Shelepov et al. 2009]

16. Metric Deterministic slowdown
𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑖𝑠𝑡𝑖𝑐 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒 𝑛𝑜𝑛 𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑖𝑠𝑡𝑖𝑐 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒
Same conditions during both runs, for example
𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑖𝑠𝑡𝑖𝑐 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒 𝑤𝑖𝑡ℎ 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑 𝑝𝑟𝑜𝑐𝑒𝑠𝑠𝑒𝑠 𝑛𝑜𝑛 𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑖𝑠𝑡𝑖𝑐 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒 𝑤𝑖𝑡ℎ 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑 𝑝𝑟𝑜𝑐𝑒𝑠𝑠𝑒𝑠

17. Benchmarks Quiet Small perturbations Backgroud proc. DVFS NUMA
Asym. Arch.
balanced
unbalanced
auto
manual
4/4
1/7
splash
barnes
0.96
0.95
0.97
0.92
0.91
0.94
cholesky
1.05
1.06
1.25
1.02
1.08
1.03
1.09
fft
1.01
1.07
1.00
fmm
1.13
1.19
1.24
1.14
1.15
lu_cb
0.99
0.98
lu_ncb
ocean_cp
ocean_ncp
radiosity
1.94
1.11
1.46
1.71
radix
raytrace
1.92
1.44
1.69
volrend
1.38
1.55
water_nsquared
water_spatial
parsec
blackscholes
bodytrack
1.04
1.51
1.33
1.56
dedup
1.35
1.31
1.29
1.32
1.64
facesim
ferret
1.23
1.21
1.37
1.10
fluidanimate
1.77
1.39
1.63
streamcluster
swaptions
vips
1.43
1.53
avg. slowdown
1.17
max. slowdown

18. Insisting on the same schedule in the presence of skewed conditions
can slow down execution by a
factor of almost 2x.

19. Conclusions
Employed the schedule-record-replay framework to divorce implementation overhead from the fundamental cost of enforcing deterministic execution
Fundamental cost of determinism is small (4% on avg., 33 % max.)
There is room for lowering overheads in current deterministic systems
Measured this fundamental cost across a range of execution environments
The cost of raises to almost 2x when threads face skewed conditions
Do we need a more relaxed definition of determinism?
Quantified various sources of non-determinism
Deterministic logical clocks are not deterministic (not only due to the performance counters imperfections [Weaver et al. 2013])

20. Thank you!