1. Omega: flexible, scalable schedulers for large compute clusters
April 17th, 2013
Omega: flexible, scalable schedulers for large compute clusters
Malte Schwarzkopf (University of Cambridge Computer Lab)
Andy Konwinski (UC Berkeley)
Michael Abd-El-Malek (Google)
John Wilkes (Google)
Hello everyone!
I'll be talking about scheduling in Omega today. Omega is Google's next-generation cluster management system, and Andy and myself were privileged to make a small contribution to it while interning with Mike and John at Google. 1
EuroSys 2013

2. the scheduling problem
Job
Tasks
Machines
Now, Omega is a complex system with many aspects to it. Today, I'll focus on scheduling, that is, the problem of mapping work to resources. In a shared compute cluster, this means mapping tasks, which are shown as circles here, and that are part of a job, to machines.
The example job has two tasks here, but in reality, this varies between one and many thousands of tasks. 2
EuroSys 2013

3. Increasing cluster sizes
trends observed
Diverse workloads
Increasing cluster sizes
At Google, we have observed a few trends over the last years of running the old cluster management system. First of all, workloads are very diverse in shapes, sizes and requirements. This makes the cluster schedulers job a lot more difficult, as we'll see. <CLICK> At the same time, cluster sizes keep increasing, <CLICK> and so does the number of jobs arriving, as more and more applications are added.
Growing job arrival rates 3
EuroSys 2013

4. scheduling logic why is this a problem?
Arriving jobs and tasks (1,000s)
60+ seconds!
scheduling logic
Cluster scheduler
Cluster machines
(10,000s)
So how does the typical cluster scheduler work? Well, it tracks the state of tens of thousands of machines, shown as squares here. Many jobs from all kinds of different workloads arrive all the time, and proceed through scheduling logic in the cluster scheduler to be assigned to machines.<CLICK>
This scheduling logic could be the same for everyone, <CLICK>but is much more likely to become quite heterogeneous over time, with the scheduler adding special features for different job types.
For example, some important service jobs require careful placement for fault tolerance and optimal resource use. <CLICK> This can take a long time: for example, using constraint solvers or Monte Carlo simulations means that scheduling a job may take many seconds or even a minute. 4
EuroSys 2013

5. Increasing complexity!
why is this a problem?
Arriving jobs and tasks (1,000s)
Increasing complexity!
scheduling logic
Cluster scheduler
Cluster machines
(10,000s)
Hence:Break up into independent schedulers.
This feature creep, however, leads to an increasingly complex scheduler implementation. <CLICK> We've been through this with the current Google cluster scheduler. Over the years, many features, short-cuts and extensions were requested and implemented for specific workloads. This significantly increases the engineering and maintenance complexity for the scheduler.
There is a solution, though: we can break the monolithic scheduler up into modular ones. These are simpler to maintain and can have independent code bases and even development teams. But breaking up the scheduler means we have to somehow coordinate resource assignment, and decide who gets what!
But: How do we arbitrate resources between schedulers?
EuroSys 2013

6. existing approaches monolithic scheduler SCHEDULER static partitioning
hard to diversify
code growth
scalability bottleneck
static partitioning
poor utilization
inflexible S0 S1 S2
Now, there are various ways this can be done. The straightforward one is to simply break up the cluster too, and just have a scheduler for each subcluster. So, for example, the MapReduce scheduler runs on the MapReduce subcluster, while critical infrastructure services are scheduled and run in an infrastructure subcluster. That approach has some grave disadvantages, though: the partitioning is static, so one subcluster can be overloaded while others have space capacity. 6
EuroSys 2013

7. existing approaches two-level shared-state RESOURCE MANAGER
e.g. UCB Mesos [NSDI 2011]
hoarding
information hiding S0 S1 S2
RESOURCE MANAGER
two-level S0 S1 S2
CLUSTER STATE
shared-state
Clearly, a possible way to address this is by making the partitioning dynamic. This is what existing systems do, for example Mesos from Berkeley: they schedule resources on two levels, with a higher-level resource manager and a set of schedulers that each work within their resource allocations. We will later see that this has some problems, too.
In Omega, we look at a different way of doing things: every scheduler can lay claim to any resource at any point in time. The cluster state data structure is effectively shared between all schedulers. There is no need for any reservation or a-priori coordination between schedulers -- they just try, hope for the best and sort out any problems afterwards!
Let's see how this works in detail.... 7
EuroSys 2013

8. how does omega work? S0 S1 8 EuroSys 2013
Consider this example. Here, we have our shared cluster state in beige. This represents the ground truth, i.e. which tasks are actually assigned to which machines.
We have two schedulers, shown in red and blue here. Each scheduler contains its own local replica of the cluster state, which is frequently updated to reflect changes. Now, let's see what happens if jobs arrive at the red and blue scheduler... 8
EuroSys 2013

9. S0 S1 how does omega work? 9 EuroSys 2013
In the red scheduler, the tasks come in, and are considered for scheduling. Using red's specific scheduling logic, it decides on two machines to place these tasks on. Then, it sends a delta to the shared cluster state, asking for its scheduling change to be committed. 9
EuroSys 2013

10. S0 S1 how does omega work? EuroSys 2013 EuroSys 2013
Meanwhile, the blue scheduler has been busy, too: it considers two tasks, finds machines to place them on using its specialized scheduling logic, and sends a delta to shared state.
EuroSys 2013
EuroSys 2013

11. how does omega work? Conflict!
In the shared cluster state, the deltas are applied. However, in this case, both tried to place a task on the same machine! This leads to a conflict: two schedulers are trying to change the same machine concurrently, which could lead to over-commit of that machine.
Conflict! 11
EuroSys 2013

12. how does omega work? S0 S1 failure! success! 12 EuroSys 2013
Only one delta can be applied successfully -- in this case, the blue one makes it, and blue's tasks start running. The red delta, however, fails to apply, and a conflict indication is returned to the red scheduler, who may try again after updating its state replica.
Clearly, these conflicts lead to wasted work, and may be detrimental to scheduling performance. The Omega model's viability thus hinges on how often conflicts occur, and how well we can avoid them. 12
EuroSys 2013

13. 2) workload characterization 3) comparison of approaches
overview
1) intro & motivation
2) workload characterization
3) comparison of approaches
4) trace-based simulation
5) flexibility case study
After this brief explanation of how the Omega scheduling architecture works, I'll now use a set of practical case studies to investigate how viable the Omega model is and how it compares to the alternatives. Before I go into details, we need to take a look at the workload setup we'll be using, though. 13
EuroSys 2013

14. workload: batch/service split
We'll be looking at a simple two-way workload split into batch and service jobs initially. This is a real differentiation that we're doing in Omega, and it is also a challenging one:
Batch jobs that run for some time and finish, while Service jobs are those production jobs that require careful scheduling for resource optimality and which run for a very long time.
Let's have a look at how this breaks up the real-world cluster workload. 14
EuroSys 2013

15. workload: batch/service split
Cluster A
Medium size
Medium utilization
Cluster B
Large size
Medium utilization
Cluster C
Medium (12k mach.)
High utilization
Public trace
We had a look at three representative Google clusters, denoted A, B and C here. They differ a bit in size and utilization; importantly, cluster C is also the one for which Google released a public trace last year. We used the same time period as the public trace, so the workload is identical to the published one.
The bar chart shows, for each cluster, the relative shares batch jobs, which are in solid colour, and service jobs, which are the dotted portion.
Jobs/tasks: counts
CPU/RAM: resource seconds [i.e. resource job runtime in sec.] 15
EuroSys 2013

16. workload: batch/service split
Cluster A
Medium size
Medium utilization
Cluster B
Large size
Medium utilization
Cluster C
Medium size
High utilization
Public trace
TAKEAWAY
Most jobs are batch, but most resources are consumed by service jobs.
There's a very clear take-away here: by job and task count, almost all jobs are batch, but the majority of resource-time is devoted to service jobs.
Jobs/tasks: counts
CPU/RAM: resource seconds [i.e. resource job runtime in sec.] 16
EuroSys 2013

17. workload: batch/service split
Batch jobs
Service jobs
80th %ile runtime
12-20 min.
29 days
80th %ile inter-arrival time
Individual batch and service jobs have different properties, too. Batch jobs are much shorter than service ones, and they arrive a lot more often. This is why we want to handle them differently.
4-7 sec.
2-15 min. 17
EuroSys 2013

18. 2) workload characterization 3) comparison of approaches
overview
1) intro & motivation
2) workload characterization
3) comparison of approaches
4) trace-based simulation
5) flexibility case study
So, let's see how the Omega approach compares to the competition! 18
EuroSys 2013

19. methodology: simulation
simulation using
empirical workload parameters distributions
For this comparison, we wrote a simple simulator that can simulate monolithic schedulers, Mesos-style schedulers and Omega. It simplifies things a little: for example, it generates jobs based on empirical parameter distributions and uses a simple placement algorithm, but this lets us implement all of the architectures.
We are making the source code of this simulator available for others to use.
Code [soon to be] available: 19
EuroSys 2013

20. parameters Scheduler decision time
ttask: per-task
(usually 0.005s per task)
tjob: constant
(usually 0.1s per job)
One important parameter for our simulation is the scheduler decision time, that is, the time it takes the scheduling logic to make a decision for each job.
We model the decision time using a constant per-job element and an element that is linear in the number of tasks, so that larger jobs take longer to schedule. TODO: we have up to thousands, remember!
If not otherwise specified, we set t_job to 100ms and t_task to 5ms in all of our experiments. These values are extremely conservative worst-case bounds on how long Google's current scheduling logic runs for; in practice, it would usually be much faster, so we're making things deliberately difficult for Omega here.
n: num. tasks 20
EuroSys 2013

21. How do does the shared-state design compare with other architectures?
Experiment 1:
How do does the shared-state design compare with other architectures?
Experiment details:
all clusters, 7 simulated days
2 schedulers
varying Service scheduler
So, let's see how we do! 21
EuroSys 2013

22. monolithic, uniform decision time (single logic)
red => unscheduled jobs remained
time spent scheduling total time
scheduler busyness
blue => all jobs were scheduled
We start off with a monolithic scheduler that only has a single scheduling logic, which it applies to all jobs. We call this "single-path". On this diagram, we vary the per-job constant scheduling time and the per-task scheduling time on the X and Y axes, while the Z-axis shows the scheduler utilization.
This demands a bit of explanation: scheduler utilization indicates what fraction of time the scheduler is busy making decisions. Once the value approaches 1, the scheduler is overwhelmed and can't keep up with the workload. This is what causes the red indication of unscheduled jobs here.
In this case, we are varying the decision time for all jobs, as there is only a single scheduling logic. This is a useful baseline, but clearly not very realistic.
tjob for ALL jobs
ttask for ALL jobs 22
EuroSys 2013

23. monolithic, fast-path batch decision time
scheduler busyness
head-of-line
blocking
So let's consider something more realistic: a "multi-path" monolithic scheduler, which has a fast-path for batch jobs. We now vary the decision times for service jobs only on the X and Y axes.
Utilization drops by a lot, and we only see unscheduled jobs if we take 100s per job plus 1s for each task. That's clearly pushing it.
Now, an ideal Omega implementation without any conflicts would produce a similar-looking graph. We'll see -- let's first have a look at a Mesos-style two-level scheduler.
tjob for service jobs
ttask for service jobs 23
EuroSys 2013

24. Ooops... mesos v0.9 (of May 2012) scheduler busyness
Huh? <CLICK> That's not quite what we expected! The scheduler utilization is low, but yet there are unscheduled jobs in every case.
We were a little confused by this result at first, but it actually makes sense. To understand, we need to look at how Mesos works.
tjob for service jobs
ttask for service jobs 24
EuroSys 2013

25. mesos 1. Green receives offer of all available resources.
2. Blue's task finishes.
RESOURCE MANAGER
3. Blue receives tiny offer.
4. Blue cannot use it.
[repeat many times]
We have a Mesos resource manager, and three schedulers, which receive offers in turn. Let's see what happens.
<CLICK> First, S2 receives an offer. This offer contains all available cluster resources, as Mesos guarantees dominant resource fairness and offers all resources in turn. <CLICK> Now, while S2 is making its scheduling decisions -- which may take a long time -- a task of S1 finishes. At this point, the resource manager will make a tiny offer of the newly available resources to S1. <CLICK> However, S1 cannot make use of this offer, as is trying to schedule 3 tasks, and only one machine is available to it.
So it does not schedule anything, and the offer returns. It may be offered again immediately after -- <CLICK> this cycle can iterate many times while S2 is still making its decision.
<CLICK> Eventually, S2 finishes scheduling a job, and <CLICK> S1 now finally receives a suitably large offer. However, by this point it has already given up on the job it was trying to schedule, as it suffered too many retries, or too much time elapsed.
5. Green finishes scheduling.
6. Blue receives large offer.
By now, it has given up. 25
EuroSys 2013

26. omega, no optimizations
scheduler busyness
So here's Omega. Remember: our main goal was additional flexibility. But it doesn't perfom too badly. Just not as well as we like.
tjob for service jobs
ttask for service jobs 26
EuroSys 2013

27. omega, optimized scheduler busyness tjob for service jobs
So we did a few optimizations, and things got much better.
The utilization is low, indicating that the the service scheduler can easily keep up with the workload.
tjob for service jobs
ttask for service jobs 27
EuroSys 2013

28. omega, optimized
TAKEAWAY
The Omega shared-state model performs as well as a (complex) monolithic multi-path scheduler.
This is a nice result: despite having to do more work, checking for conflicts and re-trying if they occur, the scheduler utilization is not significantly worse than with a monolithic multi-path scheduler, as we can see by comparing the graphs at the bottom.
Monolithic
Mesos
Omega 28
EuroSys 2013

29. Does the shared-state design scale to many schedulers?
Experiment 2:
Does the shared-state design scale to many schedulers?
Experiment details:
cluster B, 7 simulated days
2 schedulers
varying job arrival rate and number of schedulers
Now, you might be thinking, well, two schedulers is not a very taxing setup, is it? We thought so, too, and decided to do an experiment where we vary the workload and the number of schedulers. 29
EuroSys 2013

30. scaling to many schedulers
In this experiment, we vary batch workload on the X-axis: "4x" means four times as many batch jobs arrive within the same time frame. We load-balance this batch workload across a variable number of batch schedulers using a simple hashing function.
As there are more jobs, there is more scheduling work to be done. Consequently, the scheduler utilization goes up towards the right. However, adding more schedulers ameliorates this: load-balancing across schedulers reduces the average utilization.
As there are more schedulers, there are also additional opportunities for conflicts to arise. The graceful scaling towards the right hand end suggests that this isn't happening, but we made sure to check... 30
EuroSys 2013

31. 2) workload characterization 3) comparison of approaches
overview
1) intro & motivation
2) workload characterization
3) comparison of approaches
4) trace-based simulation
5) flexibility case study
This is all good news for Omega, but we were wondering if this really works at Google scale, and with all the complexities inherent to real production systems. We thus also ran a high-fidelity simulation with real workload traces. 31
EuroSys 2013

32. lightweight simulator high-fidelity simulator
simulator comparison
lightweight simulator
high-fidelity simulator
machines
homogeneous
real-world
empirical distribution
workload trace
job parameters
constraints
not supported
supported
The high-fidelity simulator is much more complex than the simple simulator, and in addition to replaying real workloads, it supports heterogeneous machines, a complex resource model, constraints and uses the real Google production scheduling algorithm. As a result, it also takes much longer to run.
scheduling algorithm
Google algorithm
random first fit
runtime
fast (24h ≃ 5min)
slow (24h ≃ 2h) 32
EuroSys 2013

33. How much scheduler interference do we see with real Google workloads?
Experiment 3:
How much scheduler interference do we see with real Google workloads?
Experiment details:
cluster C, 29 days
2 schedulers, non-uniform decision time
varying Service scheduler
Let's see how things work out with a month-long trace of cluster C. 33
EuroSys 2013

34. scheduler busyness scheduler busyness 34 EuroSys 2013
overhead due to conflicts
scheduler busyness
Here, we vary the per-job decision time for service jobs on the X-axis, and again show scheduler utilization on the Y-axis. The batch scheduler's utilization is shown in blue, while the service scheduler's utilization is in red. We also show a dotted line that approximates what the ideal curve would look like if there were no conflicts.
<CLICK> As we can see, there is a huge overhead due to conflicts once we go beyond, say 30s of decision time per service job. Let's see exactly how much. 34
EuroSys 2013

35. Interference is higher for real-world settings.
scheduler busyness
TAKEAWAY
Interference is higher for real-world settings.
overhead due to conflicts
Here, we vary the per-job decision time for service jobs on the X-axis, and again show scheduler utilization on the Y-axis. The batch scheduler's utilization is shown in blue, while the service scheduler's utilization is in red. We also show a dotted line that approximates what the ideal curve would look like if there were no conflicts.
<CLICK> As we can see, there is a huge overhead due to conflicts once we go beyond, say 30s of decision time per service job. Let's see exactly how much. 35
EuroSys 2013

36. optimizations 1. Fine-grained conflict detection#
2. Incremental commits
So, all for nothing then? Not quite. We looked at this, and thought about some optimizations that we could make to the Omega approach in order to reduce the number of conflicts.
<CLICK>First, we moved from a straightforward conflict detection mechanism based on sequence numbers that would increment <CLICK> on task placement to <CLICK> fine-grained notion of conflicts, checking each time whether deltas feasibly commute, <CLICK> rather than just checking if the machine was touched. This should reduce the number of spurious conflicts detected.
<CLICK>Second, we stopped gang-scheduling and added support for incremental commits: <CLICK>when only some tasks in a large job experience a conflict, <CLICK> the others may start running, and <CLICK> only the conflicted ones are re-tried. This should reduce the size of the retry commit, and thus make it more likely to succeed.
1st
2nd 36
EuroSys 2013

37. How do the optimizations affect performance?
Experiment 4:
How do the optimizations affect performance?
Experiment details:
cluster C, 29 days
2 schedulers, non-uniform decision time
varying Service scheduler
How much impact do these changes have? 37
EuroSys 2013

38. impact on scheduler utilization
In this experiment, we compare the effect of the two optimizations on scheduler busyness. Fine grained conflict detection, shown in cyan, approximately halves the number of conflicts seen from the baseline in red. If we add incremental commit on top of this, the scheduler busyness is halved again, as shown by the purple line. The average number of conflicts per job is well below 1 per job, even for decision times over 100s, at this point.
In this graph, we compare the impact of the optimizations on scheduler utilization. As we would expect, the benefits from a reduced number of conflicts translate into lower scheduler utilization across the board. 38
EuroSys 2013

39. practical implications – scheduler utilization
scheduler busyness
overhead due to conflicts
What does this look like on our previous graph, which compared the observed scheduler utilization to the ideal no-conflict case? Here it is.
As we can see, the overhead due to conflicts is now relatively minor, and scales much more pleasantly than when it skyrocketed before. 39
EuroSys 2013

40. practical implications – scheduler utilization
TAKEAWAY
We can make simple improvements that significantly improve scalability.
overhead due to conflicts
What does this look like on our previous graph, which compared the observed scheduler utilization to the ideal no-conflict case? Here it is.
As we can see, the overhead due to conflicts is now relatively minor, and scales much more pleasantly than when it skyrocketed before. 40
EuroSys 2013

41. MapReduce scheduler with opportunistic extra resources
Case study
MapReduce scheduler with opportunistic extra resources
Finally, we would also like to know if there are any qualitative benefits of moving to Omega's shared-state architecture for independent schedulers.
To find out, we did a simple case study, in which we looked at a MapReduce scheduler that can introspect cluster state in order to opportunistically harness idle resources. 41
EuroSys 2013

42. Count of jobs with X workers
workers in MR jobs
200 11 5
1000 50 3
450
100
Count of jobs with X workers
8000
The motivation for this idea is that the number of workers per MapReduce job is a manually configured value at Google.
Number of workers [log10]
Snapshot over 29 days

43. case study: a MapReduce scheduler
60% of MapReduces
3-4x speedup!
Fraction of MR jobs with speedup < X
Very simple linear speedup model, speedup on x-axis, CDF on y-axis
Green = happy zone
60% of MRs benefit
3-4x speedup in the 50th percentile
tail maybe less realistic, >100x speedups probably result of naive linear model
Relative speedup [log10]
better
cluster C, 29 days 43
EuroSys 2013

44. case study: a MapReduce scheduler
TAKEAWAY
The Omega approach gives us the flexibility to easily support custom policies.
60% of MapReduces
Fraction of MR jobs with speedup < X
3-4x speedup!
Relative speedup [log10]
better
cluster C, 29 days 44
EuroSys 2013

45. Flexibility and scale require parallelism,
conclusion
TAKEAWAYS
Flexibility and scale require parallelism,
parallel scheduling works if you do it right, and
using shared state is the way to do it right!
So, let's finish up -- in this work, we have shown that, for real Google workloads, our needs for flexibility and scale require parallelism in the cluster scheduler.
Parallel scheduling with full state exposure to all schedulers is tricky, but works if you do it right.
The right way to do it is to use shared state and optimistic concurrency!
Thank you very much! 45
EuroSys 2013

46. BACKUP SLIDES
EuroSys 2013

47. Why might scheduling take 60 seconds?
scheduling policies
Why might scheduling take 60 seconds?
Large jobs (1,000s of tasks)
Optimization algorithms (constraint solving, bin packing)
Very picky jobs in a full cluster (preemption consequences)
Monte Carlo simulations (fault tolerance)
For this comparison, we wrote a simple simulator that can simulate monolithic schedulers, Mesos-style schedulers and Omega. It simplifies things a little: for example, it generates jobs based on empirical parameter distributions and uses a simple placement algorithm, but this lets us implement all of the architectures.
We are making the source code of this simulator available for others to use. 48
EuroSys 2013

48. methodology: simulation
Initial cluster state
empirical distribution
Experiment configuration
Workload
Batch
Event-driven simulator
Service
...
Cluster state
For this comparison, we wrote a simple simulator that can simulate monolithic schedulers, Mesos-style schedulers and Omega. It simplifies things a little: for example, it generates jobs based on empirical parameter distributions and uses a simple placement algorithm, but this lets us implement all of the architectures.
We are making the source code of this simulator available for others to use.
MapReduce
Code [soon to be] available: 19
EuroSys 2013

49. workload: job runtime distributions
Batch
Service
Fraction of jobs running for less than X
Let's drill into this a little more, and consider the job runtime distributions for the two types. This is a cumulative distribution, so this point indicates that about 25% of service jobs in cluster C ran for less than an hour, and while over 90% of batch jobs finished within an hour.
The service lines do not reach 1.0 at the right hand edge because some jobs ran for more than 29 days.
EuroSys 2013

50. workload: job runtime distributions
Batch
Service
TAKEAWAY
Service jobs, once scheduled, run for much longer than batch jobs do.
Fraction of jobs running for less than X
Again, the take-away is clear: service jobs run for much longer than batch jobs do, so we need to make good scheduling decisions for them!
EuroSys 2013

51. workload: inter-arrival time distributions
Service
Batch
Fraction of inter-arrival gaps less than X
Next, we look at the inter-arrival time for the two job types. This is the time that elapses between two neighbouring jobs of this type arriving. As we can see, the batch inter-arrival time is always much less than minute, while service jobs arrive fairly infrequently.
EuroSys 2013

52. workload: inter-arrival time distributions
Service
Batch
TAKEAWAY
Service jobs arrive much less frequently than batch jobs do.
Fraction of inter-arrival gaps less than X
This leads to the obvious take-away: service job turnover is much lower than batch turnover. This gives us sufficient time to make complex scheduling decisions for service jobs.
EuroSys 2013

53. the omega approach Shared state
Deltas against shared state
Easy to develop & maintain
Heterogeneous scheduling logic supported
CLUSTER STATE
Optimistic concurrency
No explicit coordination required
Post-hoc interference resolution
Scales well
EuroSys 2013

54. num. conflicts total num. transactions
conflict fraction
num. conflicts total num. transactions
This graph shows, on the same X-axis, the average number of conflicts per job before it successfully schedules.
Up to about 30s of per-job decision time, we have less than a single conflict per job, on average. Beyond 30s, however, the number of conflicts skyrockets, and we already see about 7 conflicts for each successful service job scheduling event at 100s.
EuroSys 2013

55. impact on conflict fraction
In this experiment, we compare the average number of conflicts per successfully scheduled job. Fine grained conflict detection, shown in purple, approximately halves the number of conflicts seen. If we add incremental commit on top of this, the per-job conflicts are halved again, and the average stays below 1 conflict per job, even for decision times over 100s.
EuroSys 2013

56. case study: a MapReduce scheduler
50% of MapReduces
Fraction of MR jobs with speedup < X
4.5x speedup
Relative speedup [log10]
cluster A, 29 days
EuroSys 2013

57. caveats, or when this won't work well
Possible problems...
aggressive, systematically adverse workloads or schedulers
small clusters with high overcommit
Now, while our experiments are generally very promising, we do believe that there are situations in which the Omega model will run into trouble.
deal with using out-of-band or post-facto enforcement mechanisms
EuroSys 2013