The C++11 Memory Model CDP Based on “C++ Concurrency In Action” by Anthony Williams, The C++11 Memory Model and GCC Wiki and Herb Sutter’s atomic<> Weapons talk. Created by Eran Gilad

Reminder: what is a memory model? The guarantees provided by the runtime environment to a multithreaded program, regarding the order of memory operations Each level of the environment might have a different memory model CPU, virtual machine, compiler The correctness of parallel algorithms depends on the memory model

Why C++ needs a memory model Isn’t the CPU memory model enough? Until now, different code was required for every compiler, OS and CPU combination Threads are now part of the language (C++11) Their behavior must be fully defined The same code should run the same on different environments regardless of the CPU (Intel, AMD, ARM, …) or the OS Having a standard guarantees portability and simplifies the programmer’s work Dependencies examples: Compiler – Different optimizations in general. VS provides special semantics to volatile on some cases. OS – Different threading facilities, Win32 and pthreads for instance. CPU – Pretty strict memory model on x86, weak on ARM. Having a standard basically passes the burden from the programmer to the compiler.

C++ in the past and now Given the following data race: Pre-2011 C++ Thread 1: x = 10; y = 20; Thread 2: if (y == 20) assert(x == 10); Pre-2011 C++ What’s a “Thread”? Namely – behavior is unspecified (not even undefined) C++11 Undefined Behavior (don’t try the above code at home) But now, C++11 introduces a new set of tools to get it right Pre-C++11 had no mention of threads in the standard. Everything was implementation-dependent. Undefined behavior is a clear statement about situations the programmer should never reach. It’s not an “I have no clue if it’s ok”. It’s a “don’t you dare!”.

Why Undefined Behavior matters Can the compiler convert the switch to a quick jump table, based on x’s value? Optimizations that are safe on sequential programs might not be safe on parallel ones So, races are forbidden! At this point, on some other thread: x = 8; if (x >= 0 && x < 3) { switch (x) { case 0: do0(); break; case 1: do1(); break; case 2: do2(); break; } Switch jumps to unknown location. Your monitor catches fire. The compiler knows x is bound between 0 and 2, so instead of creating a series of if’s, it can simply jump to (switch base address + x * constant). But if x is shared and unprotected, it might be changed between the if and the switch, causing an unrestricted jump, which can cause unpredictable damage… Races are forbidden because the compiler can’t always recognize them (static analysis in not required by the standard), and to avoid them the programmer must help the compiler. In the example above, even if x were somehow protected, it could still be modified between the if and the switch. This would have caused the code to behave differently from the serial implementation. However, had the compiler known x might be modified by another thread, it would not have used the unsafe jump table optimization.

Dangerous optimizations Optimizations are crucial for increasing performance, but might be dangerous Compiler optimizations: Reordering to hide load latencies Using registers to avoid repeating loads and stores CPU optimizations: Loose cache coherence protocols Out Of Order Execution Thumb rule: a thread must appear to execute serially to other threads Optimizations must not be visible to other threads Single-threaded programs are fairly easy to optimize, because there’s no “external viewer” (other than the output). So most of the time they execute “atomically”. Shared variables, however, might be viewed by different threads at (almost) any given moment. So, their modifications by one thread must seem to be ordered to other threads. The listed optimizations are just an example – obviously there are many more. Notice that in c++11 some optimizations can be seen by other threads, which is fully defined by c++11’s memory model

Memory layout definitions Every variable of a scalar (simple) type occupies one memory location Bit fields are different One memory location must not be affected by writes to an adjacent memory location! Can’t read and write all 4 bytes when changing just one struct s { char c[4]; int i: 3, j: 4; struct in { double d; } id; }; A memory location does not depend on the size of the type, and can be part of an array or struct/class. Bit fields are different – several might occupy a single memory location. About the example: writing single bytes can be inefficient on some systems. It could be faster to read the whole array of chars, for instance, as a single 32-bit word, update one element, and write the whole word back. But that might introduce a race on the other elements! This is similar to having two variables on the same cache line. Updating one must not override an update of the other. This requires non-trivial synchronization, and a performance overhead. This wasn’t a problem on single threaded programs. In Java, update of a variable smaller or equal to a dword should not affect an adjacent variable (like C++11). However, longs and doubles are not guaranteed to be atomic (unless defined as volatile, in which case they are always atomic). This is due to the fact that it is a multi-step operation at the machine code level. Note: int i:3 means that the compiler will use 3 bits to store i. Same memory location Considered one memory locaction even if hardware has no 64-bit atomic ops

Threads in C++11 (std::mutex) #include <iostream> #include <thread> void func(mutex * m, int * x) { m->lock(); std::cout << "Inside thread " << ++(*x) << std::endl; m->unlock(); } int main() { mutex m; int x = 0; std::thread th(func, &m, &x); // or std::ref(m), std::ref(x) th.join(); std::cout << "Outside thread " << x << std::endl; return 0; std::thread and std:mutex is used as we know it thread_local (like static, extern, auto, register, etc…) – define the variable to be local to each thread. The object is allocated when the thread begins and deallocated when the thread ends.

Threads in C++11 (std::atomic) std::atomic provides both atomicity and ordering guarantees Default order is sequential consistency Other orders are also possible The language offers specializations for basic types Basic types: bool, numbers, pointers Custom wrappers can be created by the programmer (i.e. template) A mutex is sometimes required to ensure atomic reads and writes Most architectures provide atomic primitives for up to a dword In case the variable is too large, a mutex must be used

Threads in C++11 (std::atomic definition) template<typename T> struct atomic { void store(T val, memory_order m = memory_order_seq_cst); T load(memory_order m = memory_order_seq_cst); // etc... }; // and on some specializations of atomic: T operator++(); T operator+=(T val); bool compare_exchange_strong(T& expected, T desired, memory_order m = memory_order_seq_cst);

Order Definitions: Sequenced Before The order imposed by the appearance of (most) statements in the code, for a single thread Thread 1 read x, 1 write y, 2 Sequenced Before On some occasions, the sequence of execution is not specified (such as function argument evaluation). This cases are therefore unordered with respect to each other. For instance, in f(h(), g()) it is unspecified whether h() or g() will be executed first. Yet, both h() and g() are always sequenced before f().

Order Definitions: Synchronized with The order imposed by an atomic read of a value that has been atomically written by some other thread Thread 1 write y, 2 Synchronized with Thread 2 read y, 2 The said synchronization does not rely on any synchronization object. It means that when E1 synchronizes with E2, every write done before E1 should be visible to reads done after E2; and no write done after E1 should be visible to reads done before E2. This defines an order that crosses thread boundaries.

Order Definitions: Inter-thread happens-before A combination of sequenced before and synchronized with Thread 1 read x, 1 write y, 2 Thread 2 read y, 2 write z, 3 Sequenced Before Synchronized With Sequenced Before Here, the events in thread 1 inter-thread happen-before the events in thread 2.

Order Definitions: Happens Before An event A happens-before an event B if: A inter-thread happens-before B or A is sequenced-before B Thread 1 (1) read x, 1 (2) write y, 2 Thread 2 (3) read y, 2 (4) write z, 3 Java has similar order definitions (using slightly different wording): happens before is determined by (intra-thread) program order and synchronized-with actions. Happens before Happens before Happens before

Using atomics Memory Order: Sequential Consistency Strongest ordering, default memory order Atomic writes are globally ordered Compiler and processor optimizations are limited No action can be reordered across an atomic action Non-atomic actions can be safely reordered as long as the reorder doesn’t cross an atomic command Atomic writes flush non-atomic writes that are sequenced-before them They are externally visible after the atomic action If the hardware doesn’t already guarantee sequential consistency, a barrier is required after every write. Non-atomic actions take place in the happen-before relation due to the sequenced-before relation, and the inter-thread happens before relation imposed by adjacent atomic operations. Example in next slide.

Using atomics Memory Order: Sequential Consistency (2) atomic<int> x; int y; // not atomic void thread1() { y = 1; x.store(2); } void thread2() { if (x.load() == 2) assert(y == 1); } The blue arrows are sequenced-before. The green arrow is synchronized-with. Together they form an inter-thread happens-before relation, which guarantees the write to y happens-before reading it. Note that y isn’t atomic, and in general not safe to use as shared variable. However, in this particular scenario, its accesses are synchronized by the use of the atomic x. The assert is guaranteed not to fail

This was the part you have to know in order to write correct code Now, if you like to play with sharp tools…

Using atomics Memory Order: Acquire Release Synchronized-with relation exists only between the releasing thread and the acquiring thread Other threads might see updates in a different order All writes before a release are visible after an acquire Even if they were relaxed or non-atomic Similar to release consistency memory model

Using atomics Memory Order: Acquire Release (2) #define rls memory_order_release /* save slide space… */ #define acq memory_order_acquire /* save slide space… */ atomic<int> x, y; void thread1() { y.store(20, rls); } void thread3() { assert(y.load(acq) == 20 && x.load(acq) == 0); } The writes are on different threads. Each acquiring thread is synchronized with the thread that has done the release. Thus, the releases are unordered. If sequential consistency had been used instead of acquire-release, the results would have been different. There would have been a total order of writes, so if one of the asserts had succeeded, the other must have failed. void thread4() { assert(y.load(acq) == 0 && x.load(acq) == 10); } void thread2() { x.store(10, rls); } Both asserts might succeed No order between writes to x and y

Using atomics Memory Order: Relaxed Consistency Happens-before now only exists between actions to the same variables Similar to cache coherence But can be used on systems with incoherent caches Imposes fewer limitations on the compiler Atomic actions can now be reordered if they access different memory locations Less synchronization is required on the hardware side as well In this model, two processors might see writes to two variables (done by some third processor) in a different order. This is where C++ atomic differs from Java volatile: Java never relaxes program order between volatile variables, while C++ allows reordering of (and across) atomic variables.

Using atomics Memory Order: Relaxed Consistency (2) #define rlx memory_order_relaxed /* save slide space… */ atomic<int> x, y; void thread3() { if (y.load(rlx) == 10) assert(x.load(rlx) == 10); } void thread1() { y.store(20, rlx); x.store(10, rlx); } R y, 10 W y, 20 R x, 0 W x, 10 void thread2() { if (x.load(rlx) == 10) { assert(y.load(rlx) == 20); y.store(10, rlx); } Notice: the sequenced before was removed. Both asserts might fail because the stores on thread 1 are not ordered, and different threads might see them in a different order. One way for this to happen is if thread 3 has a speculative load of x before the if. When this is the case, x might be read before threads 1 and 2 started to execute. Important: the rlx define is used just to reduce text in the presentation. There’s absolutely no reason to use this in real code! R x, 10 R y, 20 W y, 10 Both asserts might fail No order between operations on x and y

Using atomics Memory Order: Relaxed Consistency (3) Let’s look at the trace of a run in which the assert failed T1: W y, 20; W x, 10; T2: R x, 10; R y, 20; W y, 10; T3: R y, 10; R x, 0;

Using atomics Memory Order: Relaxed Consistency (4) Let’s look at the trace of a run in which the assert failed T1: W y, 20; W x, 10; T2: R x, 10; R y, 20; W y, 10; T3: R y, 10; R x, 0;

Using atomics Memory Order: Relaxed Consistency (5) Let’s look at the trace of a run in which the assert failed T1: W y, 20; W x, 10; T2: R x, 10; R y, 20; W y, 10; T3: R y, 10; R x, 0;

volatile in multi-threaded C++ Predates C++ multi-threading Serves different purpose Intended for reading from memory that was written by a device Does not guarantee atomic Unspecified memory order However Java (and C#) volatile is very similar to C++ atomic volatile was actually the only option in pre-C++11 code, but other than avoiding the use of a register it has no multi-threading semantics, because it was introduced way before C++ supported multi-threading. It does usually do the trick, but not due to some standard guarantee. Some compilers (VS) add ordering semantics (release consistency) on certain CPUs (Itanium) as an extension – but we’re talking about standard C++ here.

Summary and recommendations (1) Lock-free code is hard to write Unless speed is crucial, better use mutexes Sequential consistency is the default It is also the memory model for function that does not have memory-model as an input parameter Use the default sequential consistency and avoid data races. Really. Acquire-release is hard Relaxed is much harder Mixing orders is insane (but allowed)

Summary and recommendations (2) The memory model can be passed as run-time parameter But it is better to use it as a compilation constant to allow optimizations We only covered main orders C++11’s basic features are fully supported by all common compilers Current (end of 2012) compiler support status of the presented C++11 threading features: VS2010 – no VS2012 – yes Cygwin official distro (4.7) – no MinGW official distro (4.7) – no MinGW nuwen distro (4.7) – yes Linux GCC (4.7) – yes In 2016, most common compilers support atomic operations, thread-local storage and etc. as stated here: http://en.cppreference.com/w/cpp/compiler_support