Ferad Zyulkyarov 1,2, Tim Harris 3, Osman S. Unsal 1, Adrián Cristal 1, Mateo Valero 1,2 1 BSC-Microsoft Research Centre 2 Universitat Politècnica de Catalunya 3 Microsoft Research Cambridge 15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming January 2010 – Bangalore Debugging Programs that use Atomic Blocks and Transactional Memory

2 Our Motivation It is difficult to debug transactional applications with existing debuggers Existing debuggers are not aware of atomic blocks and transactional memory –Lessons learnt from developing complex TM applications such as Atomic Quake[1] and Quake TM [2]. [1] Zyulkyarov et al. “Atomic Quake: Using Transactional Memory in an Interactive Multiplayer Game Server“, PPoPP'09 [2] Gajinov et al. “QuakeTM: Parallelizing a Complex Serial Application Using Transactional Memory “, ICS'09

Overview Debugging atomic blocks atomically Examining the TM state Managing the TM state at debug-time Conflict point discovery Design and implementation 3

Debugging at the Level of Atomic Blocks Debugger is extended with the semantics of atomic blocks –Atomicity – atomic blocks are treated as single instruction –Isolation – the user does not observe intermediate results of concurrently running transactions Hides the low level implementation details of atomic blocks (TM or lock inference) 4

Atomicity in Debugging Step over atomic blocks as if single instruction. Good for debugging sync errors at granularity of atomic blocks vs. individual statements inside the atomic blocks. 5 atomic { } atomic { } Current DebuggerTM Aware Debugger

Using Existing Debuggers 6 SV_LinkEdict 1 atomic { 2 InitializeBoundingBox(ent); 3 SV_FindTouchedLeafs(ent, sv.worldmodel->nodes); 4 node = FindNewLocation(ent, sv_areanode); 5 InsertLinkBefore(&ent->area, &node->trigger_edicts); 6 } // end atomic sv_areanode Want to move from A to B and take the keys. Compute the bounding box. Find the leafs the bounding box maps to. Find location B Another player took the keys before us. Change the location and take the keys. Conflict. Rollback. Re-execute. Conflict. Rollback. Re-execute.

Using TM Aware Debugger 7 SV_LinkEdict 1 atomic { 2 InitializeBoundingBox(ent); 3 SV_FindTouchedLeafs(ent, sv.worldmodel->nodes); 4 node = FindNewLocation(ent, sv_areanode); 5 InsertLinkBefore(&ent->area, &node->trigger_edicts); 6 } // end atomic sv_areanode Moved from A to B and took the keys.

Isolation in Debugging What if we want to debug wrong code within atomic block? –Put breakpoint inside atomic block. –Validate the transaction –Step within the transaction. The user does not observe intermediate results of concurrently running transactions –Switch transaction to irrevocable mode after validation. 8 atomic { }

Debugging Wrong Code Inside Atomic Blocks Why isolation is important? 9 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; } 1 56

Example Importance of Isolation 10 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; } InsertSorted(4)

Example Importance of Isolation 11 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; }

Example Importance of Isolation 12 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; } Another transaction changed the value to 3. The current TX will be doomed and operate on invalid data.

Example Importance of Isolation 13 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; } Confuse the user because of consuming speculative values and not detecting conflict.

Why Isolate Speculative State? 14 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; } Confuse the user because of consuming speculative values and not detecting conflict.

Why Isolate Speculative State? 15 InsertSorted(object value) { Node n = new Node(value); atomic { Node previous = FindPrevious(value); n.Next = previous.Next; previous.Next = n; } Conflict. Rollback. Re-execute. Conflict. Rollback. Re-execute.

Debugging atomic blocks atomically Examining the TM state Managing the TM state at debug-time Conflict point discovery Design and implementation 16

Debugging at the Level of Transactions Assumes that atomic blocks are implemented with transactional memory. Examine the internal state of the TM –Read/write set, re-executions, status TM specific watch points –Break when conflict happens –Filters Concurrent work with Herlihy and Lev [PACT’ 09]. 17

Querying the TM State 18 atomic { } Read Set 0x30FA00 0x30FE16 0x320B0A 0x4A11F8 Write Set 0x30FE16 0x3AEE0D 0x4A1124 Status: Active Re-executions: 1 Query TM State Priority: 1 Nesting: 1

TM Specific Watchpoints 19 atomic { } Conflict Information Conflicting Threads: T1, T2 Address: 0x84D2F0 Symbol: Readers: T1 Writers: T2 Break when conflict happens Filter: Break if Address = Thread = T2 Filter: Break if Address = Thread = T2 AND

Debugging atomic blocks atomically Examining the TM state Managing the TM state at debug-time Conflict point discovery Design and implementation 20

Managing Transactions at Debug-Time At the level of atomic blocks –Debug time atomic blocks –Splitting atomic blocks At the level of transactions –Changing the state of TM system (i.e. adding and removing entries from read/write set, change the status, abort) Analogous to the functionality of existing debuggers to change the CPU state 21

Debug Time Atomic Blocks Create and remove atomic blocks while debugging. Useful to investigate and patch synchronization errors such as data races, atomicity, and order violation. User marks the places that should execute atomically. 22

Example Debug Time Atomic Bblocks 23

Example Debug Time Atomic Bblocks 24 StartDebugAtomic EndDebugAtomic User marks the start and the end of the transactions

Data Race 25 Thread 1 atomic { local = counter; local++; counter = local; } Thread 2 atomic { local = counter; local++; } counter = local; StartDebugAtomic atomic { local = counter; local++; } counter = local; EndDebugAtomic

Split Atomic Block Useful to find unnecessarily large atomic blocks or searching for data races 26 atomic { s1; s2; } atomic { s1; s2; } atomic { s1; } atomic { s2; }

Debugging atomic blocks atomically Examining the TM state Managing the TM state at debug-time Conflict point discovery Design and implementation 27

Conflict Point Discovery 28 TM applications have unanticipated overheads –Problem raised by Pankratius [talk at ICSE’09] and Rossbach et al. [PPoPP’10] Tells the users at which source lines how many conflicts happen. –Useful to profile and optimize TM applications. More comprehensive than Reach Points Gajinov et al. [ICS’ 09]

Example Output File:Line#Conf.MethodLine Hashtable.cs:51152AddIf (_container[hashCode]… Hashtable.cs:4862Adduint hashCode = HashSdbm(… Hashtable.cs:535Add_container[hashCode] = n … Hashtable.cs:835Addwhile (entry != null) … ArrayList.cs:793Containsfor (int i = 0; i < count; i++ ) ArrayList.cs:521Addif (count == capacity – 1) … 29

Optimizing Genome 30

Debugging atomic blocks atomically Examining the TM state Managing the TM state at debug-time Conflict point discovery Design and implementation 31

Design 32 Debugger Process WinDbg TmDbgExt DbgEng Target Process Program TmTargetDbg StmLib Show Read set Implements debugger commands such as setting watchpoints, query and modify the STM. Wrapper around the STM to hide its implementation details. Function call on the target process.

Conclusion New principles and approaches for debugging TM applications –Debugging at the level of atomic blocks –Debugging at the level of transactions –Managing transactions at debug-time Conflict point discovery –Optimized Genome Generic and decoupled design 33

Край Slides available at

Backup Slides 35

Implementing Conflict Point Discovery 36 atomic { a = 5; } Addr: 01 : StartAtomic(); 02 : OpenObjForWrite(a); 03 : a = 5; 04 : CommitAtomic(); Obj. Addr. 0x40D8E2 … … Ret. AddrCounter 0x03152 …… …… Compiler Instrumentation Conflict DbgEng Addr = 0x03 Hashtable.cs:51 Translate to source line Return Addr. 0x03 … … Log the address of a for conflict detection. To know where conflict happens log the return address of OpenObjForRead.

Internal Breakpoint Debugger breakpoint used to implement –Atomicity –Watchpoints –Splitting atomic blocks –Debug time atomic blocks 37

Breakpoint-time Callback Distinguishes ordinary breakpoints from internal breakpoints Overrides the debugger behavior of suspending the target process May execute complementary actions 38

Example Use of Internal Breakpoint 39 atomic { } Internal Breakpoint -> StartAtomic() -> Commit() Stepping over atomic block as if a single instruction

Atomicity Violations 40 initially a = b = 0; Thread 1 Thread 2 1 atomic{ 2 a++; 3 } atomic { 4 a++; 5 atomic{ } 6 b--; 7 assert(a + b == 0); 8 }

Atomicity Violations 41 initially a = b = 0; Thread 1 Thread 2 StartDebugAomic 1 atomic{ 2 a++; 3 } atomic { 4 a++; 5 atomic{ } 6 b--; 7 assert(a + b == 0); 8} EndDebugAtomic

Extending the Scope of Atomic Blocks We can build over debug time transactions to allow users to extend the scope of existing compile time atomic blocks. 42 atomic { } atomic { } StartDebugTx EndDebugTx We wantHow we do

Shrinking the Scope The implementation of shrinking the scope of atomic blocks might be tricky for STMs because of the code instrumentation. 43 atomic { } atomic { } StartTx - Ignore StartDebugTx EndDebugTx EndTx - Ignore We wantHow we do

Debugging Wrong Code Inside Atomic Blocks? 44 SV_LinkEdict 1 atomic { 2 InitializeBoundingBox(ent); 3 SV_FindTouchedLeafs(ent, sv.worldmodel->nodes); 4 node = FindNewLocation(ent, sv_areanode); 5 InsertLinkBefore(&ent->area, &node->trigger_edicts); 6 } // end atomic InsertLinkBefore(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); }

Isolating Speculative State Insert(3) Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); } 3

Isolating Speculative State Insert(3) Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); } Another transaction changed the value to 2. The current TX will be doomed and operate on invalid data. Another transaction changed the value to 2. The current TX will be doomed and operate on invalid data.

Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); } Isolating Speculative State Rotate Right 5 Mislead the user because of consuming speculative values and not detecting conflict.

Isolating Speculative State Fix Colors Insert(object value) { Node n = new Node(value); atomic { AddNode(n); RightRotate(ParentOf(n)); FixColors(); } Conflict. Rollback. Re-execute. Conflict. Rollback. Re-execute.