Performance Pathologies in Hardware Transactional Memory Jayaram Bobba, Kevin E. Moore, Haris Volos, Luke Yen, Mark D. Hill, Michael M. Swift, and David A. Wood Multifacet Project (www.cs.wisc.edu/multifacet) Computer Sciences Dept., University of Wisconsin Madison

Executive Summary (1/2) Multiple Hardware Transactional Memory Designs Best Performance ? Depends ! Livelock © 2007 Multifacet Project University of Wisconsin Madison

Executive Summary (2/2) Why? Performance Pathologies What to do? Better Conflict Resolution pa●thol●o●gy: any deviation from a normal, healthy or efficient condition Livelock Performance Pathologies Performance © 2007 Multifacet Project University of Wisconsin Madison

Talk Outline Motivation Base HTM Results Performance Pathologies Enhanced HTM Systems © 2007 Multifacet Project University of Wisconsin Madison

Motivation Which HTM system performs best? Different Assumptions Hardware Broadcast vs Directory Based Software: Continuous Transactions vs Critical Section Transactions How to compare? © 2007 Multifacet Project University of Wisconsin Madison

LL EE EL HTM Design Space Version Management Conflict Detection Lazy Eager Stanford TCC Illinois Bulk LL No HTMs (yet) Herlihy/Moss TM MIT LTM Intel/Brown VTM MIT UTM Wisconsin LogTM EL EE Conflict Resolution: Adhoc “Better By Design” ? © 2007 Multifacet Project University of Wisconsin Madison

LL EL EE Experimental Setup Base HTM Systems Common Hardware Platform 32 core CMP Directory-Based Coherence Common TM Programming Model Critical Section Based Transactions Common Workloads 5 SPLASH + 2 Microbenchmarks (Btree, LFUCache) LL EL EE © 2007 Multifacet Project University of Wisconsin Madison

Performance Pathologies And the winner is… Performance Pathologies Livelock Low Contention: Similar High Contention: Depends © 2007 Multifacet Project University of Wisconsin Madison

Talk Outline Motivation Base HTM Results Performance Pathologies Enhanced HTM Systems © 2007 Multifacet Project University of Wisconsin Madison

Performance Pathologies StarvingElder SerializedCommit RestartConvoy FriendlyFire DuelingUpgrades FutileStall StarvingWriter EE EL LL © 2007 Multifacet Project University of Wisconsin Madison

LL System Conflict Detection: Lazy Version Management: Lazy Detect conflicts at commit time (Validation) Version Management: Lazy New value elsewhere, Update on commit + Abort - Commit Conflict Resolution: Committer Wins compare with TCC, Bulk © 2007 Multifacet Project University of Wisconsin Madison

StarvingElder CAUSE Committer Wins Time EFFECT(S) Starvation Load Imbalance FIX Elder gets Priority ? T1 T2 T3 store A load A ABORT Time store A load A ABORT store A load A ABORT COMMITING STALLED ABORTING © 2007 Multifacet Project University of Wisconsin Madison

RestartConvoy CAUSE Fast Abort + Immediate Restart EFFECT(S) Resource Contention Serialized Execution ≈ Lock Convoys FIX Stagger Restarts ? T2 T3 T4 load A store A ABORT load A store A ABORT load A © 2007 Multifacet Project University of Wisconsin Madison

LL LL LL_B Pathologies Pathology (% ExecutionTime) Enhancement Linear Backoff on Abort Eliminate RestartConvoys StarvingElder Serialized Commit Restart Convoy SerializedCommit Raytrace 45 27 5.2 0.1 0.3 1.0 LL LL_B © 2007 Multifacet Project University of Wisconsin Madison

LL_B LL_B LL Enhanced LL ( ) Pathology (% ExecutionTime) Performance StarvingElder Serialized Commit Restart Convoy SerializedCommit Raytrace 45 27 5.2 0.1 0.3 1.0 LL LL_B © 2007 Multifacet Project University of Wisconsin Madison

Performance Pathologies StarvingElder SerializedCommit RestartConvoy FriendlyFire DuelingUpgrades FutileStall StarvingWriter LL EL EE © 2007 Multifacet Project University of Wisconsin Madison

EL System Conflict Detection: Eager Version Management: Lazy Detect conflicts immediately Version Management: Lazy New Value elsewhere, Update on commit + Abort - Commit Conflict Resolution: Requester Wins Exponential Backoff on Abort compare with HMTM, LTM © 2007 Multifacet Project University of Wisconsin Madison

FriendlyFire CAUSE Requester Wins EFFECT(S) Unnecessary Aborts load A store A ABORT CAUSE Requester Wins EFFECT(S) Unnecessary Aborts Livelock FIX Priority Based Conflict Resolution ? load A store A ABORT © 2007 Multifacet Project University of Wisconsin Madison

EL EL EL_T Pathologies Pathology (% ExecutionTime) Enhancement Timestamp Based Conflict Resolution Eliminate FriendlyFire Friendly Fire FriendlyFire Raytrace 73 0.2 EL EL_T © 2007 Multifacet Project University of Wisconsin Madison

EL_T EL EL_T Enhanced EL ( ) Pathology (% ExecutionTime) Performance Friendly Fire FriendlyFire Raytrace 73 0.2 EL EL_T © 2007 Multifacet Project University of Wisconsin Madison

Performance Pathologies StarvingElder SerializedCommit RestartConvoy FriendlyFire DuelingUpgrades FutileStall StarvingWriter LL EL EE © 2007 Multifacet Project University of Wisconsin Madison

EE System Conflict Detection: Eager Version Management: Eager Detect conflicts immediately Version Management: Eager New Value in-place, Restore on Abort - Abort + Commit Conflict Resolution Stall Requester Abort Requester on possible deadlock (Conservative Deadlock Avoidance) compare with LogTM © 2007 Multifacet Project University of Wisconsin Madison

DuelingUpgrades CAUSE Conservative Deadlock Avoidance EFFECT(S) load A store A T1 T2 STALL ABORT CAUSE Conservative Deadlock Avoidance EFFECT(S) Unnecessary Stall Unnecessary Abort FIX Acquire Store Permissions Early ? © 2007 Multifacet Project University of Wisconsin Madison

StarvingWriter CAUSE Stall Requester EFFECT(S) Starvation store A load A T1 T2 T3 T4 CAUSE Stall Requester EFFECT(S) Starvation Load Imbalance ≈ Reader/Writer Locks FIX Writer Priority ? © 2007 Multifacet Project University of Wisconsin Madison

EE EE_HP EE Pathologies Pathology (% ExecutionTime) Enhancements DuelingUpgrades: Store-Set Predictor StarvingWriter: Writer Aborts Readers immediately % Execution Time Aborts /Transaction FutileStall StarvingWriter Dueling Upgrades Futile Stall Raytrace 1.0 4.6 4.2 <0.1 0.3 EE EE_HP © 2007 Multifacet Project University of Wisconsin Madison

EE_HP EE EE_HP Enhanced EE ( ) Pathology (% ExecutionTime) Performance Aborts /Transaction FutileStall StarvingWriter Dueling Upgrades Futile Stall Raytrace 1.0 4.6 4.2 <0.1 0.3 EE EE_HP © 2007 Multifacet Project University of Wisconsin Madison

LL LL_B Caveats / FutureWork High Contention Scenarios Restart Convoy StarvingElder Serialized Commit RestartConvoy SerializedCommit Mp3d 21 36 30 9.0 28 25 High Contention Scenarios Better Conflict Resolution Unidentified Pathologies Better TM Workloads LL LL_B © 2007 Multifacet Project University of Wisconsin Madison

Executive Summary Performance Pathologies Enhanced HTM systems Livelock © 2007 Multifacet Project University of Wisconsin Madison

FutileStall DuelingUpgrades FriendlyFire RestartConvoy StarvingWriter StarvingElder SerializedCommit