OPS-28: A New Spin on Some Old Latches Richard Banville Fellow

© 2008 Progress Software Corporation2 OPS-28: A New Spin on Some Old Latches Agenda  Monitoring  Latch usage  Performance

© 2008 Progress Software Corporation3 OPS-28: A New Spin on Some Old Latches Some Definitions First  What is a “latch” Synchronization mechanism Latches vs “locks”  What is a critical section Code access to shared resource Types of resources  How is a latch obtained test, test-and-set, spin loop –Performed on “shared” value Single threading access

© 2008 Progress Software Corporation4 OPS-28: A New Spin on Some Old Latches Interactions  Cache coherency  Multi CPU issues  False sharing  Avoiding cache line ping pong Size Alignment Increase # of latches Reduced latch usage – need & time Playing nicely with the hardware

© 2008 Progress Software Corporation5 OPS-28: A New Spin on Some Old Latches Latch Info VST access to latch information _Latch-Type Latch type _Latch-Name Name _Latch-Lock # times locked _Latch-Wait Nap count _Latch-HoldUser # of last holder _Latch-QholdUser # of last holder of queue latch for each _latch:

© 2008 Progress Software Corporation6 OPS-28: A New Spin on Some Old Latches _Latch-Type: MT_LT_QUEUE Queue Latch governor LatchSemaphore Multi level latch Wakeup order guaranteed -spin 0 forces all latches Test, test-and-set spin loop

© 2008 Progress Software Corporation7 OPS-28: A New Spin on Some Old Latches _Latch-Type: MT_LT_SPIN Queue Latch governor LatchSemaphore Multi level latch Wakeup order guaranteed -spin 0 forces all latches Spin Latch Latch Short Nap Single level latch Most performant “backoff” Test, test-and-set spin loop -nap to -napmax

© 2008 Progress Software Corporation8 OPS-28: A New Spin on Some Old Latches Latch Implementations Spin or queue latch acts as governor of “mux” latch table governor Resources protected Mux latch table (128) Multi level latch with one governor Associates critical section w/resource Underlying one to many relationship Turns 1 latch into 128 latches “Mux” latch request

© 2008 Progress Software Corporation9 OPS-28: A New Spin on Some Old Latches Latch Implementations Spin or queue latch acts as governor of object latch Resources protected Multi level latch with one governor Associates critical section w/resource Underlying one to one relationship Turns 1 latch into many latches governor Enhanced mux latch Object latch - type 1 request

© 2008 Progress Software Corporation10 OPS-28: A New Spin on Some Old Latches Latch Implementations Spin or queue latch maintained within the resource itself Resources protected Single level latch. No governor Associates critical section w/resource One to one relationship Greatly improves concurrency No single access funneling Statistics maintained as if 1 latch exists Object latch - type 2 request

© 2008 Progress Software Corporation11 OPS-28: A New Spin on Some Old Latches Latch Implementations Table of spin or queue latchs Resources protected Latch table Single level latch with no governor Associates critical section w/resource Underlying one to many relationship Latch families request Statistics typically make this look like one latch Modified “mux” latch mechanism

© 2008 Progress Software Corporation12 OPS-28: A New Spin on Some Old Latches Latch Holder  _Latch-QHolder Last owner of queue type latch Not last queued user  _Latch-Holder Last owner of spin latch Not honored for “true” object latches –Type 2 Who am I waiting on?

© 2008 Progress Software Corporation13 OPS-28: A New Spin on Some Old Latches Latch Counts  _Latch-Lock # times latch locked/acquired High numbers are OK  _Latch-Wait # times user had to wait (after spin) High numbers may not be good –…but is not wrong –Performance issue potential Balance with –spin value (see tuning) Activity statistics (or lack there of)

© 2008 Progress Software Corporation14 OPS-28: A New Spin on Some Old Latches Latch Names _Latch-Name  LKP  LKF  LHT  LHT2  LHT3  LHT4  SEQ  PWQ  AIW  BIW  BFP  BHT  BF1  BF2  BF3  BF4  BF5  BF6  BF7  CPQ  LRU  LR2  LR3  LR4  L27  L28  L30  L31  MTX  USR  OM  AIB  BIB  SCC  GST  TXT  TXQ

© 2008 Progress Software Corporation15 OPS-28: A New Spin on Some Old Latches Agenda  Monitoring  Latch usage  Performance

© 2008 Progress Software Corporation16 OPS-28: A New Spin on Some Old Latches Miscellaneous Latches  USR* Protects login (user) control structures  SCC* Protects schema locking operation  GST Protects shared memory allocation *Indicates queue latch

© 2008 Progress Software Corporation17 OPS-28: A New Spin on Some Old Latches Obsolete Latches  Removed AIW, LR2, LR3, LR4  Not in use BIW  Renamed & reserved for future use BF5 L27 BF6 L28 BF7 L29 BF8 L30 Latches removed or reserved

© 2008 Progress Software Corporation18 OPS-28: A New Spin on Some Old Latches Latches Protecting Database Update Actions

© 2008 Progress Software Corporation19 OPS-28: A New Spin on Some Old Latches Update Latches - MTX DB Update Action Record BI note Record AI note Perform Action DB Update Action Record BI note Record AI note Perform Action DB Update Action Record BI note Record AI note Perform Action Top of the food chain – busiest/longest update latch Protects txn allocation & bi/ai note order Quiet point maintenance Online backup startup Latches protecting recovery logging procedure High activity High nap rate MT X Not allowed User 1 User 2 User 3

© 2008 Progress Software Corporation20 OPS-28: A New Spin on Some Old Latches Rollback Processing BI Buffer Pool – Recording a change -bibufs 10 NF - a NF - b NF - c NF - d NF - e Modified Queue Free List 15 Current Input Buffer 9 Backout Buffer 12 Backout Buffer BI Current Output Buffer New Notes (Actions) Forward Processing

© 2008 Progress Software Corporation21 OPS-28: A New Spin on Some Old Latches BIB and AIB latch usage -bibufs 10 Free(a) Free(b) Free(c) Free(d) Free(e) Modified Queue Free List BI Current Output Buffer New Notes (Actions) Forward Processing Bi (ai) buffer life cycle B I W Very little contention until BIW active OLTP latch order: MTX, BIB, AIB Page writer latch order: BIB or AIB BI B User

© 2008 Progress Software Corporation22 OPS-28: A New Spin on Some Old Latches Latches – LRU & PWQ B least recently used (LRU) chain maintenance LRU MRU High activity High nap rate LRU

© 2008 Progress Software Corporation23 OPS-28: A New Spin on Some Old Latches Latches – LRU & PWQ B least recently used (LRU) chain maintenance LRU MRU Page writer queue (PWQ) myDB Page Writer Page Writer Dirty buffer Very little contention unless too many APWs LRU PWQ LRU User

© 2008 Progress Software Corporation24 OPS-28: A New Spin on Some Old Latches Latches – LRU & CPQ B least recently used (LRU) chain maintenance LRU MRU Page writer queue (PWQ) myDB Page Writer Page Writer Check point queue (CPQ) Check- point Check- point Very little contention CPQ

© 2008 Progress Software Corporation25 OPS-28: A New Spin on Some Old Latches Online backup queue - BFP Online “point in time” backup User 1 BI Area 6 Area 7Area 8 … Online Backup.bak Backup queue Modify request Very little contention Only used w/online backup Back it up and mark it BFP TXE MTX BFP

© 2008 Progress Software Corporation26 OPS-28: A New Spin on Some Old Latches Update Latches  TXT Protects transaction table modifications  TXQ Protects acquire/release of transaction end lock (TXE)  SEQ Protects sequence control structures  Moderate activity  Little contention Latches for write operation structures

© 2008 Progress Software Corporation27 OPS-28: A New Spin on Some Old Latches Latches Protecting Data Access Actions and Housekeeping

© 2008 Progress Software Corporation28 OPS-28: A New Spin on Some Old Latches Disk Storage Files Storage Areas Data Area Extent Table A Table BCust TblTable D Index A-1 Index B-1Name Idx Cust# IdxSRep IdxIndex D-1Index D-2 Logical Physical

© 2008 Progress Software Corporation29 OPS-28: A New Spin on Some Old Latches Disk Storage Files Storage Areas Schema AreaArea 7Area 8 Extent Table A Table BCust TblTable D Index A-1 Index B-1Name Idx Cust# IdxSRep IdxIndex D-1Index D-2 Logical Physical Languages Storage Engine

© 2008 Progress Software Corporation30 OPS-28: A New Spin on Some Old Latches Disk Storage Files Storage Areas and Object Mapping Schema AreaArea 7Area 8 Extent Table A Table B Cust Tbl Table D Index A-1 Index B-1 Name IdxCust# IdxSRep IdxIndex D-1Index D-2 Logical Physical Mapping Languages Storage Engine

© 2008 Progress Software Corporation31 OPS-28: A New Spin on Some Old Latches Object Cache – OM Latch Table A Table B Cust Tbl Table D Index A-1 Index B-1 Name IdxCust# IdxSRep IdxIndex D-1Index D-2 Logical Mapping Languages -omsize OM cache loaded on demand Very high activity High nap rate OM latch needed for paging maintenance Get _StorageObject record Schema OMOM

© 2008 Progress Software Corporation32 OPS-28: A New Spin on Some Old Latches Object Cache - OM Latch Table A Table B Cust Tbl Table D Index A-1 Index B-1 Name IdxCust# IdxSRep IdxIndex D-1Index D-2 Logical Mapping Languages OM cache loaded at startup No latching, no paging Zero contention! Primary Cache Little contention 2 nd cache for over flow & OM latch needed for paging maintenance Get _StorageObject record Schema Secondary Cache new objects OMOM Schema -omsize

© 2008 Progress Software Corporation33 OPS-28: A New Spin on Some Old Latches Buffer Pool Hash Table - BHT Hash Table (-hash) Shared memory block lookup List of (–B) buffer pool entries (unordered) Buffer pool location lookup single threaded User 1 User 2 User 3 User 4 Block ID BH T High activity, few naps Ptr to buffer

© 2008 Progress Software Corporation34 OPS-28: A New Spin on Some Old Latches Buffer Pool Hash Table - BHT Hash Table (-hash) BHT now latch family of 256 List of (–B) buffer pool entries (unordered) Buffer pool location lookup multi-threaded Block ID User 1 User 2 User 3 User 4 Block ID High activity, few naps Ptr to buffer BHT... BHT

© 2008 Progress Software Corporation35 OPS-28: A New Spin on Some Old Latches -B buffer pool entry info – BF[1-4] Database access latches List of (–B) buffer pool entries (unordered) Ptr to buffer Hash Table (-hash) BHT... BHT

© 2008 Progress Software Corporation36 OPS-28: A New Spin on Some Old Latches -B buffer pool entry info – BF[1-4] List of (–B) buffer pool entries (unordered) Buffer pool info structure supports Hash Table (-hash) Database access latches max of 4 concurrent threads. BF 1 BF 2 BF 3 BF 4 Latch family of 4 Ptr to buffer BHT... BHT

© 2008 Progress Software Corporation37 OPS-28: A New Spin on Some Old Latches -B buffer pool entry info – BF[1-4] List of (–B) buffer pool entries (unordered) Block ID User 1 User 2 User 3 User 4 Block ID BF latch family of 4 Object latch type 2 Ptr to buffer Hash Table (-hash) BHT... BHT

© 2008 Progress Software Corporation38 OPS-28: A New Spin on Some Old Latches -B buffer pool entry info – BF[1-4] Hash Table (-hash) List of (–B) buffer pool entries (unordered) Block ID User 1 User 2 User 3 User 4 Block ID -B info structure supports –B threads 4 latch slots used for statistics only Ptr to buffer BF latch family of 4 Object latch type 2 BHT... BHT

© 2008 Progress Software Corporation39 OPS-28: A New Spin on Some Old Latches Lock Table Latches (LKF, LKP & LHT) Free Chain (-L entries) Hash Chain Anchor Table -lkhash entries Acquire a lock LK F Plain latch Single threaded access High activity Low nap rate

© 2008 Progress Software Corporation40 OPS-28: A New Spin on Some Old Latches Lock Table Latches (LKF, LKP & LHT) Free Chain (-L entries) Hash Chain Anchor Table -lkhash entries Protected by LHT (LKT) Record locks Record “get” locks Table locks Purged record locks and LKP latches Type 1 object latch Single threaded latch access Multi threaded chain access LHT LKP LH T LK P High activity/waits Low activity/waits

© 2008 Progress Software Corporation41 OPS-28: A New Spin on Some Old Latches LHT1 LHT2 LHT3 LHT4 Lock Table Latches (LKF, LKP & LHT) Free Chain (-L entries) Hash Chain Anchor Table -lkhash entries Protected by LHT[1-4] Record locks Record “get” locks Type 1 object latch family Multi threaded latch access Multi threaded chain access - Added 4 governors Enhanced concurrency Low contention/waits LK P High activity, few waits

© 2008 Progress Software Corporation42 OPS-28: A New Spin on Some Old Latches Agenda  Monitoring  Latch usage  Performance

© 2008 Progress Software Corporation43 OPS-28: A New Spin on Some Old Latches Tuning parameters  -spin # retries before nap –0 or 1 or 10,000 or 6,000 * # cpus or … Can be changed online  -mux Use multiplexing/governor (1)  -nap Initial amount to nap (10 ms)  -napmax* Max amount to nap (5 sec)  -napinc and -napstep Both obsolete and ignored Nothing new here

© 2008 Progress Software Corporation44 OPS-28: A New Spin on Some Old Latches Foresight and insight  CPU usage -spin ~6,000 * # CPUs CPU thrashes, decrease –spin > 16 CPUs, avoid cache line ping pong Activity matters  Activity # latches acquired # latch waits Application throughput What to monitor

© 2008 Progress Software Corporation45 OPS-28: A New Spin on Some Old Latches Performance – 10.1C vs 10.1B House KeepingData Access

© 2008 Progress Software Corporation46 OPS-28: A New Spin on Some Old Latches Performance – 10.1C vs 10.1B HousekeepingData Access Latches/sec LRU 31%Buf 43%

© 2008 Progress Software Corporation47 OPS-28: A New Spin on Some Old Latches Performance – wait for it 4,294 (~9%) fewer latch waits/sec in 10.1c

© 2008 Progress Software Corporation48 OPS-28: A New Spin on Some Old Latches Performance – you got it! ~40% ~56% # Users

© 2008 Progress Software Corporation49 OPS-28: A New Spin on Some Old Latches In Summary  Better insight  We’ve done some things  You can too Tuning Move to 10.1C

© 2008 Progress Software Corporation50 OPS-28: A New Spin on Some Old Latches Questions ?

© 2008 Progress Software Corporation51 OPS-28: A New Spin on Some Old Latches Thank You

© 2008 Progress Software Corporation52 OPS-28: A New Spin on Some Old Latches