1. Slide 1 LIRA: Linux Inter-process Race Analyzer Tipp Moseley Intel Corporation University of Colorado

2. Slide 2 Introduction 1.Motivation for Software Quality 2.Related work 3.Why Lira? 4.Race & Deadlock Detection Algorithms 5.Design 6.Pitfalls 7.Results

3. Slide 3 Motivation for Software Quality Various tools, vendors, targets –Many gaps in availability (X-Scale, IPF) Suite of tools adds value to underlying hardware Quality + manageability + security => buy more silicon Platform enabling –IBM’s Power, Motorola both have quality tools –These tools are fundamental for supporting platform

4. Slide 4 Related Work Memory tools –Purify –Valgrind –Etnus –Bitraker –3 rd Race/deadlock tools –Eraser –Ladybug –Intel Thread Checker –Multirace –Lira

5. Slide 5 What is Lira? –Linux Inter-process Race Analyzer –Dynamically detect race conditions for Memory read/write Generic resource (file, socket, etc) –Dynamically detect potential deadlock for Any combination of inter-process synchronization primatives –User must insert callbacks in locking code

6. Slide 6 Why Lira? Many enterprise systems depend on shared memory and concurrency No tool exists for debugging across processes Java w/ native threads XF86, Gnome, KDE Baan Samba Sybase Oracle MySQL PostgreSQL Apache SAP

7. Slide 7 Race Detection Eraser memory states Shamelessly borrowed from Savage, et al. SOSP 1997

8. Slide 8 Race Detection Eraser Lockset Algorithm Let locks held(t) be the set of locks held in any mode by thread t. Let write locks held(t) be the set of locks held in write mode by thread t. For each v, initialize C(v) to the set of all locks. On each read of v by thread t, set C(v) := C(v) ∩ locks held(t); if C(v) := { }, then issue a warning. On each write of v by thread t, set C(v) := C(v) ∩ write locks held(t); if C(v) == { }, then issue a warning. Also shamelessly borrowed from Savage, et al. SOSP 1997

9. Slide 9 Deadlock Detection Only checks full ordering of lock hierarchy –Does not recognize that a->b->c and a->c->b are both OK (though bad practice) Data structures: –For each lock, maintain before and after set Each time a lock l is acquired: before(l) = before(l)  locks-held If before(l)  after(l) != {} then ERROR for l2 in parents(l) do after(l2) = after(l2)  l If contains(before(l2), l) then ERROR

10. Slide 10 Design - Issues Must follow exec() and communicate via shared memory as well Different address spaces –Shared memory binds to different addresses –Files to different file descriptors No common synchronization api or model –OS Semaphores –flock(), fcntl() –lock; xchgb

11. Slide 11 Design – Overview

12. Slide 12 Design – Front End Initialize Pin –Instrument memory refs, system calls, and user locking callbacks e.g. LIRA_LockEx_HW(&my_lock) –Patch execve() system call with pin –t lira – children get Pin’d, also Unique feature to Lira –Other tools require user to modify scripts by hand –$ pin –t lira – make test

13. Slide 13 Design – Front End –Filter irrelevent information –Maintain information about shared memory, file descriptors Client needs + offset because effective address differs across address spaces Client needs entire file path instead of fd –Send shared memory refs, lock ops, other callbacks to log buffer

14. Slide 14 Design – Back End LiraClient: –Parse ASCII data from ShmLogReader –Maintain state tables for each shared memory address, file descriptor –Drive RaceAnalyzer –Report race conditions RaceAnalyzer: –Generic implementation of Eraser algorithm –Check lock ordering LockModel: –Generic representation of various locking primitives –LockModelSemaphore, LockModelHardware, etc

15. Slide 15 Design – IPC How do we communicate data from multiple processes to the client process? ShmLogWriter -> ShmLogReader –Maintain a synchronized log file, protected by shared memory lock –If file becomes to big, begin new file –Online client deletes files when done processing (data may take up gigabytes of space in minutes) –Offline client processes files after execution completes

16. Slide 16 Pitfalls Maintaining state of sem/shm/fds from syscalls was painful –Solution: cache information from /proc Offline processing can lead to enormous logs –Solution: Online processing and delete processed info

17. Slide 17 Pitfalls Inferring meaning of lock operations was faulty at best –Solution: Offer user callbacks to capture intended meaning of synchronization operations. Unacceptably slow –Solution: -O6, cache frequently used data, do some work at instrumentation time, inline frequent calls

18. Slide 18 Sample Program // INITIALIZATION int *shmem = getShmem(sizeof(int)); sharedlock_t lock0 = getSharedLock(); sharedlock_t lock1 = getSharedLock(); lock_init(&lock0); lock_init(&lock1); fork(); // make 2 processes

19. Slide 19 Sample Program int i = 0; while( i < 100000 ) { lock(&lock0); lock(&lock1); *shmem++; // ERROR: UNINITIALIZED READ! unlock(&lock1); unlock(&lock0); } // ERROR: wrong lock hierarchy – potential deadlock! lock(&lock1); lock(&lock0); unlock(&lock0); unlock(&lock1); *shmem++; // ERROR: no locks held! exit(0);

20. Slide 20 Results Uninitialized LOAD –WARNING: possible uninitialized LOAD for segment=/SYSV000004d3@0 offset=0 opsize=4 at pc=0x8049208 tid=0 pid=32576 srcfile=tests/locktest0.C srcline=42 No locks held for stdout –ERROR: no locks held for FWR to file=/dev/pts/3 at pc=0x420d18bc tid=0 pid=32584 srcfile=tests/locktest0.C srcline=40

21. Slide 21 Results Inconsistent locks held –ERROR: inconsistent locks held for LOAD to segment=/SYSV000004d3@0 offset=0 opsize=4 at pc=0x804953b tid=0 pid=32576 srcfile=tests/locktest0.C srcline=86 Bad lock hierarchy –ERROR: inconsistent lock order at pc=0x8048ebc tid=0 pid=32576 srcfile=tests/../LiraCallbacks.h srcline=64

22. Slide 22 Future Work Lira: –Further optimization Still at ~500x slowdown (improved from >100000x) Work with Pin team to only instrument shared memory segments –Code that does not touch shared memory not instrumented –Find some bugs! LIRA can find potential errors, user must verify –Lots of work to figure out if a LIRA report really is an error in a large program (i.e. PostgreSQL, Oracle) –Potential analysis integration with Intel Thread Checker

23. Slide 23 Questions?

24. Slide 24 References http://www- 2.cs.cmu.edu/afs/cs/academic/class/1574 0-f03/public/doc/atom-user.pdfhttp://www- 2.cs.cmu.edu/afs/cs/academic/class/1574 0-f03/public/doc/atom-user.pdf http://www.eecs.harvard.edu/~jonathan/pa pers/1997/eraser-sosp97.ps.gzhttp://www.eecs.harvard.edu/~jonathan/pa pers/1997/eraser-sosp97.ps.gz