/ 23 Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information Slides taken from Shin Hong’s MS Thesis Defense Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information Moonzoo Kim CS Dept. KAIST

/ 23 Approach Verification techniques Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 2 Model Checking Testing Lock-based static analysis Pattern- based analysis Precision Scalability Related works - Lock-based static analysis techniques : RacerX, RELAY  Lock discipline, Partial order among locks - Pattern-based bug detection : MetaL, FindBugs  Low precision (Too many false alarms)

/ 23 Goal Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 3 Linux file system code Reported bug survey Common bug patterns Concurrency bug detection technique Level of Abstraction Linux file system verification Concurrency Bug Classification Concurrency Bug Patterns Semantic Augmented Concurrency Bug Patterns Change Log & Code Review Change Log & Code Review Code-based automatic concurrency bug detection for large size system software Idea: Automatically detect common concurrency bugs using both syntactic and semantics code pattern matching

/ 23 Concurrency Bug Classification (1/3) We survey previous concurrency bugs from Linux file systems –Search Linux Change Log ~ –Keyword: concurrency, data race, deadlock, livelock, file system, ext, etc. –In almost 300 documents, we found 40 bug reports (patches) related to both file system and concurrency bugs. We construct concurrency bug classification to analyze the bug reports. –5 different aspects Symptom: –Data race (machine exception), Data race (Faulty state), Deadlock, Livelock. Fault : –Design decision violations, Incorrect use of synch. idioms, Program logic error Resolution: –{Insert, Remove, Change, Reorder} £ {Sync. operation, Data operation, Control operation} Related synchronization mechanism: –Instruction, Barrier, Thread operations, Conditional variable, Lock, Complex lock, Semaphore Synchronization granularity: –Kernel-level, File system-level, File-level, Inode-level –27 bugs are classified Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 4

/ 23 Example of Concurrency Bug Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 5 Bug08 (Reported in Linux Change Log /fs/ext3/balloc.c) - Symptom: Data race (Faulty state) - Fault: Incorrect use of synchronization idioms In ext3_discard_reservation(), it incorrectly use test and test-and-set idiom by missing the second testing. This may cause race condition, so that rsv_window_remove() can be invoked twice for the same object. - Resolution: Insert control operations - Related synchronization mechanism : Binary lock - Lock granularity: File system Example ext3_discard_reservation(inode * inode) { if (!rsv_is_empty(&rsv->rsv_window)) { spin_lock(rsv_lock) ; rsv_window_remove(inode->i_sb,rsv) ; spin_unlock(rsv_lock) ; } /* Linux kernel */ ext3_discard_reservation(inode * inode) { if (!rsv_is_empty(&rsv->rsv_window)) { spin_lock(rsv_lock) ; if (!rsv_is_empty(&rsv->rsv_window)) rsv_window_remove(inode->i_sb,rsv) ; spin_unlock(rsv_lock) ; } /* Linux kernel */

/ 23 Concurrency Bug Classification Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 6 Symptom \ Fault Design decision violation Incorrect use of synchronization idioms Program logic error Data race resulting machine exception Bug1,Bug5,Bug9,Bug13, Bug18,Bug20,Bug23 Bug16 Data race resulting faulty states Bug2,Bug7,Bug22, Bug25Bug8,Bug12,Bug26, Bug27Bug3,Bug15,Bug17, Bug19 DeadlockBug11,Bug24Bug10,Bug14,Bug21 LivelockBug4,Bug6 Sync. granularity \ Sync. primitives InstructionBarrier Thread operation Binary Lock Complex Lock Semaphore Kernel levelBug4Bug12 Bug1,Bug2,Bug3, Bug5,Bug10,Bug11 Bug19Bug6 File System levelBug17Bug8,Bug9,Bug22Bug14 File levelBug15, Bug16Bug20Bug13Bug7 Inode levelBug26 Bug18,Bug23,Bug25, Bug26,Bug27 Bug21,Bug24 13 bugs among 27 bugs in the concurrency bug classifications cannot be detected by conventional lock-based concurrency bug detection techniques.

/ 23 Concurrency Bug Patterns Based on the bug analysis result by the classification, we define the 10 concurrency bug patterns in order to detect unrevealed bugs automatically by code pattern matching Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 7 Race conditionDeadlockLivelock Barrier No Memory Barrier After Object Initializations Busy-waiting on atomic variable without memory barrier Instruction Use atomic instructions in Non- atomic ways Thread operation Unsynchronized Data Passing to Child Thread Waiting Already Finished Thread Conditional variable Waiting with Lock Held Lock Misused Test and Test-and-Set Unlock before I/O Operations Releasing and Re-taking Outer Lock Complex lock Unintended Big Kernel Lock Releasing

/ 23 Misused Test and Test-and-Set Bug Pattern Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 8 int data; /*shared data*/ int func(){ if (test(data)) { /*test*/ lock () ; if (test(data)) { /*test*/ data = newvalue ; } unlock () ; } /* Correct Code */ int data; /*shared data*/ int func(){ if (test(data)) { /*test*/ lock () ; data = newvalue ; } unlock () ; } /* Incorrect Code */ test(data) might be false

/ 23 Concurrency Bug Patterns (3/3) Bug detection result by pattern matching –Misused Test and Test and Set pattern: We analyze 9 file systems with VFS layer in Linux suspected bugs are revealed as false positive by manual review & discussion with maintainers Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 9 Ext2Ext3Ext4NFSReiserFSProcSysfsUDFBtrFSTotal # of suspected bugs # of files LOC 28K104K25K56K27K18K37K9K82K386K Time(sec) /* In Linux kernel /fs/ nfs/nfs4state.c */ void nfs_free_seqid(nfs_seqid *seqid) { if (!list_empty(&seqid->list)) { sequence = seqid->sequence->sequence ; spin_lock(&sequence->lock); list_del(&seqid->list) ; spin_unlock(&clp->cl_lock); rpc_wake_up(&sequence->wait) ; } - Ex. A suspected bug detected from NFS

/ 23 Semantics Augmented Pattern Matching We improve the bug pattern matching using semantic information to refine the bug detection results Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 10 Program behavior (reachable states) Safe states (no error) State space Bug pattern # 1 Bug pattern # 2 with semantics Bug pattern # 1 with semantics There are the following main sources of false positives: –No parallel thread to be scheduled –Synchronized by other locks –Shared variable initializations without holding locks Thread sensitive analysis Lock analysis Simple points-to analysis

/ 23 COBET Framework We build a COncurrency Bug pattErn maTching framework (COBET) to support programming template for effective bug pattern detector generation upon EDG C/C++ parser COBET: Pattern-driven Concurrency Bug Detection Framework 11 AST generator Path analysis Semantic analysis engine Alias analysis Config. Thread starting functions Lock spec. Target C src code Memory alloc. spec. Lock analysis Bug 1 description Bug pattern detector 1 COBET synthesizer Bug 2 description Bug pattern detector 2 Semantic condition checker Syntactic pattern matcher Phase I: Bug pattern description and pattern detector construction Phase II: Code pattern matching with semantic code checking

/ 23 COBET Synthesizer Pattern description language (PDL) Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 12 Grammar of COBET PDL 1a:pattern1 { 2a: fun $f1{ 3a: if $cond { 4a: lock $l; 5a: \{if $cond { }} 6a: unlock $l; 7a:}}} 1b:pattern 2 { 2b: fun $f2 { 3b: write $w; 4b: }} Example of incorrect test-test-set bug pattern

/ 23 Thread Sensitive Analysis (1/2) Most concurrency errors are interleaved executions of two or more threads. We extend each bug pattern to specify multiple disconnected code patterns which are necessary for bug occurrence. The extended bug pattern matching inputs thread starting functions (system call handlers), thread-spawning operation, and interrupt-handler registering operations to notice execution starting points Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 13 Code pattern match#1 Code pattern match #2 Error execution

/ 23 Thread Sensitive Analysis (2/2) Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 14 Code pattern #1 if (expr) { lock(m) ; /* no if(expr) here */... unlock(m) ; Code pattern #2 write(x) ; Ex. Extended Misused Test and Test-and-Set bug pattern proc_get_sb() {... ei = PROC_I(sb->s_root->d_inode); if (!ei->pid) { rcu_read_lock(); ei->pid=get_pid(find_pid_ns(1,ns)); rcu_read_unlock(); }... proc_alloc_inode() { … ei = kmem_cache_alloc(...) ; if (!ei) return NULL ; ei->pid = NULL ; proc_get_sb() { … if (!ei->pid) ei->pid = find_get_pid(1) ; Ex. A suspected bug of Misused Test and Test-and-Set from Proc file system Code pattern#2 Match-2 Code pattern#2 Match-1 Code pattern #1 Match-1 Condition: expr may read x

/ 23 Lock Analysis (1/3) Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 15 Code pattern match#1 Code pattern match #2 Lock analysis We applied lock analysis to compute the set of locks held when a code pattern match is executed. –Inter-procedural, flow-sensitive, path-insensitive analysis (similar to RacerX). –The specification of lock acquiring(releasing) operations are given by users. Lock(A) Lock(C) Lock(A) LS = {A} LS = {A,B} Conflict! No concurrent execution is possible due to lock A. Lock(B) Unlock(C)

/ 23 Lock Analysis (2/3) Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 16 proc_alloc_inode() { … ei = kmem_cache_alloc(… ; if (!ei) return NULL ; ei->pid = NULL ; proc_get_sb() { … if (!ei->pid) ei->pid = find_get_pid(1) ; Lockset: {lock_kernel} sys_mount() do_mount() do_new_mount() do_kern_mount() vfs_kern_mount() proc_get_sb() { … ei=PROC_I(sb->s_root->d_inode); if (!ei->pid) { rcu_read_lock(); ei->pid=get_pid(find_pid_ns(1,… rcu_read_unlock(); } … sys_mount() do_mount() do_new_mount() do_kern_mount() vfs_kern_mount() compat_sys_futimesat() __user_walk_fd() do_utimes() Lockset: {lock_kernel} Lockset: {inode.i_mutexl} acquire lock_kernel() acquire inode.i_mutex real_lookup() Code pattern # 1 Match-1 Code pattern # 2 Match-1 Code pattern # 2 Match-2 Conflict

/ 23 Lock Analysis (3/3) COBET uses an inter-procedural lock analysis –Transfer the lockset of a call site to the caller function analysis To avoid redundant inter-procedural lock analysis, COBET semantic engine uses a cache that record analysis result for functions Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 17

/ 23 Points-to Analysis Non-shared variables are free from concurrency errors. A newly allocated heap variable is non-shared until it is linked to a global data structure. These variables are initialized without any synchronization during non-shared period. This type code would result false positives in the bug pattern matching for the lack of alias- sensitive analysis Concurrency Bug Detection through Improved Pattern Matching Using Semantic Information 18 Match 2-2 0: proc_alloc_inode() { 1:... 2: ei = kmem_cache_alloc(... 3: if (!ei) return NULL ; 4: ei->pid = NULL ;... Match 1-1 0: proc_get_sb() { 1:... 2: ei = PROC_I(sb->s_root->d_inode); 3: if (!ei->pid) { 4: rcu_read_lock(); 5: ei->pid = get_pid(...; 6: rcu_read_unlock(); 7: } 8:... Non-shared: { ei } Non-shared: {} Non-shared: { ei } No other thread can access ei->pid.

/ 23 Experiment Result (1/3) We reviewed Linux Change Logs on file systems from 2.6 to and defined the four bug patterns based on actual bug history. Effectiveness –We applied the four bug pattern detectors to seven Linux file systems in kernel –We detected the four new bugs confirmed by Linux maintainers COBET: Pattern-driven Concurrency Bug Detection Framework 19

/ 23 Experiment Result (2/3) Efficiency –We measured the false alarm reduction rate through the semantic analyses and time cost –The false alarms are reduced as additional analysis techniques are employed, and the time costs were not burdensome (8.34 sec for analyzing 193KLOC) COBET: Pattern-driven Concurrency Bug Detection Framework 20

/ 23 Experiment Result (3/3) Applicability –We applied the four bug detectors to Linux device drivers and network modules COBET: Pattern-driven Concurrency Bug Detection Framework 21

/ 23 Conclusion COBET framework –Pattern-based concurrency bug detection framework –Able to define and match various bug patterns. –Utilizes composite patterns with semantic conditions to improve accuracy Empirical result –Defined four concurrency bug patterns with various synchronization mechanisms –Implement the four bug pattern detectors through COBET –Detect ten fresh and real bugs in Linux COBET: Pattern-driven Concurrency Bug Detection Framework 22