November 27, 2007 Verification of a Concurrent Priority Queue Bart Verzijlenberg

November 27, Agenda Brief Review Brief Review Verification Overview Verification Overview Verification Results Verification Results Errors Errors Further Work Further Work Conclusion Conclusion

November 27, Brief Review Priority Queue Priority Queue Lock-free Lock-free Relies on atomic Compare and Swap operations Relies on atomic Compare and Swap operations

November 27, Algorithm Review The algorithm uses the Skip-List data structure The algorithm uses the Skip-List data structure Extends the Skip-List for concurrent use Extends the Skip-List for concurrent use The Skip-List is sorted on the priority of the nodes The Skip-List is sorted on the priority of the nodes The algorithm is lock-free The algorithm is lock-free No blocking No blocking Prevent dead locks Prevent dead locks Always progress by at least one operation Always progress by at least one operation Risk of starvation Risk of starvation

November 27, Algorithm Review Skip-List Multi-layer linked list Multi-layer linked list H forward pointers for a node of height h H forward pointers for a node of height h Each pointer i points to the next node with a height of at least i Each pointer i points to the next node with a height of at least i

November 27, Inserting in a Skip-List Inserting 17 in the list

November 27, Verification Overview Java Path Finder Java Path Finder Breadth-First Search Breadth-First Search Depth-First Search Depth-First Search Modified JPF script Modified JPF script Increased stack size to 1536Mb Increased stack size to 1536Mb Vary the number of inserts/deletions Vary the number of inserts/deletions At what point does JPF crash? At what point does JPF crash?

November 27, Test Classes Driver Class Driver Class Creates Creates N insertThreads N insertThreads M deleteThreads M deleteThreads Insert Thread Insert Thread Inserts a single number into the queue Inserts a single number into the queue Delete Thread Delete Thread Loops until a single number removed from queue Loops until a single number removed from queue Use of sleep important Use of sleep important

November 27, Code Modifications Maximum Level = 3 Maximum Level = 3 Probabilistic level selection replaced with a random level Probabilistic level selection replaced with a random level Probabilistic function calls random multiple times Probabilistic function calls random multiple times Too many additional paths for JPF Too many additional paths for JPF Removed back-off period Removed back-off period

November 27, A Note At the end of the code At the end of the code Print “End of Code” Print “End of Code” Marks each time JPF reaches the end of a potential execution path Marks each time JPF reaches the end of a potential execution path I consider each “End of Code” to correspond with a different thread interleaving (i.e. a path) I consider each “End of Code” to correspond with a different thread interleaving (i.e. a path)

November 27, Verification Results (1,2) = 1 Insert Thread, 2 Delete Threads (1,2) = 1 Insert Thread, 2 Delete Threads 1,01,12,02,12,23,0 BFS 2 Paths No Error <1 Min 5 Paths No Error 1 Min 8 Paths No Error 1 Min ? Paths Out of memory 27 Min ? Paths Out of Memory 35 Min ? Paths Out of memory 34 Min DFS 2 Paths No Error <1 Min 5 Paths No Error <1 Min 10 Paths No Error <1 Min 15+ Paths Out of Memory 2:15 Hours 6+ Paths Out of Memory 15:40 Hours 18+ Paths Out of Memory 5:20 Hours

November 27, By Comparison Implemented simple priority queue test Implemented simple priority queue test java.util.PriorityQueue java.util.PriorityQueue Not safe for concurrency Not safe for concurrency Synchronized insert statement using a lock on the queue Synchronized insert statement using a lock on the queue Able to test inserting 8 numbers quickly Able to test inserting 8 numbers quickly 3713 Paths 3713 Paths Tested in 45 Min Tested in 45 Min Inserting 10 numbers Inserting 10 numbers Crashed after Paths Crashed after Paths

November 27, What is the Difference? Using synchronized method Using synchronized method Very few points where a thread can take over Very few points where a thread can take over I.e. When one thread is inserting, none of the others can insert at the same time I.e. When one thread is inserting, none of the others can insert at the same time Lock Free Lock Free Threads can be interrupted after every instruction Threads can be interrupted after every instruction This results in many more potential interleavings This results in many more potential interleavings

November 27, Errors ! java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: GC overhead limit exceeded GC overhead limit exceeded More than 98% of time spent on GC More than 98% of time spent on GC Recovering less than 2% of the heap Recovering less than 2% of the heap java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space Java heap space [SEVERE] JPF out of memory [SEVERE] JPF out of memory [SEVERE] JPF exception, terminating: [SEVERE] JPF exception, terminating: class java.lang.NullPointerException: null class java.lang.NullPointerException: null

November 27, Further Work Reduce Max level to 2 Reduce Max level to 2 Further reducing number of paths Further reducing number of paths Enable back-off function Enable back-off function Find linearization points in the code Find linearization points in the code Add each number into two queues at the same time Add each number into two queues at the same time Try some of the other search classes Try some of the other search classes

November 27, Conclusion Testing at this point has not found errors Testing at this point has not found errors When inserting up to 2 numbers When inserting up to 2 numbers When inserting 3 numbers (before crashing) When inserting 3 numbers (before crashing) More work needed to reduce the chance of errors further More work needed to reduce the chance of errors further Will not eliminate completely Will not eliminate completely

November 27, 2007 Thank You Questions?