1 Testing Concurrent Programs Why Test?  Eliminate bugs?  Software Engineering vs Computer Science perspectives What properties are we testing for? 

Slides:



Advertisements
Similar presentations
Annoucements  Next labs 9 and 10 are paired for everyone. So don’t miss the lab.  There is a review session for the quiz on Monday, November 4, at 8:00.
Advertisements

Atomicity in Multi-Threaded Programs Prachi Tiwari University of California, Santa Cruz CMPS 203 Programming Languages, Fall 2004.
Software Engineering COMP 201
Review Amit Shabtay. March 3rd, 2004 Object Oriented Design Course 2 Review What have we done during the course? Where to learn more? What is for the.
Concurrency CS 510: Programming Languages David Walker.
JVM-1 Introduction to Java Virtual Machine. JVM-2 Outline Java Language, Java Virtual Machine and Java Platform Organization of Java Virtual Machine Garbage.
Microprocessors Introduction to ia64 Architecture Jan 31st, 2002 General Principles.
OOP #10: Correctness Fritz Henglein. Wrap-up: Types A type is a collection of objects with common behavior (operations and properties). (Abstract) types.
1 Sharing Objects – Ch. 3 Visibility What is the source of the issue? Volatile Dekker’s algorithm Publication and Escape Thread Confinement Immutability.
CprE 458/558: Real-Time Systems
Using JetBench to Evaluate the Efficiency of Multiprocessor Support for Parallel Processing HaiTao Mei and Andy Wellings Department of Computer Science.
Approaches to ---Testing Software Some of us “hope” that our software works as opposed to “ensuring” that our software works? Why? Just foolish Lazy Believe.
Proxy Design Pattern Source: Design Patterns – Elements of Reusable Object- Oriented Software; Gamma, et. al.
Introduction to High-Level Language Programming
CSE 486/586 CSE 486/586 Distributed Systems PA Best Practices Steve Ko Computer Sciences and Engineering University at Buffalo.
Lecture 4: Parallel Programming Models. Parallel Programming Models Parallel Programming Models: Data parallelism / Task parallelism Explicit parallelism.
Objectives Understand the basic concepts and definitions relating to testing, like error, fault, failure, test case, test suite, test harness. Explore.
Advanced Operating Systems CIS 720 Lecture 1. Instructor Dr. Gurdip Singh – 234 Nichols Hall –
Parallel Programming Models Jihad El-Sana These slides are based on the book: Introduction to Parallel Computing, Blaise Barney, Lawrence Livermore National.
Lecture 10 : Introduction to Java Virtual Machine
1 Concurrent Languages – Part 1 COMP 640 Programming Languages.
Lecture 2 Foundations and Definitions Processes/Threads.
 2004 Deitel & Associates, Inc. All rights reserved. 1 Chapter 4 – Thread Concepts Outline 4.1 Introduction 4.2Definition of Thread 4.3Motivation for.
Java Threads. What is a Thread? A thread can be loosely defined as a separate stream of execution that takes place simultaneously with and independently.
Reference: Ian Sommerville, Chap 15  Systems which monitor and control their environment.  Sometimes associated with hardware devices ◦ Sensors: Collect.
COMP 111 Threads and concurrency Sept 28, Tufts University Computer Science2 Who is this guy? I am not Prof. Couch Obvious? Sam Guyer New assistant.
1 An Aspect-Oriented Implementation Method Sérgio Soares CIn – UFPE Orientador: Paulo Borba.
ICS 313: Programming Language Theory Chapter 13: Concurrency.
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University IWPSE 2003 Program.
Java Thread and Memory Model
CISC Machine Learning for Solving Systems Problems Presented by: Suman Chander B Dept of Computer & Information Sciences University of Delaware Automatic.
Department of Computer Science and Software Engineering
Monitors and Blocking Synchronization Dalia Cohn Alperovich Based on “The Art of Multiprocessor Programming” by Herlihy & Shavit, chapter 8.
Concurrency Properties. Correctness In sequential programs, rerunning a program with the same input will always give the same result, so it makes sense.
CSCI1600: Embedded and Real Time Software Lecture 28: Verification I Steven Reiss, Fall 2015.
CS510 Concurrent Systems Jonathan Walpole. RCU Usage in Linux.
13-1 Chapter 13 Concurrency Topics Introduction Introduction to Subprogram-Level Concurrency Semaphores Monitors Message Passing Java Threads C# Threads.
Algorithm Analysis (Big O)
C H A P T E R E L E V E N Concurrent Programming Programming Languages – Principles and Paradigms by Allen Tucker, Robert Noonan.
PROGRAMMING TESTING B MODULE 2: SOFTWARE SYSTEMS 22 NOVEMBER 2013.
The single most important skill for a computer programmer is problem solving Problem solving means the ability to formulate problems, think creatively.
Week 9, Class 3: Java’s Happens-Before Memory Model (Slides used and skipped in class) SE-2811 Slide design: Dr. Mark L. Hornick Content: Dr. Hornick Errors:
CS 153 Design of Operating Systems Winter 2016 Lecture 7: Synchronization.
1 Critical Section Problem CIS 450 Winter 2003 Professor Jinhua Guo.
Agenda  Quick Review  Finish Introduction  Java Threads.
Concurrency in Java MD. ANISUR RAHMAN. slide 2 Concurrency  Multiprogramming  Single processor runs several programs at the same time  Each program.
CSC Multiprocessor Programming, Spring, 2012 Chapter 12 – Testing Concurrent Programs Dr. Dale E. Parson, week 12.
CSE 486/586 CSE 486/586 Distributed Systems PA Best Practices Steve Ko Computer Sciences and Engineering University at Buffalo.
Lecture 5 Page 1 CS 111 Summer 2013 Bounded Buffers A higher level abstraction than shared domains or simple messages But not quite as high level as RPC.
Chapter 4 – Thread Concepts
Healing Data Races On-The-Fly
Advanced Operating Systems CIS 720
Module 30 (Unix/Linux Security Issues II)
Advanced Topics in Concurrency and Reactive Programming: Asynchronous Programming Majeed Kassis.
Chapter 4 – Thread Concepts
Multiple Writers and Races
New trends in parallel computing
Threads and Memory Models Hal Perkins Autumn 2011
Concurring Concurrently
Synchronization Lecture 23 – Fall 2017.
Security in Java Real or Decaf? cs205: engineering software
Threads and Memory Models Hal Perkins Autumn 2009
Shared Memory Programming
Background and Motivation
21 Threads.
CS510 Operating System Foundations
Foundations and Definitions
“The Little Book on Semaphores” Allen B. Downey
Testing & Security Dr. X.
Software Engineering and Architecture
Presentation transcript:

1 Testing Concurrent Programs Why Test?  Eliminate bugs?  Software Engineering vs Computer Science perspectives What properties are we testing for?  Safety properties – nothing bad happens No deadlock All expected results returned  Liveness properties – something good happens, eventually  Performance

2 Alternatives to testing “Proving programs correct”  Formalized, mathematical, characterization of what programs do Lacking: performance formalization; characterization of JIT compilation; characterization of GC; … Often unwieldy  What is “correct”? Specification Size of specification often exceeds size of code – what confidence do you have that the spec is correct?  Overall, usually unwieldy even for sequential programs Sometimes successful for carefully defined properties Takes lots of experience Usually applied in very-high-value settings: life-critical embedded systems; national security and banking cryptographic systems

3 Code Reviews Basic conformance to organizational coding standards  Embodies a belief that coding standards result in more correct code Way to involve experts – especially helpful for situations like concurrent code where language behavior is subtle – practice helps! Just more eyes Can be counterproductive if review turns into a clash of egos

4 Static Analysis Tools Get a computer to help with code review  Can keep in mind much more context than any human  Can be absolutely rigorous in checking For example: check that every instance of a method that accesses a field is either synchronized or not synchronized  Premise: programmers are consistent, mostly  Can check for indicators of a great many common mistakes

5 Findbugs for Java Checks for hundreds of different errors Actually checks the compiled code (class files) Ru: Invokes run on a thread (did you mean to start it instead?) (RU_INVOKE_RUN)  This method explicitly invokes run() on an object. In general, classes implement the Runnable interface because they are going to have their run() method invoked in a new thread, in which case Thread.start() is the right method to call. SP: Method spins on field (SP_SPIN_ON_FIELD)  This method spins in a loop which reads a field. The compiler may legally hoist the read out of the loop, turning the code into an infinite loop. The class should be changed so it uses proper synchronization (including wait and notify calls). Checks based on observed mistakes in Java code that compiles!

6 Aspect-oriented testing AOP is a method for systematically modifying source code at compile time or run time  E.g. Attach code to method entry and exit, exception handling, etc. based on names and types of methods Used for implementing “cross-cutting” concerns such as locking and logging – something that should be done the same way at multiple locations in a program Also handy for inserting testing code

7 Performance testing No good tools for predicting system performance (but algorithmic analysis can suggest relative asymptotic behaviors) Have to do performance testing  Will the code meet the performance spec?  May be highly dependent on machine architecture, exact input, etc.  Also used in tuning program choices: e.g. buffer and thread pool sizes

8 Kinds of performance Latency or responsiveness – average (units: ) Latency distribution – variance Throughput (units: ) Resource consumption

9 Gotchas in Java Performance Testing (General) Garbage collection  May take arbitrarily long and be triggered at arbitrary points  Turn it off or better, make tests big enough to require multiple GCs Dynamic compilation  Just-in-time (JIT) compiler  Invoked at arbitrary points – time taken affects measured time for program  Mixing interpreted and compiled execution is nonsensical – result depends on when compilation happened

10 Concurrency Performance Gotchas Beware “micro” benchmarks  Focus narrowly on performance of synchronization and contention primitives  Miss the big picture of overall application performance – better to have no contention than a fast contention primitive Ensuring that significant concurrency occurs in the run  Use a small MP (want more threads than processors)

11 Counterintuitive Concurrent Performance Examine Figure 12.1: best performance occurs for 1 thread!

12 Testing Safety Properties of Concurrent Programs Remember about interleavings – there are a lot of them; any may have a bug! Book’s suggestion: first test for sequential correctness then Create a lot of different interleavings  Use various numbers of threads  Start them all at once  Run them for a long time

13 Ways to ensure many different interleavings Use a barrier or latch to ensure that all threads start the test at the same time  Otherwise, scheduler may run them all sequentially, esp. if thread creation overhead is high relative to the test Use strategically placed Thread.yield() calls at points where a thread switch might be disadvantageous. Unfortunately, only a hint to the scheduler

14 Testing is expensive A comprehensive test framework may be larger and more costly than the program being tested, esp. for very high assurance needs One good idea: if a test ever finds a bug (or is developed in response to an observed bug) keep it in the test suite forever (so-called regression testing)