1 Semantics Driven Dynamic Partial-order Reduction of MPI-based Parallel Programs Robert Palmer Intel Validation Research Labs, Hillsboro, OR (work done.

Slides:



Advertisements
Similar presentations
Parallel Jacobi Algorithm Steven Dong Applied Mathematics.
Advertisements

Software & Services Group PinPlay: A Framework for Deterministic Replay and Reproducible Analysis of Parallel Programs Harish Patil, Cristiano Pereira,
1 Chao Wang, Yu Yang*, Aarti Gupta, and Ganesh Gopalakrishnan* NEC Laboratories America, Princeton, NJ * University of Utah, Salt Lake City, UT Dynamic.
Message Passing: Formalization, Dynamic Verification Ganesh Gopalakrishnan School of Computing, University of Utah, Salt Lake City, UT 84112, USA based.
Abstraction and Modular Reasoning for the Verification of Software Corina Pasareanu NASA Ames Research Center.
Current Techniques in Language-based Security David Walker COS 597B With slides stolen from: Steve Zdancewic University of Pennsylvania.
Module 7: Advanced Development  GEM only slides here  Started on page 38 in SC09 version Module 77-0.
1 Symbolic Execution for Model Checking and Testing Corina Păsăreanu (Kestrel) Joint work with Sarfraz Khurshid (MIT) and Willem Visser (RIACS)
Model Checker In-The-Loop Flavio Lerda, Edmund M. Clarke Computer Science Department Jim Kapinski, Bruce H. Krogh Electrical & Computer Engineering MURI.
EFFICIENT DYNAMIC VERIFICATION ALGORITHMS FOR MPI APPLICATIONS Dissertation Defense Sarvani Vakkalanka Committee: Prof. Ganesh Gopalakrishnan (advisor),
Hydra (A General Framework for Formalizing UML with Formal Languages for Embedded Systems*) *from the Ph.D. thesis of William E. McUmber Software Engineering.
Background for “KISS: Keep It Simple and Sequential” cs264 Ras Bodik spring 2005.
Iterative Context Bounding for Systematic Testing of Multithreaded Programs Madan Musuvathi Shaz Qadeer Microsoft Research.
ISBN Chapter 3 Describing Syntax and Semantics.
The Path to Multi-core Tools Paul Petersen. Multi-coreToolsThePathTo 2 Outline Motivation Where are we now What is easy to do next What is missing.
An Automata-based Approach to Testing Properties in Event Traces H. Hallal, S. Boroday, A. Ulrich, A. Petrenko Sophia Antipolis, France, May 2003.
Scheduling Considerations for building Dynamic Verification Tools for MPI Sarvani Vakkalanka, Michael DeLisi Ganesh Gopalakrishnan, Robert M. Kirby School.
Argonne National Laboratory School of Computing and SCI Institute, University of Utah Formal Verification of Programs That Use MPI One-Sided Communication.
PTIDES: Programming Temporally Integrated Distributed Embedded Systems Yang Zhao, EECS, UC Berkeley Edward A. Lee, EECS, UC Berkeley Jie Liu, Microsoft.
1 An Approach to Formalization and Analysis of Message Passing Libraries Robert Palmer Intel Validation Research Labs, Hillsboro, OR (work done at the.
Ordering and Consistent Cuts Presented By Biswanath Panda.
1 Distributed Dynamic Partial Order Reduction based Verification of Threaded Software Yu Yang (PhD student; summer intern at CBL) Xiaofang Chen (PhD student;
Model Checking. Used in studying behaviors of reactive systems Typically involves three steps: Create a finite state model (FSM) of the system design.
A Proof of Correctness of a Processor Implementing Tomasulo’s Algorithm without a Reorder Buffer Ravi Hosabettu (Univ. of Utah) Ganesh Gopalakrishnan (Univ.
1 In-Situ Model Checking of MPI Parallel Programs Ganesh Gopalakrishnan Joint work with Salman Pervez, Michael DeLisi, Anh Vo, Sarvani Vakkalanka, Subodh.
ECE Synthesis & Verification1 ECE 667 Spring 2011 Synthesis and Verification of Digital Systems Verification Introduction.
Gauss: A Framework for Verifying Scientific Software Robert Palmer Steve Barrus, Yu Yang, Ganesh Gopalakrishnan, Robert M. Kirby University of Utah Supported.
1 MPI Verification Ganesh Gopalakrishnan and Robert M. Kirby Students Yu Yang, Sarvani Vakkalanka, Guodong Li, Subodh Sharma, Anh Vo, Michael DeLisi, Geof.
Partial Order Reduction for Scalable Testing of SystemC TLM Designs Sudipta Kundu, University of California, San Diego Malay Ganai, NEC Laboratories America.
Validating High-Level Synthesis Sudipta Kundu, Sorin Lerner, Rajesh Gupta Department of Computer Science and Engineering, University of California, San.
Argonne National Laboratory School of Computing and SCI Institute, University of Utah Practical Model-Checking Method For Verifying Correctness of MPI.
The Problem  Rigorous descriptions for widely used APIs essential  Informal documents / Experiments not a substitute Goals / Benefits  Define MPI rigorously.
1 Ivan Lanese Computer Science Department University of Bologna Italy Concurrent and located synchronizations in π-calculus.
Penn ESE535 Spring DeHon 1 ESE535: Electronic Design Automation Day 8: February 11, 2009 Dataflow.
1 In-Situ Model Checking of MPI Parallel Programs Ganesh Gopalakrishnan Joint work with Salman Pervez, Michael DeLisi Sarvani Vakkalanka, Subodh Sharma,
Utah Verifier Group Research Overview Robert Palmer.
Describing Syntax and Semantics
Formal verification Marco A. Peña Universitat Politècnica de Catalunya.
Introduction In the process of writing or optimizing High Performance Computing software, mostly using MPI these days, designers can inadvertently introduce.
The shift from sequential to parallel and distributed computing is of fundamental importance for the advancement of computing practices. Unfortunately,
Formal Techniques for Verification Using SystemC By Nasir Mahmood.
Cheng/Dillon-Software Engineering: Formal Methods Model Checking.
An Introduction to MBT  what, why and when 张 坚
Software Engineering Prof. Dr. Bertrand Meyer March 2007 – June 2007 Chair of Software Engineering Static program checking and verification Slides: Based.
Parallel and Distributed Computing in Model Checking Diana DUBU (UVT) Dana PETCU (IeAT, UVT)
Chapter 8 – Software Testing Lecture 1 1Chapter 8 Software testing The bearing of a child takes nine months, no matter how many women are assigned. Many.
Scientific Computing By: Fatima Hallak To: Dr. Guy Tel-Zur.
1 New Development Techniques: New Challenges for Verification and Validation Mats Heimdahl Critical Systems Research Group Department of Computer Science.
1 Hybrid-Formal Coverage Convergence Dan Benua Synopsys Verification Group January 18, 2010.
The Roadmap to Software Factories Tools, Patterns and Frameworks.
Formal Verification Lecture 9. Formal Verification Formal verification relies on Descriptions of the properties or requirements Descriptions of systems.
1 CSEP590 – Model Checking and Automated Verification Lecture outline for August 6, 2003.
The shift from sequential to parallel and distributed computing is of fundamental importance for the advancement of computing practices. Unfortunately,
HACNet Simulation-based Validation of Security Protocols Vinay Venkataraghavan Advisors: S.Nair, P.-M. Seidel HACNet Lab Computer Science and Engineering.
CS265: Dynamic Partial Order Reduction Koushik Sen UC Berkeley.
February 22-25, 2010 Designers Work Less with Quality Formal Equivalence Checking by Orly Cohen, Moran Gordon, Michael Lifshits, Alexander Nadel, and Vadim.
Bronis R. de Supinski and Jeffrey S. Vetter Center for Applied Scientific Computing August 15, 2000 Umpire: Making MPI Programs Safe.
CAPP: Change-Aware Preemption Prioritization Vilas Jagannath, Qingzhou Luo, Darko Marinov Sep 6 th 2011.
©Ian Sommerville 2006Software Engineering, 8th edition. Chapter 4 Slide 1 Software Processes.
Static Techniques for V&V. Hierarchy of V&V techniques Static Analysis V&V Dynamic Techniques Model Checking Simulation Symbolic Execution Testing Informal.
Software Systems Verification and Validation Laboratory Assignment 4 Model checking Assignment date: Lab 4 Delivery date: Lab 4, 5.
September 1999Compaq Computer CorporationSlide 1 of 16 Verification of cache-coherence protocols with TLA+ Homayoon Akhiani, Damien Doligez, Paul Harter,
Faithful mapping of model classes to mathematical structures Ádám Darvas ETH Zürich Switzerland Peter Müller Microsoft Research Redmond, WA, USA SAVCBS.
Symbolic Model Checking of Software Nishant Sinha with Edmund Clarke, Flavio Lerda, Michael Theobald Carnegie Mellon University.
The PLA Model: On the Combination of Product-Line Analyses 강태준.
Synthesis from scenarios and requirements
Alon Flaisher Alon Gluska Eli Singerman Intel Corporation
Model Checking for an Executable Subset of UML
Over-Approximating Boolean Programs with Unbounded Thread Creation
Foundations and Definitions
Presentation transcript:

1 Semantics Driven Dynamic Partial-order Reduction of MPI-based Parallel Programs Robert Palmer Intel Validation Research Labs, Hillsboro, OR (work done at the Univ of Utah as PhD student) Ganesh Gopalakrishnan Robert M. Kirby School of Computing University of Utah Supported by: Microsoft HPC Institutes NSF CNS

2 MPI is the de-facto standard for programming cluster machines Our focus: Eliminate Concurrency Bugs from HPC Programs ! An Inconvenient Truth: Bugs  More CO 2, Bad Numbers ! (BlueGene/L - Image courtesy of IBM / LLNL) (Image courtesy of Steve Parker, CSAFE, Utah) 

3 So many ways to eliminate MPI bugs … l Inspection – Difficult to carry out on MPI programs (low level notation) l Simulation Based – Run given program with manually selected inputs – Can give poor coverage in practice l Simulation with runtime heuristics to find bugs – Marmot: Timeout based deadlocks, random executions – Intel Trace Collector: Similar checks with data checking – TotalView: Better trace viewing – still no “model checking”(?) – We don’t know if any formal coverage metrics are offered l Model Checking Based – Being widely used in practice – Can provide superior debugging for reactive bugs – Has made considerable strides in abstraction (data, control)

4 “Ad-hoc Testing” “Model Checking” Our Core Technique: Model Checking Why model checking works in practice: * It applies Exhaustive Analysis, as opposed to Incomplete Analysis * It relies on Abstraction (both manual, and automated) Exhaustive analysis of a suitably abstracted system Incomplete analysis of an unabstracted system Exhaustive analysis of suitably abstracted systems helps catch more bugs than incomplete analysis of unabstracted systems [ Rushby, SRI International ]

5 Model Checking Approaches for MPI l MC Based On “Golden” Semantics of MPI Limited Subsets of MPI / C Translated to TLA+ (FMICS 2007) Limited C Front-End with Slicing using Microsoft Phoenix l Hand Modeling / Automated Verif. in Executable Lower Level Formal Notations Modeling / Verif in Promela (Siegel, Avrunin, et.al. – several papers) Non-Blocking MPI Operations in Promela + C (Siegel) Limited Modeling in LOTOS (Pierre et.al. – in the 90’s) l Modeling in MPI / C – Automatic Model Extraction Limited Conversion to Zing (Palmer et.al. – SoftMC 05) Limited Conversion to MPIC-IR (Palmer et.al. – FMICS 07) l Direct Model Checking of Promela / C programs Pervez et.al. using PMPI Instrumentation – EuroPVM / MPI Demo of One-Sided + a Few MPI Ops (Pervez etal, EuroPVM / MPI 07)

6 Model Checking Approaches for MPI 1. MC Based On “Golden” Semantics of MPI 1.Limited Subsets of MPI / C Translated to TLA+ (FMICS 2007) 2.Limited C Front-End with Slicing using Microsoft Phoenix 2. Hand Modeling / Automated Verif. in Executable Lower Level Formal Notations 1.Modeling / Verif in Promela (Siegel, Avrunin, et.al. – several papers) 2.Non-Blocking MPI Operations in Promela + C (Siegel) 3.Limited Modeling in LOTOS (Pierre et.al. – in the 90’s) 3. Modeling in MPI / C – Automatic Model Extraction 1.Limited Conversion to Zing (Palmer et.al. – SoftMC 05) 2.Limited Conversion to MPIC-IR (Palmer et.al. – FMICS 07) 4. Direct Model Checking of Promela / C programs 1.Pervez et.al. using PMPI Instrumentation – EuroPVM / MPI 2.Demo of One-Sided + a Few MPI Ops (Pervez etal, EuroPVM / MPI 07) THIS PAPER : Explain new DPOR Idea Underlying 3.2, 4.2

7 The Importance of Partial Order Reduction During Model Checking l With 3 processes, the size of an interleaved state space is p s =27 l Partial-order reduction explores representative sequences from each equivalence class l Delays the execution of independent transitions 5/3/2015

8 The Importance of Partial Order Reduction for Model Checking l With 3 processes, the size of an interleaved state space is p s =27 l Partial-order reduction explores representative sequences from each equivalence class l Delays the execution of independent transitions l In this example, it is possible to “get away” with 7 states (one interleaving) 5/3/2015

9 POR in the presence of FIFO Channels… 5/3/2015 S S R R S S R R l Can do S, R, S, R l Or S, S, R, R l Prefer to do SR, SR (diagonal) – This is what the “urgent” algorithm tries to do (Siegel)

10 Static POR Won’t Always Do (Flanagan and Godefroid, POPL 05) a[ j ]++ a[ k ]-- Action Dependence Determines COMMUTABILITY (POR theory is really detailed; it is more than commutability, but let’s pretend it is …) Depends on j == k, in this example Can be very difficult to determine statically Can determine dynamically

11 Similar Situation Arises with Wildcards… 5/3/2015 l Dependencies may not be fully known, JUST by looking at enabled actions l So Conservative Assumptions to be made (as in Urgent Algorithm) l If not, Dependencies may be Overlooked l The same problem exists with other “dynamic situations” – e.g. MPI_Cancel Send(to Q) Recv(from *) Send(to Q) Some Stmt Proc P:Proc Q:Proc R:

12 POR in the presence of Wildcards… 5/3/2015 l Illustration of a Missed Dependency that would have been detected, had Proc R been scheduled first… Send(to Q) Recv(from *) Send(to Q) Some Stmt Proc P:Proc Q:Proc R:

13 DPOR Exploits Knowledge of “Future” to Compute Dependencies More Accurately Ample determined using “local” criteria Current State Next move of Red process Nearest Dependent Transition Looking Back Add Red Process to “Backtrack Set” This builds the “Ample set” incrementally based on observed dependencies Blue is in “Done” set { BT }, { Done }

14 How to define “Dependence” for MPI ? l No a Priori Definition of when Actions Commute l MPI Offers MANY API Calls l So need SYSTEMATIC way to define “Dependence” l CONTRIBUTION OF THIS PAPER: – Define Formal Semantics of MPI – Define Commutability Based on Formal Semantics

15 Spec of MPI_Wait (Slide 1/2) – FMICS07

16 Spec of MPI_Wait (Slide 2/2)

17 MPI Formal Specification Organization 5/3/2015 MPI 1.1 API Point to Point Operations Collective Operations Requests Communicator Collective Context Group Constants

18 Example: Challenge posed by a 5-line MPI program… 5/3/2015 p0: { Irecv(rcvbuf1, from p1); Irecv(rcvbuf2, from p1); … } p1: { sendbuf1 = 6; sendbuf2 = 7; Issend(sendbuf1, to p0); Isend (sendbuf2, to p0); … } In-order message delivery (rcvbuf1 == 6) Can access the buffers only after a later wait / test The second receive may complete before the first When Issend (synch.) is posted, all that is guaranteed is that Irecv(rcvbuf1,…) has been posted

19 One of our Litmus Tests

20 The Histrionics of FV for HPC (1)

21 The Histrionics of FV for HPC (2)

22 Error-trace Visualization in VisualStudio

23 This paper: Simplified Semantics (e.g. as shown by MPI_Wait) 5/3/

24 Independence Theorems based on Formal Semantics of MPI Subset 1. Local actions (Assignment, Goto, Alloc, Assert) are independent of all transitions of other processes. 2. Barrier actions (Barrier_init, Barrier_wait) are independent of all transitions of other processes. 3. Issend and Irecv are independent of all transitions of other processes except Wait and Test. 4. Wait and Test are independent of all transitions of other processes except Issend and Irecv. 5/3/2015

25 Executable Formal Specification and MPIC Model Checker Integration into VS 5/3/2015 TLA+ MPI Library Model TLA+ MPI Library Model TLA+ Prog. Model MPIC Program Model Visual Studio 2005 Phoenix Compiler TLC Model Checker MPIC Model Checker Verification Environment MPIC IR FMICS 07 PADTAD 07

26 A Simple Example: e.g. mismatched send/recv causing deadlock /* Add-up integrals calculated by each process */ if (my_rank == 0) { total = integral; for (source = 0; source < p; source++) { MPI_Recv(&integral, 1, MPI_FLOAT,source, tag, MPI_COMM_WORLD, &status); total = total + integral; } } else { MPI_Send(&integral, 1, MPI_FLOAT, dest, tag, MPI_COMM_WORLD); } 5/3/2015 p1:to 0p2:to 0p3:to 0 p0:fr 0p0:fr 1p0:fr 2

27 Partial Demo of DPOR Tool for MPIC

28 So, the whole story (i.e. Conclusions)… l Preliminary Formal Semantics of MPI in Place (50 point-to-point functions) l Can Model-Check this Golden Semantics l About 5 of these 30 have a more rigorous characterization thru Independence Theorems l For MPI Programs using These MPI functions, we have a DPOR based model checker MPIC l Integrated in the VS Framework with MPI-TLC also l Theory Expected to Carry Over into In-Situ Dynamic Partial Order Reduction (model-check without model building – EuroPVM / MPI 2007)

29 Questions ? The verification environment is downloadable from It is at an early stage of development

30 Answers! 1. We are extending it to Collective Operations -lesson learned from de Supinski 2. We may perform Formal Testing of MPI Library Implementations based on the Formal Semantics 3. We plan to analyze mixed MPI / Threads 4. That is a very good question – let’s talk!