1. 1 An Approach to Formalization and Analysis of Message Passing Libraries Robert Palmer Intel Validation Research Labs, Hillsboro, OR (work done at the Univ of Utah as PhD student) Michael DeLisi (undergraduate; his first research paper) Ganesh Gopalakrishnan Robert M. Kirby School of Computing University of Utah Supported by: Microsoft HPC Institutes NSF CNS 0509379

2. 2 MPI is the de-facto standard for programming cluster machines Our focus: Eliminate Concurrency Bugs from HPC Programs ! An Inconvenient Truth: Bugs  More CO 2, Bad Numbers ! (BlueGene/L - Image courtesy of IBM / LLNL) (Image courtesy of Steve Parker, CSAFE, Utah) 

3. 3 So many ways to eliminate bugs … Our Contribution: A Formal Model of 50 (of the 300) MPI functions An execution environment from which to check simple “litmus tests” VisualStudio Integration with Microsoft Phoenix Compiler Front-end Has spawned other research (e.g. develop POR for MPI programs, and In-Situ model checker for MPI Programs) Formalization helped reveal omissions in standard Can potentially help designer understand today’s complex standards Recommended for future libraries (APIs)

4. 4 A Simple MPI /C Program /* Add-up integrals calculated by each process */ if (my_rank == 0) { total = integral; for (source = 0; source < p; source++) { MPI_Recv(&integral, 1, MPI_FLOAT,source, tag, MPI_COMM_WORLD, &status); total = total + integral; } } else { MPI_Send(&integral, 1, MPI_FLOAT, dest, tag, MPI_COMM_WORLD); } 6/11/2015

5. 5 Library Semantics Dictates Behavior e.g. mismatched send/recv causing deadlock /* Add-up integrals calculated by each process */ if (my_rank == 0) { total = integral; for (source = 0; source < p; source++) { MPI_Recv(&integral, 1, MPI_FLOAT,source, tag, MPI_COMM_WORLD, &status); total = total + integral; } } else { MPI_Send(&integral, 1, MPI_FLOAT, dest, tag, MPI_COMM_WORLD); } 6/11/2015 p1:to 0p2:to 0p3:to 0 p0:fr 0p0:fr 1p0:fr 2

6. 6 Challenges for SW Model Checking 6/11/2015 Build Debugging Tools that “understand” Library Semantics Perform Static Analysis and Model Reductions modulo Library Semantics! A new world-order where the embedding program serves as a ‘control scaffolding’ with the “action” being within library calls

7. 7 Library Semantics Modeling Approaches 6/11/2015 Natural Language Documents - They alone don’t suffice (obvious drawbacks) Formal Descriptions - Use standard notations, ideally executable

8. 8 Practitioners must be able to benefit… 6/11/2015 1.They must be able to gain deeper understanding of the library thru the spec 2.Must be able to submit “litmus tests” and see outcomes in familiar ways

9. 9 Retain the Level of Detail of Interest 6/11/2015

10. 10 Example: Challenge posed by a 5-line MPI program… 6/11/2015 p0: { Irecv(rcvbuf1, from p1); Irecv(rcvbuf2, from p1); … } p1: { sendbuf1 = 6; sendbuf2 = 7; Issend(sendbuf1, to p0); Isend (sendbuf2, to p0); … } In-order message delivery (rcvbuf1 == 6) Can access the buffers only after a later wait / test The second receive may complete before the first When Issend (synch.) is posted, all that is guaranteed is that Irecv(rcvbuf1,…) has been posted

11. 11 Our Contributions 6/11/2015 1.Formal Executable Spec of the point-to-point operations of MPI – written in TLA+ 2.Simple MPI / C programs are compiled into TLA+ models and linked with Formal Semantics – all under Microsoft VisualStudio 3.Errors in Litmus Tests generate error traces that can step the Visual-Studio debugger 4.Same Framework includes a customized MPI model checker and soon a Dynamic Execution-based Model Checker with DPOR

12. 12 One of our Litmus Tests

13. 13 Executable Formal Specification and MPIC Model Checker Integration into VS 6/11/2015 TLA+ MPI Library Model TLA+ MPI Library Model TLA+ Prog. Model MPIC Program Model Visual Studio 2005 Phoenix Compiler TLC Model Checker MPIC Model Checker Verification Environment MPIC IR

14. 14 MPI Formal Specification Organization 6/11/2015 MPI 1.1 API Point to Point Operations Collective Operations Requests Communicator Collective Context Group Constants

15. 15 The Histrionics of FV for HPC (1)

16. 16 The Histrionics of FV for HPC (2)

17. 17 Error-trace Visualization in VisualStudio

18. 18 Spec of MPI_Wait (Slide 1 of 2)

19. 19 Spec of MPI_Wait (Slide 2 of 2)

20. 20 Related Work (Formalization and tool integration) 1. Use of TLA+ (or similar notations) to write executable specs is nothing new 2. Use to model a subset of MPI is new 3. Integration with VS and VS-debugger (or similar tools) may help designers become comfortable with formal specs

21. 21 Related Work (MPI formalization) Siegel (VMCAI 2007) has proposed a Promela model for MPI Uses Promela constructions to mimic MPI behavior Uses the Promela / C interface Uses an elaborately hand-crafted state machine Is much faster, and rides on established technology The declarative reading (emphasizing “what”) is lost

22. 22 Control state machine used in Siegel (VMCAI 2007)

23. 23 Concluding Remarks (1 of 3) l Quote from Lynn Conway (quote in VLSI, paraphrased): “There are two realities that must be met – - the architecture of a microprocessor, and - the polygons of the layout. Everything in-between is a luxury to be availed depending on our resources.”

24. 24 Concluding Remarks (2 of 3) l In our world: - The executable formal spec: the “what” (architecture) - An In-Situ Dynamic Partial Order Reduction (ISP) model checker is the “how” (or “polygons”) (paper in EuroPVM / MPI 2007) - The “in-between” is a customized MPI model checker for some of its constructs (PADTAD 2007)

25. 25 Concluding Remarks (3 of 3) – Formal Spec of Concurrency Libraries is essential for the development of a whole range of FV tools – It helps programmers avoid misunderstandings about the library – It can help during the platform testing of Library Implementations (think about multicores and transactions used in future library implementations The Model Checking Community and the Formal Spec Community must work hand-in-hand in addressing the issues in tomorrow’s Parallel and Distributed Programs

26. 26 Partial Demo

27. 27 Questions ? The verification environment is downloadable from http://www.cs.utah.edu/formal_verification/mpic It is at an early stage of development

28. 28 Answers! 1. We are extending it to Collective Operations -lesson learned from de Supinski 2. We may perform Formal Testing of MPI Library Implementations based on the Formal Semantics 3. We plan to analyze mixed MPI / Threads 4. That is a very good question – let’s talk!