Semantics of Minimally Synchronous Parallel ML Myrto Arapini, Frédéric Loulergue, Frédéric Gava and Frédéric Dabrowski LACL, Paris, France

F. GavaSNPD Outline Introduction Pure functional minimally synchronous parallel programming High level semantics Distributed evaluation Conclusions and future work

Introduction

F. GavaSNPD Our previous work: Bulk Synchronous Parallelism + Functional Programming = BSML Bulk Synchronous Parallelism : Scalability Portability Simple cost model Functional Programming : High level features (higher order functions, pattern matching, concrete types, etc.) Programs proofs Safety of the environment

F. GavaSNPD What is MSPML ? Why MSPML ? = Minimally Synchronous Parallel ML  BSML without synchronization barriers Comparison of efficiency BSML/MSPML Further extensions restricted or impossible in BSML (divide-and-conquer, nesting of parallel values) Basis for an additional level to BSML for Grid computing: BSML on each nodes (clusters) + MSPML for coordination

Pure Functional Minimally Synchronous Parallel Programming

F. GavaSNPD The MSPML Library Parallel ML library for the Objective Caml language operations on a parallel data structure Abtract type:  par access to the machine parameters: p: unit  int p() = number of processes

F. GavaSNPD Creation of Parallel Vectors mkpar: (int  )  par (mkpar f ) f (p-1)…(f 1)(f 0)

F. GavaSNPD Pointwise Parallel Application apply: (    ) par   par   par apply = f p-1 …f1f1 f0f0 v p-1 …v1v1 v0v0 f p-1 v p-1 …f 1 v 1 f 0 v 0

F. GavaSNPD Example (1) let replicate x = mkpar(fun pid->x) replicate: ’a -> ’a par (replicate 5) 5…55

F. GavaSNPD Example (2) let parfun f v = apply (replicate f) v parfun: (’a->’b)->’a par->’b par parfun (fun x->x+1) parallel_vector f v p-1 …v1v1 v0v0 f…ff apply = (f v p-1 )…(f v 1 )(f v 0 )

F. GavaSNPD Communication Operation: get get:  par  int par  par get v p-1 …v1v1 v0v0 i p-1 …i1i1 i 0 (=1) v i p-1 …vi1vi1 vi0vi0 =

F. GavaSNPD Example (3) let bcast_direct root parallel_vector = if not(within_bounds root) then raise Bcast_Error else get parallel_vector (replicate root) bcast_direct: int->’a par->’a par v p-1 …v1v1 v0v0 v root … ( bcast_direct root = )

F. GavaSNPD Global Conditional if parallel_vector at n then … else … if at n then E1 else E2 = E1 …truef p-1 …b1b1 b0b0 n

F. GavaSNPD Implementation of MSPML MSPML v 0.05 : F. Loulergue Library for Ocaml (uses threads and TCP/IP) October

High level semantics

F. GavaSNPD Terms Functional semantics: can be evaluated sequentially or in parallel Terms :

F. GavaSNPD Values and judgements Values: Judgement: e  v Term « e » evaluates to value « v »

F. GavaSNPD Some rules

Distributed evaluation

F. GavaSNPD Informal presentation (1) MSPML programs seen as SPMD programs The parallel machine runs p copies of the same MSPML program Rules for local reduction + rule for communication For example at processor i the expression (mkpar f) is evaluated to (f i)

F. GavaSNPD Informal presentation (2) Proc.012 empty 0,v’ Local computation Communication environment get v 1 0,v request 0 1v’ A BIT LATER 0,v’’

F. GavaSNPD Informal presentation (3) Proc.012 empty 0,v’ 0,v’’ 1,w’ 2,w’’ Local computation Communication environment 0,v 0,v’ 1,w request 2 0 Not Ready !!

F. GavaSNPD Terms and judgments New term: request i j the processor asks the value stored at processor j during the i th step Step = each step is ended by a call to get Judgment: At process i, the term e d with communication environment E c is evaluate to e’ d with new communication environment E’ c

F. GavaSNPD Local computation rules

F. GavaSNPD Communication rule

F. GavaSNPD Conclusion and Future Work Conclusion Minimally Synchronous Parallel ML: Functional semantics & Deadlock free Two semantics: High level semantics (programmer’s view) Distributed evaluation (implementation’s view) Implementation Future Work MPI Implementation Comparison with BSMLlib Extensions: parallel composition, etc.