1. Chapel: User-Defined Distributions Brad Chamberlain Cray Inc. CSEP 524 May 20, 2010

2. Chapel Distributions Distributions: “Recipes for parallel, distributed arrays” help the compiler map from the computation’s global view… …down to the fragmented, per-processor implementation = α · + = + = + = + = + MEMORY Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

3. Domain Distribution Domains may be distributed across locales var D: domain(2) dmapped Block(CompGrid, …) = …; A distribution defines… …ownership of the domain’s indices (and its arrays’ elements) …default work ownership for operations on the domains/arrays  e.g., forall loops or promoted operations …memory layout/representation of array elements/domain indices …implementation of operations on its domains and arrays  e.g., accessors, iterators, communication patterns, … D A B CompGrid L0L1L2L3 L4L5L6L7 Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

4. Domain Distributions  Any domain type may be distributed  Distributions do not affect program semantics only implementation details and therefore performance “ steve ” “ lee ” “ sung ” “ david ” “ jacob ” “ albert ” “ brad ” Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

5. Domain Distributions “ steve ” “ lee ” “ sung ” “ david ” “ jacob ” “ albert ” “ brad ”  Any domain type may be distributed  Distributions do not affect program semantics only implementation details and therefore performance Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

6. Distributions: Goals & Research  Advanced users can write their own distributions specified in Chapel using lower-level language features  Chapel will provide a standard library of distributions written using the same user-defined distribution mechanism (Draft paper describing user-defined distribution strategy available by request) Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

7. A Simple Distribution: Block1D Intent: block a 1D index space across a set of locales L0L1L2 01…… Use a bounding box to compute the blocking 8…… Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

8. Distributions vs. Domains Q1: Why distinguish between distributions and domains? Q2: Why do distributions map an index space rather than a fixed index set? A: To permit several domains to share a single distribution amortizes the overheads of storing a distribution supports trivial domain/array alignment and compiler optimizations const D : …dmapped B1 = [1..8], outerD: …dmapped B1 = [0..9], innerD: subdomain(D) = [2..7], slideD: subdomain(D) = [4..6]; L0L1L2 Sharing a distribution supports trivial alignment of these domains Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

9. Distributions vs. Domains Q1: Why distinguish between distributions and domains? Q2: Why do distributions map an index space rather than a fixed index set? A: To permit several domains to share a single distribution amortizes the overheads of storing a distribution supports trivial domain/array alignment and compiler optimizations const D : …dmapped B1 = [1..8], outerD: …dmapped B2 = [0..9], innerD: …dmapped B3 = [2..7], slideD: …dmapped B4 = [4..6]; L0L1L2 When each domain is given its own distribution, the compiler cannot reason about alignment of indices. Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

10. The Block Distribution maps the indices of a domain in a dense fashion across the target Locales according to the boundingBox argument The Block Distribution const Dist = new dmap(new Block(boundingBox=[1..4, 1..8])); var Dom: domain(2) dmapped Dist = [1..4, 1..8]; L0L1L2L3 L4L5L6L7 distributed over

11. The Cyclic Distribution maps the indices of a domain in a round-robin fashion across the target Locales according to the startIdx argument The Cyclic Distribution const Dist = new dmap(new Cyclic(startIdx=(1,1))); var Dom: domain(2) dmapped Dist = [1..4, 1..8]; L0L1L2L3 L4L5L6L7 distributed over

12. Domain Maps: Distributions and Layouts Domain Map: The general concept that indicates how to implement a domain and its arrays Two flavors: layout: a domain map targeting a single locale  row-major order, column major order, Morton order  hierarchically tiled, linked data structure, etc.  compressed sparse row, open hashing w/ quadratic probing distribution: a domain map targeting multiple locales  block, cyclic, block-cyclic  recursive bisection, hashing across locales, graph partitioning  …

13. Domain Map Framework: Layouts distributiondomainarray Responsibility: How to generate new domains Responsibility: How to store, iterate over domain indices Responsibility: How to store, access, iterate over array elements three descriptors: Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

14. Domain Map Framework: Distributions distributiondomainarray Responsibility: How to generate new domains and map indices to locales Responsibility: How to store, iterate over domain indices Responsibility: How to store, access, iterate over array elements global descriptors (one global instance or replicated per locale) local descriptors (one instance per locale) Responsibility: How to store, iterate over local domain indices Responsibility: How to store, access, iterate over local array elements Responsibility: How to store a single locale’s portion of the index space Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

15. Domain Map Framework: Distributions global descriptors (one global instance or replicated per locale) local descriptors (one instance per locale) distributiondomainarray domain descriptor distribution descriptor local array descriptor local domain descriptor array descriptor target locale set distribution args map index to locale … global index info index type index iteration allocate array … element type global value iteration random access … store local indices local index iteration add new indices … store local values local value iteration local random access … = descriptor state/type = descriptor methods legend local distribution descriptor Distributions Data Parallelism Task Parallelism Locality Control Target Machine Base Language

16. myElems = 45 68 31 1D Block Distribution Classes global descriptors local descriptors distributiondomainarray code L0L1 L2 myChunk = min(idxType)3 myChunk = max(idxType)6 myChunk = 45 myElems = 45 68 31 const B1 = new dmap( new Block(bbox=[1..8])); const D: domain(1) dmapped B1 = [1..8]; var A, B: [D] real; myBlock = 45 68 31 boundingBox = 18 targetLocales = L0L1L2 whole = 18 (LocaleSpace = [0..2])

17. myElems = 45 6 1D Block Distribution Classes global descriptors local descriptors distributiondomainarray code L0L1 L2 myChunk = min(idxType)3 myChunk = max(idxType)6 myChunk = 45 myElems = 45 6 const B1 = new dmap( new Block(bbox=[1..8])); const sliceD: domain(1) dmapped B1 = [4..6]; var A2, B2: [sliceD] real; myBlock = 45 6 0… boundingBox = 18 targetLocales = L0L1L2 whole = 46 (LocaleSpace = [0..2])