1. JavaSeis Parallel Arrays JavaSeis data structures Synchronous parallel model Arrays in Java mpiJava Parallel Distributed Arrays Examples

2. JavaSeis Data Structures View data as N-dimensional array Most common view is a series of 3D volumes Use generic names for each axis: Sample, Trace, Frame, Volume, Hypercube, … Associate “LogicalNames” with each axis: –Time, Offset, X-Line, InLine –Time, Channel, Shot, Swath

3. JavaSeis Dataset Logical View Sample Trace Frame Volume

4. JavaSeis Bounding Box X-Line InLine

5. Parallel Distributed Objects pdoExec pdoRead pdoWrite Node 0Node 1Node 2Node 3

6. The Transpose Problem CPU 0 CPU 1 CPU 2CPU 3 Local Access Remote Access

7. Data Parallel Transpose CPU 0 CPU 1 CPU 2CPU 3 Local Tile Transpose

8. Arrays in Java 1D arrays or “arrays of arrays” Subarrays and multi-dimensional “views” of a 1D array are not supported by the language Subarrays are constructed by passing the full array and an upper and lower bound Example reference from Matuszkek, University of PennsylvaniaMatuszkek

9. Design Sources NCAR / UCAR NetCDFNetCDF –University Center for Atmospheric Research High Performance Fortran –Ken Kennedy, Rice University 1993 Colorado School of Mines –Dave Hale, Mines Java ToolkitMines Java Toolkit Landmark ProWESS / SeisSpace ARCO Parallel Seismic Environment

10. DistributedArray Class Structure IMultiArray MultiArray DistributedArray ITranspose Transpose IParallelContext MPIContext mpiJava java.lang.Array TransposeType Decomposition

11. “parallel” and “array” packages org.javaseis.parallel –IParallelContext, ParallelContext Message passing support –Decomposition Define decompositions for array dimensions across processors org.javaseis.array –IMultiArray, MultiArray Containers for Fortran style multidimensional arrays –ITranspose, Transpose, TransposeType Transpose operations for Fortran style arrays –DistributedArray Extends MultiArray to distribute across processors

12. MultiArrayMultiArray Design Targets 1D Java arrays of primitive elements or Objects A superimposed "shape" that follows Fortran conventions Access via "range" triplets (start,end,increment) Ranges for Java zero based indexing or Fortran 1 to N based indexing Access to the "native" storage array for more arbitrary access Array "elements" can have multiple values (i.e. complex, multi-component) Designed to be extended to provide JavaSeis DistributedArrays Allow use of other array utility classes (java.util.Arrays, edu.mines.jtk.dsp.Array)

13. Transpose Operations TransposeType –Java “enum” that defines the set of available transpose operations (i.e. T312, T1243, T21) ITranspose, TransposeITransposeTranspose –Interface and “pure java” implementation –In-place 2D transpose is the basic operation –Extended to “132” transpose for 3D arrays –Combinations yield full set of 3D transposes –A single “1243” transpose provided for 4D

14. Message Passing IParallelContext –Interface for the minimal set of message passing needed to support JavaSeis Parallel Arrays –Send, Receive, getSize, getRank –Barrier, Broadcast (optional) –Shift, Transpose, BinaryTree built from the above –Init and Finish

15. MPI for Java mpiJava from Syracuse University (NPAC) selected for SeisSpacempiJava Java wrappers for native MPI calls Support for sending serialized objects MPIContext implements IParallelContextMPIContext MPICH for native methods Mpirun –np 16 –machinefile machines.txt java ClassName arguments

16. DistributedArray Extends MultiArray Requires IParallelContext for constructor Adds distributed tiled transpose (ttran) Last dimension is spread across processors (Decomposition, BLOCK or CIRCULAR)Decomposition Transpose operations support arbitrary distribution of a single dimension Multiple decompositions possible but not currently supported

17. Decomposition Design concept from High Performance Fortran Default decomposition is BLOCK –Allocates a fixed number of array indices per node –Remainder is “pushed” to the edge, NOT evenly allocated –May result in zero elements on high rank nodes –Simple start,end indexing with stride 1 CIRCULAR decomposition –Round robin allocation –Remainder spread across nodes –Good for load balancing –Permutation logic required to keep track of indices

18. BLOCK vs CIRCULAR Decomposition: 13 array indices on 4 nodes 1:4:1 5:8:1 9:12:1 13:13:1 1:13:4 2:10:4 3:11:4 4:12:4

19. 012 3 Time InLine X-Line 012 3 Frequency InLine Transform - Transpose Pattern

20. // Create a 3D distributed array DistributedArray a = new DistributedArray( Seis.getParallelContext(), 3, float.class, new int {512,256,128}, Decomposition.BLOCK ); // Transform x axis of an array in xyz order computeTransform1D( a ); // Transpose to yxz a.tran213(); // Transform y axis computeTransform1D( a ); // Transpose to zyx a.tran132(); // Transform z axis computeTransform1D( a ); // Transpose back to xyz a.tran321();

21. Distributed Array Padding Decomposition will likely have a remainder that requires padding Constructor allocates an array that accounts for padding Use constructor with an array of Decomposition’s if transpose operations will be used Index and range methods only traverse the “live” section of the array

22. Distributed Array Padding Live Section Decomposition Padding Padded Array Partial Array Section

23. Planned Additions Support for other patterns: –Transpose-Reduce –Transpose-Overlap Arrays of Arrays – optional variable length float[][]a = new float[][10]; for (int i=0; i<10; i++) a[i] = new float[i]; Parallel Sorting –Requires variable length “array of arrays”

24. PDO ( x, y | f ) PDO ( x, y | n ) PDO ( x, n | y ) Reduce - Transpose Pattern

25. The Overlap-Transpose Pattern 012301230123 Distributed array Overlap-Expand Locally Transpose to Distributed Overlap

26. Sort Parallel Data Sort

27. Variable length Transpose and resort within tile Block parallel output