Array Operation Synthesis to Optimize Data Parallel Programs Department of Computer Science, National Tsing-Hua University Student:Gwan-Hwan Hwang Advisor: Dr. Jenq Kuen Lee

Array Operation Synthesis to Optimize Data Parallel Programs 國立清華大學 資訊工程系 Student: 黃冠寰 Advisor: 李政崑博士

Array Operation Synthesis on Distributed-memory Machines 國立清華大學 資訊工程學系 黃冠寰, Phd.

Compiler Optimization for Compiler Optimization for Parallel Computations on Parallel Computations on Distributed & Shared Memory Distributed & Shared Memory Machines Machines Communication Code for Block-Cyclic Distribution of HPF(IPPS’98) Array Operation Synthesis for Intrinsic Array Functions (JPDC, ACM PPoPP’95, ICPP’96) Research Interests Key Issues Automatic Alignment for Data Parallel Languages (LCPC’97) Concurrent Testing Concurrent Testing Reachability Testing of Concurrent Program (IJSEKE’95, APSEC’93) Parallel Object Program Model & Parallel Object Program Model & Heterogeneous Computing Heterogeneous Computing Java-Based Network Computing Environment Transparent Parallel Computing Environment (Ongoing)

Outline of Presentation Fortran 90 Intrinsic Array Operations Array Operation Synthesis(AOS) SYNTOOL Apply AOS to Shared-Memory Machines Apply AOS to Distributed-Memory Machines Conclusion and Future Work

Outline of Presentation Fortran 90 Intrinsic Array Operations Array Operation Synthesis(AOS) SYNTOOL Apply AOS to Shared-Memory Machines Apply AOS to Distributed-Memory Machines Integrate AOS with Automatic Data Alignment Conclusion and Future Work

Intrinsic Array Operations Provided by Modern Program Languages.  E.g. Fortran 90, High Performance Fortran(HPF), HPF2, Fortran 97, APL, MATLAB, MATHEMATICA, NESL, C* Engineering and Scientific Applications Facilitate a Compilation Analysis for Optimization Support Parallel Execution and Portability

Intrinsic Array Operations(Cont’d) Array Operations Provided by Fortran 90, HPF. Examples: CSHIFT, TRANSPOSE, MERGE, EOSHIFT, RESHAPE SPREAD, Section Move, Where Constructs, Reductions. B=CSHIFT(A,1,1) C=TRANSPOSE(B)

Consecutive Array Expressions Array Expression Consecutive Array Operations C=EOSHIFT(MERGE(RESHAPE(S,/N,N/),A+B,T),1,0,1) FXP=CSHIFT(F1,1,+1) FXM=CSHIFT(F1,1,-1) FXP=CSHIFT(F1,2,+1) FYM=CSHIFT(F1,2,-1) FDERIV=ZXP*(FXP-F1)+ZXM*(FXM-F1)+ ZYP*(FYP-F1)+ZYM*(FYM-F1)

Classification of Array Operations Model Array Operations by Data Access Functions (DAF) Type 1Type 2Type 3 Type 4

Data Access Functions Represent Array Operations by Mathematical Functions Model Array Operations by Data Access Functions (DAF)  Single-Source, Multiple-Source  Single-Clause, multiple-Clause

Type 1: Single-source Single- clause Data Access Function One Source Array One Data Access Pattern B=TRANSPOSE(A) Data Access Function is B(I,J)=A(J,I)

Single-source Single-clause Data Access Function One Source Array One Data Access Pattern B=TRANSPOSE(A) Data Access Function is B(I,J)=A(J,I)

Type 2: Multiple-source Single- clause Data Access Function Multiple Source Arrays One Data Access Pattern R=MERGE(T,F,M) Data Access Function is where Array TArray FArray MArray R

Multiple-source Single-clause Data Access Function Multiple Source Arrays One Data Access Pattern R=MERGE(T,F,M) Data Access Function is where Array TArray FArray MArray R

Type 3: Single-source Multiple- clause Data Access Function Single Source Array Multiple Data Access Patterns B=CSHIFT(A,1,1) Data Access Function is Array AArray B : a segmentation descriptor

Single-source Multiple-clause Data Access Function Single Source Array Multiple Data Access Patterns B=CSHIFT(A,1,1) Data Access Function is Array AArray B : a segmentation descriptor

Type 4: Multiple-source Multiple- clause Data Access Function Multiple Source Arrays Multiple Data Access Patterns  No array operation of Fortran 90 belongs to type 4  Synthesis of multiple array operations may derive a type 4 data access function.

Multiple-source Multiple-clause Data Access Function Multiple Source Arrays Multiple Data Access Patterns  No array operation of Fortran 90 belongs to this type  Synthesis of multiple array operations may derive a multiple-source multiple-clause data access function

Straightforward Compilation Translate each operation into a parallel loop B=CSHIFT((TRANSPOSE(EOSHIFT(A,1,0,1),1,1) FORALL (I=1:N:1; J=1:N:1) T2 (I,J)= T1 (J,I) ENDFORALL FORALL (I=1:N:1; J=1:N:1) IF (1<=I<=N-1) and (1<=J<=N) THEN B(I,J)= T2 (I+1,J) ELSE B(I,J)= T2 (I-N,J) ENDFORALL FORALL (I=1:N:1; J=1:N:1) IF (1<=I<=N-1) and (1<=J<=N) THEN T1 (I,J)=A(I+1,J) ELSE T1 (I,J)=0 ENDFORALL EOSHIFT TRANSPOSE CSHIFT

Array Operation Synthesis Construct the Parse Tree of Array Expression Represent Array Operations by Mathematical Functions (DAF) B=CSHIFT((TRANSPOSE(EOSHIFT(A,1,0,1),1,1) CSHIFT TRANSPOSE EOSHIFT

Array Operation Synthesis (Cont’d) CSHIFT TRANSPOSE Synthesis of two functions COSHIFT+ TRANSPOSE EOSHIFT

Substitution (Term Rewriting like method)  Having two Data Access Patterns:  The Synthesized Data Access Pattern is: Synthesis of two Data Access Functions where

For example, By the substitution rule  Synthesis of two DAFs (Cont’d)

For example, Synthesis of two DAFs (Cont’d)

Code Generation for Synthesized Data Access Function FORALL (I=1:N:1; J=1:N:1) IF  (/I,J/,/1:N-1,1:N/)   (/J,I+1/,/1:N-1,1:N/) THEN B(I,J)=A(J+1, I+1) IF  (/I,J/,/1:N-1,1:N/)   (/J,I+1/,/N:N,1:N/) THEN B(I,J)=0 IF  (/I,J/,/N:N,1:N/)   (/J,I+1/,/1:N-1,1:N/) THEN B(I,J)=A(J+1, I-N+1) IF  (/I,J/,/N:N,1:N/)   (/J,I+1/,/N:N,1:N/) THEN B(I,J)=0 ENDFORALL Code Generation

Code Generation for Synthesized Data Access Function After Optimization 1 N-1 N 1 N

Simplifying the ranges at compilation time instead of runtime Optimization process:  Normalize:  Intersection for each dimension: Optimization