University of Minnesota Comments on Co-Array Fortran Robert W. Numrich Minnesota Supercomputing Institute University of Minnesota, Minneapolis

2 Philosophy Behind the CAF Model A minimum number of new features that look and feel like Fortran. The rules for co-dimensions are the same as those for normal dimensions with a few exceptions. The CAF model is purely local. The compiler performs normal optimization between synchronization points. Compiler is not required to perform, is not expected to perform, is deliberately prevented from performing, global optimization. Programmer is responsible for explicit data distribution, explicit communication, and explicit synchronization. Programmer is responsible for memory consistency.

3 The Essential Features of CAF Co-array syntax Co-dimensions (How images are related to each other) Data communication (How data moves between images) Synchronization Full barrier Pair-wise handshake Dynamic Memory management Allocatable co-arrays Allocatable/pointer components of co-array derived types

4 Recommendations Delete all of Section 8.5 Substitute SYNC[([myPal])] Delete everything on I/O except Only image 1 reads stdin Records from different images not mixed to stdout Delete all the stuff on collectives Substitute the functions glbSum(x), glbMin(x), glbMax(x) Argument is not necessarily a co-array.

5 Ragged Arrays 1.The most important, the most powerful, feature of the CAF model. Allows each image to have data structures with different sizes and different locations on each image. No other model can handle this. Fortran with CAF extensions handles it in a very natural way. 2.Allocatable/pointer components of co-array derived types type x real,allocatable,target :: z(:) real,pointer :: ptr(:) type(x) :: y[  ] end type x allocate(z(someRule(this_image))) y%ptr => z sync y[p]%ptr = Most difficult feature to implement on crummy hardware and/or crummy operating systems. Good systems should be rewarded for getting this right. We should not allow bad systems to drag everything down to the lowest common denominator.

6 Memory Consistency The programmer is responsible for maintaining memory consistency, not the compiler. The rules must be very simple and very clear. With just SYNC, it is very simple The addition of NOTIFY/QUERY makes it not so simple Segment boundary statements - Compiler may optimize between segment boundaries but not across segment boundaries - Processor must make co-arrays “visible” to all images across segment boundaries - The programmer must make sure that one and only one image defines a co-array variable “at the same time” - The programmer must make sure that no image tries to reference a co-array variable “at the same time” as another image is trying to define it.

7 Input/Output What’s the minimum needed? –stdin/stdout are special cases – Always connected to all images – Only image 1 can read stdin – System must not mix records to stdout from different images –Shared files – Allow each image to open to same unit – Direct access only –Allowing sequential access requires changes to backspace, rewind, etc. – open(unit=u,access= ,…) – open(unit=u,access=connectList,…) –Do we need teams? –Do we need to sync?

8 Collectives Avoid language bloat - Not everything in the MPI Library needs to be reproduced in CAF as intrinsic procedures - What do we really need and want? - What should the interface be? CAF is intended as a low-level language - Collectives are easy to write in CAF for any specific procedure - Throwing a long list of new intrinsic procedures over the wall may discourage vendors from adopting CAF If anything, supply intrinsic functions: glbSum(x), glbMin(x), glbMax(x) argument x need not be a co-array Propose a supporting library for CAF.

9 VOLATILE Co-arrays? What do we want the VOLATILE attribute to mean for co-arrays? - Inhibit optimization? - Always read from memory? - Always flush to memory? Can VOLATILE make spin-loops work without the need for an artificial sync-memory() from the programmer? - Statements with co-arrays are segment boundary statements

10 SYNC/BARRIER One new statement SYNC - Implies full BARRIER for all images - No arguments No SYNC_IMAGES - Synchronization between subsets of images can be done with NOTIFY/QUERY

11 NOTIFY/QUERY Why do we want NOTIFY/QUERY? - Split-phase sync - Subset sync - Master-slave work distribution - Should they match in pairs? - Should we expose which notify matches which query? Maybe we really want EVENTS? - EVENT_POST(tag) - EVENT_WAIT(tag) - EVENT_CLEAR(tag)