1. Charm++ Arrays, Parameter Marshalling, Load Balancing and what they have in common-- PUP 9/1/2001 Orion Sky Lawlor PPL Developer Bachelor's in Math and CS from Univ. of Alaska at Fairbanks MS in CS from UIUC

2. Arrays: the Basic Idea " So there's some parallel object " How do you talk to it? " MPI approach: send a message to the processor number " Make migration (almost) impossible " Makes load balancing (almost) impossible " Array approach: send message to array index (1D, 2D, etc.)

3. Array Syntax (.ci) module demo { array [1D] A { entry A(void); entry void foo(void); }

4. Array Syntax (.h) class A:public ArrayElement1D { public: A(void); void foo(void); //For migration: A(CkMigrateMessage *) { } void pup(PUP::er &p); };

5. Array Creation... in main chare... int n=17; CProxy_A ap=CProxy_A:: ckNew(n);//Create array //Elements numbered {0..n-1} //Always dense //Always 1D

6. Sparse Array Creation... in main chare... int n=17; CProxy_A ap=CProxy_A:: ckNew();//Create array for (int i=0;i<n;i++) ap[7+2*i].insert(); ap.doneInserting(); //Don't forget the doneInserting!

7. Array Messaging //Just index the array proxy ap[7].foo(); //Or can make element proxy: CProxyElement_A aep=ap[9]; aep.foo(); //For higher dimensions: ap2(x,y).foo(); ap3(x,y,z).foo();

8. Array Broadcast //Leave off the index: ap.foo(); " Invokes foo on every existing array element " Works even with migrations " Bad syntax: easy to broadcast by mistake, causing disaster!

9. Array Reduction int myData[2]={...}; contribute(2*sizeof(int), (void *)&myData, CkReduction::sum_int); " Called from every array element " Collects result to PE 0 & calls fn: void myCli(void *userData, int redLen,void *redData); ap.setReductionClient(myCli,userData);

10. Arrays PUP

11. PUP-- Wazzup? " Before PUP, had to write 3 functions: " size-- compute message size, in bytes " pack-- write object into message " unpack-- read object out of message " PUP combines all three into one " Basic contract: here are my fields (types, sizes, and a pointer) " C++ version uses operator overloading for very spartan syntax-- just parens() and bar|

12. PUP Single Fields class simple { int x; float y; public: simple() { }... other methods... void pup(PUP::er &p) { p|x; p|y; } }; PUPmarshall(simple);

13. PUP Arrays class withArray { public: int n; double *d;//Holds n doubles simple() {n=0;d=NULL;} ~simple() { delete[] d;}... other methods... void pup(PUP::er &p) { p|n; if (p.isUnpacking()) d=new double[n]; p(d,n); } }; PUPmarshall(withArray);

14. PUP Subobject class sub { public: simple *s;//Holds a subobject sub() {s=NULL;} ~sub() { delete s;}... other methods... void pup(PUP::er &p) { if (p.isUnpacking()) s=new simple; s->pup(p); } }; PUPmarshall(sub);

15. PUP::able & auto-allocate class A:public PUP::able { public:... other methods... A(CkMigrateMessage *) { } virtual void pup(PUP::er &p); PUPable_decl(A); }; PUPable_def(A); (...in an initcall routine...) PUPable_reg(A); (...in some object's pup...) p|a;

16. PUP::able & trees class A:public PUP::able { A *left,*right;... as before... }; void A::pup(PUP::er &p) { p|left; p|right; } " Packs, allocates, and unpacks left and right children automatically " Works properly with NULL pointers " Doesn't work for object graphs with cycles! (Never terminates)

17. PUP::able & inheritance class B:public A {... the usual... PUPable_decl(B); }; PUPable_def(B); (...in an initcall routine...) PUPable_reg(B); " Now an A pointer that actually points to a B object will be restored properly-- B::pup will be called. " Combines well with virtual functions.

18. PUP Param. Marshall

19. Parameter Marshalling entry void foo(void); entry void bar(int a,int b); entry void baz(float f[7]); entry void far(int n,double d[n]); entry void boo(int n,int m, unsigned char area[n*m]); entry void oof(char s[strlen(s)+1]); entry void fub(something_t s); " Arrays passed as pointers in C++ " Array length evaluated on call side " Can declare everything “const", “&"

20. Param. Marshalling & PUP entry void foo(T a, V b); " Gets translated into: //Send side: CProxy_A::foo(T &a, V &b) {...make PUP::er &p... p|a; p|b;...use resulting p... } //Recv. side:...in call-foo function... T a; V b; p|a; p|b; obj->foo(a,b); " Because there’s a default bar operator, T and V can be C structs; which are copied as bytes. " Because you can overload the bar operator using PUP_Marshall, if T and V have pup routines they’ll be called properly! " Makes marshalling flexible.

21. Param. Marshall Load Balancing

22. " All you need is a working pup " Load balancer will migrate constantly " Create the load balancer with: //In the.ci file: extern module RefineLB; //In the.C file: #include "RefineLB.h" //In the main chare: CreateRefineLB(); //Also: CommLB, GreedyRefLB, HeapCentLB, MetisLB, NeighborLB

23. AtSync Load Balancing " When you need more control over when balancing happens. " Create balancer as usual, then: //In the element constructor: usesAtSync=CmiTrue; //When ready to balance call: AtSync(); //When computation resumes, void ResumeFromSync(void); //gets called on each element.

24. Conclusion " Arrays support messaging, broadcasts, reductions, and migration " Migration is for load balance " Migration uses PUP " Parameter marshalling uses PUP " Can get lots of functionality (almost) for free with PUP