1. Implementation: Charm++ Orion Sky Lawlor olawlor@acm.org

2. Background & Roadmap  Your code sits on Charm++:  Interface Translator .ci file, CProxy_*, parameter marshalling  Arrays (Orion++)  Reductions, broadcasts, migrations, pup  Groups and Chares (``Milind++)  Basic communication calls atop Converse  Basics (++)  Registration, readonlies, messages

3. Where is it?  Charm++ core source code in charm/src/Common/ck-core/  Gets soft-linked over and made in charm/net-linux/tmp/  Headers in charm/net-linux/include/  Code gets linked into the library charm/net-linux/lib/libck.a  This all gets used when you pass "-language charm++" to charmc

4. Basics:  We need to be able to call some C++ method bar on user object foo  But the core`s never heard of foo  Solution: register a function pointer to a "call-method" function: void _call_foo_bar(void *msg,foo *obj) { obj->bar(msg); }...at startup... int barEpIdx= CkRegisterEp(&_call_foo_bar,..);

5. Registration & Startup  ck-core/register.h.C (300 lines)  At startup, each module registers its methods, which end up in a big entry method table  An entry point's index in this table is its int "epIdx", used everywhere  Translator writes this registration code (the call-method function, "__register" function) in.def file

6. Groups & Chares (Milind++)  charm.h, ck.C (1000 lines)  Lowest layer of Charm++  CkCreateChare, CkCreateGroup  CkSendMsg, CkSendMsgBranch  Quite simple, thin wrapper on Converse  Very stable-rarely changes  Callable from C (!)

7. CkSendMsg(int epIdx, void *msg,CkChareID *to)  envelope *env = UsrToEnv(msg);  Finds Charm++ msg fields (see envelope.h)  env->setMsgtype(ForChareMsg);  env->setObjPtr(to->objPtr);  env->setEpIdx(epIdx);  env->setSrcPe(CkMyPe());  CmiSetHandler(env, _charmHandlerIdx);  CldEnqueue(to->onPE, env, _infoIdx);

8. CkSendMsg: Clutter  Various unrelated calls clutter up the Charm++ core (FIXME?)  Keep track of message sends; give "Message being re-sent..." diagnostic  _CHECK_USED(env);  _SET_USED(env, 1);  Trace message sends for projections  _TRACE_CREATION_1(env);  Notify quiescence detection of send  CpvAccess(_qd)->create();

9. Arrays (Orion++)  Many interesting features:  Arbitrary indexed collection of Chares  Connects to load balancer  Supports migration, broadcasts, reductions  Not even clear some operations even possible  But much more complicated  Highly fragmented, er, factored code  Array Manager, Location Manager, Reduction Manager, Broadcaster, Reducer

10. Arrays Diagram CkReductionMgr-- reductions CkLocMgr-- migration CkArray-- coordinator ArrayElement ArrayElement-- user's work

11. CkReductionMgr  ckreduction.h.C.ci (1300 lines)  Accept contributions from local members of a migrating set of reduction "contributors"  Implemented as Group  Actually "IrrGroup", since normal groups inherit from CkReductionMgr.  Exactly one per array; but zero dependence on arrays  Arrays are just a client of CkReductionMgr

12. Location-- CkLocMgr  ck-core/cklocation.h.C.ci (2000)  Keep track of location of migrating elements  Handles messaging and migration  Each array has a location manager  But a location manager may have several "bound" arrays, which will migrate together  Implemented as a group

13. Array Manager-- CkArray  ck-core/ckarray.h.C.ci (1300 lines)  Keeps track of a set of local array elements  Implemented as a group  Glue that connects pieces of array system together:  Inherits from CkReductionMgr  Implements CkArrMgr (for location mgr)  Interfaces with proxies  Controls ArrayReducer, ArrayBroadcaster

14. Pup  util/pup.h.C (1000 lines)  Used to migrate, parameter marshall  Heavy C++: overload the | operator  "Easy to use, tough to understand"  Bottom line: (user view)  Say "p|x;" for all your x's.  Bottom line: (system view)  We get a "void *" for each of the user`s x.

15. Translator  xlat-i/xi-symbol.h.C (3000 lines)  Turn.ci files into.decl/.def files  Standalone program "charmxi"  Tokenize.ci with lexx: xi-scan.l  Parse.ci with yacc: xi-grammar.y  Turn parsed structures into.decl/.def  Almost completely routines named "gen*"-- code-generating routines  E.g., "Entry::genDecls"-- generate code in.decl file for an entry method

16. Not Covered  Startup & Shutdown (init.C)  Quiescence detection (qd.C)  Dynamic load balancer (ck-ldb/)  Callbacks, futures (for sync methods), debug, TeMPO  Libraries (charm/src/Common/lib/)  TCharm, FEM, AMPI, MBlock, AMR, comm,...  Details! Read manuals, headers, our minds...

17. Conclusion  Interface Translator .ci file, CProxy_*, parameter marshalling  Arrays (Orion++)  Reductions, broadcasts, migrations, pup  Groups and Chares (``Milind++)  Basic communication calls atop Converse  Basics (++)  Registration, readonlies, messages