1. 1 Creating Simulations with POSE Terry L. Wilmarth, Nilesh Choudhury, David Kunzman, Eric Bohm Parallel Programming Laboratory University of Illinois at Urbana- Champaign

2. 2 Inspiration Parallel Discrete Event Simulation of fine-grained tasks is notoriously difficult to parallelize and scale Classes of applications parallelize well with conservative synchronization, but most do not scale well Optimistic Synchronization coupled with Charm++'s virtualization, load balancing, communication optimization, etc. enable scalable general-purpose PDES (fine-grained or not)

3. 3 POSE 0.5a Prerequisites Parallel Discrete Event Simulation C++, Charm++ Minimal understanding of parallel programming Some understanding of optimistic synchronization mechanism for PDES

4. 4 Designing a POSE Simulation Decompose system to be modeled into its concurrent entities: posers Determine how the entities interact with discrete events: event methods Determine initial placement of entities Some parallel computing savvy required to maximize achievable parallelism

5. 5 Example: POSE to Charm++

6. 6 POSE to Charm++ Posers: special Charm++ chares that communicate via timestamped messages executed in order Event methods: Charm++ entry methods that receive timestamped event messages Strategy: method for synchronizing posers in parallel

7. 7 Optimistic Synchronization Events received by posers are queued by timestamp in poser's event queue Events executed in timestamp order according to synchronization strategy The poser's state is periodically checkpointed A straggler arrives, events are rolled back, spawned events are cancelled, and the poser's state is recovered

8. 8 Optimistic Synchronization in POSE After handling straggler, forward execution proceeds as before Fossil collection: checkpoint memory is recovered when no longer needed, i.e. when checkpoint is older than... GVT, global virtual time, the minimum virtual time in the entire simulation

9. 9 Using POSE POSE code translated to Charm++ Same program structure:.ci,.h,.C Same program code can be used to run sequential or parallel simulations Highly configurable

10. 10 Code Sample: Poser & Event Messages,.ci file message WorkerData; message WorkMsg; poser worker : sim adapt4 chpt { entry worker(WorkerData *); // Event methods entry [event] void work(WorkMsg *); };

11. 11 The.ci File Declare event message types: message WorkerMsg; Declare posers with synchronization strategy and representation type: poser worker : sim adapt4 chpt {... Declare event methods for posers which take an event message as parameter: entry [event] void work(WorkMsg *);

12. 12 Code Sample: Poser Declaration & Event Methods,.h file class worker { int someIntData; public: worker(); worker(WorkerData *m); ~worker(); worker& operator=(const worker& obj); void pup(PUP::er &p); // Event methods void work(WorkMsg *m); void work_anti(WorkMsg *m); void work_commit(WorkMsg *m); };

13. 13 The.h File Define posers and their state (a portion of the global state): class worker { int someIntData;... Declare any local helper methods required Declare contructors and required methods: Basic constructor, destructor, assignment operator, pup method

14. 14 The.h File, cont'd Declare event methods and corresponding anti-methods and commit methods: void work(WorkMsg *m); void work_anti(WorkMsg *m); void work_commit(WorkMsg *m); Anti-methods provide an alternative mechanism to checkpointing that allows the user to undo the state changes of an event method

15. 15 The.h File, cont'd Commit methods are executed when fossil collection is about to free the memory of a check-pointed state and “commit” to the execution of an event i.e. committed event can't be rolled back Useful for statistics collection, I/O, or any other activity that should only happen once

16. 16 Code Sample: Poser Constructor & Event Method Invocation,.C file worker::worker(WorkerData *m) { someIntData = m->someData; delete m; POSE_srand(myHandle); WorkMsg *wm; if (myHandle == 0) { wm = new WorkMsg; wm->someIntData = someIntData; POSE_invoke(work(wm), worker, POSE_rand()%42, POSE_rand()%10)); }

17. 17 The.C File Constructor receives message, uses data, deletes message (not true for event methods!) Every poser has a handle: myHandle if (myHandle == 0) {... worker::worker(WorkerData *m) { someIntData = m->someData; delete m;...

18. 18 The.C File, cont'd User decides what handle each poser has at construction time Any poser can invoke events on another poser as long as it knows the destination poser's handle Random number generation in POSE repeats same sequence in case of rollback and re-execution

19. 19 The.C File, cont'd Event method invocation: POSE_invoke(event_method(event_msg), poser_type, dest_handle, transit_time); event_msg is timestamped with OVT + transit_time when it arrives on poser dest_handle wm = new WorkMsg; wm->someIntData = someIntData; POSE_invoke(work(wm), worker, POSE_rand()% 42, POSE_rand()%10)); Each poser has its own virtual time: OVT; posers' OVTs can be out-of-sync

20. 20 Code Sample: Event Methods,.C file void worker::work(WorkMsg *m) { WorkMsg *wm; wm->someIntData = m->someIntData + someIntData; // fake computation POSE_busy_wait(1000); elapse(27); POSE_invoke(work(wm), worker, POSE_rand()%42, POSE_rand()%10); } void worker::work_anti(WorkMsg *m) { restore(this); } void worker::work_commit(WorkMsg *m) { }

21. 21 Passing Virtual Time We've seen how to make an event happen in the future via the transit_time parameter to POSE_invoke Elapse time on a poser: elapse(27); Increments poser's OVT by 27 Auto-elapse: a poser receives an event at time t > OVT; advance poser's OVT to t

22. 22 Passing Virtual Time, cont'd What if t < OVT? If the received event is inserted in the event queue before other executed events, it causes a rollback If not, the event is handled at time OVT, not at time t (events earlier than t kept the poser busy until time OVT)

23. 23 Event Methods and Event Messages When an event message arrives, it is queued on the destination poser as an event to be executed The actual message is stored in the queue along with any additional information associated with the event Because the event may be rolled back and re-executed, the message must not be deleted in the event method

24. 24 Anti-methods Typically, anti-methods only restore the checkpointed state: void worker::work_anti(WorkMsg *m) { restore(this); } But they can be used instead of checkpointing to undo simple state changes: void myClass::toggleFlag_anti(eventMsg *m) { flag ? flag=0 : flag=1; restore(this); }

25. 25 Output Printing information about progress, statistics or debugging data in PDES can be confusing in the face of rollbacks CommitPrintf(...) : buffers event execution output until the event is committed CommitError(...) : buffers error statements and aborts if an event results in an error that is committed

26. 26 A Main Program Programs that use posers are pure Charm++ --- they are not translated In main, just before creating posers, call POSE_init() to start simulation Then inject posers into the system: WorkerData *wd; for (int i=0; i<42; i++) { wd = new WorkerData; wd->Timestamp(0); int dest = rand() % CkNumPes(); (*(CProxy_worker *) &POSE_Objects)[i].insert (wd, dest); // i is this poser's handle }

27. 27 A Main Program User must timestamp constructor message and create object with Charm++ syntax Under the hood, constructs a worker in a Chare Array at index i on processor dest WorkerData *wd; for (int i=0; i<42; i++) { wd = new WorkerData; wd->Timestamp(0); int dest = rand() % CkNumPes(); (*(CProxy_worker *) &POSE_Objects)[i].insert (wd, dest); // i is this poser's handle }

28. 28 POSE_init() Initialization and Simulation start-up Can optionally specify endTime, a virtual time at which to halt the simulation Can optionally specify whether or not to use inactivity detection: terminates simulation if no events are handled for some period of time void POSE_init(); void POSE_init(int ET); void POSE_init(int IDflag, int ET);

29. 29 Choosing a Synchronization Strategy poser worker : sim adapt4 chpt {... POSE offers a wide variety of synchronization strategies ranging from conservative to aggressively optimistic Each type of poser can use the strategy best suited to its behavior opt*, spec, adapt*

30. 30 Choosing a Synchronization Strategy opt*: basic optimistic synchronization, throttled and unthrottled spec: throttled optimism, aggressive speculation adapt*: optimism and speculation adapt to recent behavior of poser

31. 31 Getting and Using POSE Got Charm++? You've got POSE. build pose... etrans.pl [-s] Worker Translates Worker.* to Worker_sim.* charmc... -module pose -language charm++ charmc... -module seqpose -language charm++

32. 32 Applications VHDL Simulation: David Kunzman Big Network Simulation: Nilesh Choudhury