OPERATIONAL SIMULATORS GOING PARALLEL: From A Dream To A Concrete Concept Vemund Reggestad DASIA 20/06/2011 Paper authors: Vemund Reggestad (1), Nikolaos-A.I.Livanos (2), Pantelis Antoniou (3), Ioannis E. Venetis (4) (1) ESA/ESOC, Robert-Bosch-Str. 5, D-64293 Darmstadt, GERMANY, Vemund.Reggestad@esa.int (2) EMTech, Vizantiou 58, Papagou, Athens, 156 69, GREECE, nliv@emtech.gr (3) Antoniou Consulting, Imitou 56, Xolargos, Athens, 155 62 , GREECE, panto@antoniou-consulting.com (4) RDTL, Technolog. & Educat. Inst. of Athens, Ag.Spiridonos, Aegaleo, Athens, 122 10, GREECE, ivenetis@teiath.gr ESA UNCLASSIFIED – For Official Use

Table of Content Introduction Theoretical background Discrete Event Simulations (DES) Parallel Discrete Event Simulations (PDES) Parallelization possibilities in Simsat History A Naive PDES proof of concept. Compliance with SMP standard Critical areas Final Concept Conclusion ESA UNCLASSIFIED – For Official Use

Introduction Outcome of TRP study: ”Linux & Multi Core Processor Technology for Simulators“ Timeframe Q3 2009 to Q3 2011. Run by: Prime contractor: EmTech – ExelMicroTechnologies, Greece. Subcontractors: Pantelies & Thomas Antoniou OE, Greece. Technological & Educational Institute of Athens, Department of Electronics, Research and Development Telecom Lab – RDTL, Greece. ESA UNCLASSIFIED – For Official Use

Study objectives Development of the required prerequisites for a high performance simulator that can run efficiently on multi-core processor technology Scheduler for multi-core CPU Parallelization possibilities Develop a test tool (PerfSim), Based on the SMP-2 Reference Architecture A performance validation of the suitability of this architecture ESA UNCLASSIFIED – For Official Use

Theoretical background (DES) Identified main concepts of a DES simulation Model, System State, Event, Current Event List, Clock Analyzed function of the main simulation engine Remove event from the Current Event List Update clock Process Event Update System State Insert new events into Current Event List ESA UNCLASSIFIED – For Official Use

Theoretical background (PDES) Partition a simulation into a set of Logical Processes Difficulties that arise when using them Send/receive messages Complicates programming Large overhead ESA UNCLASSIFIED – For Official Use

Theoretical background (PDES) Last Processed Event Unprocessed Events Node 0 Node 1 New Event Scheduled Erroneously Processed Events The simulation system now contains multiple LPs and processors. Processors can process events at different rates. Now, here is the potential problem. It may easily happen that one processor is behind the other in simulated time. If an LP on this processor sends an event to any other processor the LP on the 2nd processor will receive an event in its past. Processing of this event will violate the causality constraint imposed upon the simulation. So, the following question rises? How can causality be maintained? To be more exact, how processes are to be synchronized in order to process events in the right sequence? There are two different approaches – Synchronization techniques: The Conservative simulation protocol and the Optimistic simulation protocol. The last processed event on “Node 0” generates an event that must be processed on ”Node 1”. Problem: The time-stamp of the generated event is smaller than the time-stamp of the event currently processed on “Node 1”. How to overcome the problem? There are two approaches: ESA UNCLASSIFIED – For Official Use

Theoretical background (PDES) Conservative Mechanisms Try to avoid causality errors in the first place Execute events for which no other event with a smaller timestamp Deadlock might occur Add protocol to avoid deadlocks Null messages Add protocol to detect deadlocks and recover Conservative Mechanisms examples: Synchronous operation Conservative Time Windows Improving Look ahead by Precomputing Service Times Conditional Knowledge Conservative approach: Never allow events to be processed if it is possible for “strangler” messages to arrive. However, this approach might get into a deadlock…. So, a protocol must be added in order to avoid it and recover. Here are some of the conservative mechanisms (Perhaps john would like to add something).. ESA UNCLASSIFIED – For Official Use

Theoretical background (PDES) Optimistic Mechanisms Allow causality errors to happen Detect and recover from them Requires saving of state If a causality error happens restore state TwLinux includes support for saving state in the OS itself Node 0 Node 1 Strangler Message Next Event to Process Events Rolled Back Anti-messages Node 2 Last Processed Event When using optimistic methods in a PDES simulation, it is allowed for causality errors to appear. Mechanisms are provided to detect and recover from them. In contrast to conservative mechanisms, optimistic mechanisms do not need to determine when it is safe to process an event. They process the event and determine when an error has occurred. In this case, they invoke a procedure to recover from the error. The most well-known optimistic protocol is the Time Warp mechanism. A causality error is detected whenever an event message is received that contains a timestamp smaller than that of the process’s clock. The event causing a rollback is called a “straggler”. Recovery is accomplished by undoing the effects of all the events that have been processed and have time-stamps larger than that of the straggler. An event may do two things that have to be rolled back: it may modify the state of the logical process, and/or it may send event messages to other processes. Rolling back the state is accomplished by periodically saving the process’s state, and restoring an old state vector on rollback. In order to undo the effects of a previously sent message, a negative or anti-message is sent, that annihilates the original message when it reaches its destination. Recursively repeating this procedure allows all the effects of the erroneous computation to eventually be canceled out. ESA UNCLASSIFIED – For Official Use

Parallelization in Simsat - History Traditionally Simsat supports a “Linear Scheduler”: Typical DES scheduler. A Concurrent scheduler was developed as part of Simsat 3: Concurrent-with-all events Concurrent-with-none events Background events Concurrency never used, all events “concurrent-with-none”. Distribution was introduced by Galileo in Simsat 3: Used by Galileo and Swarm Allows distribution of models with clock synchronisation. Does not allow model interactions. ESA UNCLASSIFIED – For Official Use

Parallelization in Simulus A Naive PDES proof of concept A conservative PDES selected Multiple logical processes (LP) One event lists per LP. (Simplification) One local clock per LP. (Simplification) Clock synchronization: A sliding window protocol selected (simple) Each LP synchronized against one master LP with a sliding window. (Simplification) ESA UNCLASSIFIED – For Official Use

Compliance with SMP/ECSS-E-40-07 SMP does cover scheduling of events. SMP does not cover parallelization or distribution, but does not prevent it either. A new service for parallelization can be implemented as a SMP user defined service. Ability to dynamically update the scheduling schema from the models. Copying what has been done for the existing concurrent scheduler. Enhancements to the SMP assembly to included mapping of models/events to Logical Processes could be done. ESA UNCLASSIFIED – For Official Use

Critical areas Emulator as single source of CPU consumption. Emulators are getting much faster. Other area of S/C is getting more complicated. Model re-entrance None of the existing models are re-entrant. Protection of internal variables? Need for redesign Non trivial task to optimize a scheduling for PDES. A wrong scheduling schema would run slower than a DES. Good tools are essential. ESA UNCLASSIFIED – For Official Use

Critical area: Tools for optimization An extensive set of tools has been prototyped as part of the TRP study called Performance Indicator Tools (PIT) covering: Simsat Performance Metrics Model Internal Performance Metrics External (ie. OS related) Profiling Tools Event Reporting System Performance optimization and parallelization is an interactive process that needs tools ESA UNCLASSIFIED – For Official Use

PIT tool example: Event Reporting ESA UNCLASSIFIED – For Official Use

Final concept Final Concept: Thread based Logical processes for PDES parallelization among cores. Conservative PDES Process based for distribution with Message passing interface. ESA UNCLASSIFIED – For Official Use

Overall architecture ESA UNCLASSIFIED – For Official Use

Overall architecture ESA UNCLASSIFIED – For Official Use

Overall architecture ESA UNCLASSIFIED – For Official Use

Overall architecture ESA UNCLASSIFIED – For Official Use

Overall architecture ESA UNCLASSIFIED – For Official Use

Conclusion Exploitation of parallelization is required due to the evolution in modern computer systems. A wide spectrum of theories exist for such systems, but no silver bullet. Many theories are specific to domains/problems rather than generic. A naïve PDES scheduler has been demonstrated. A concept for a full PDES scheduler has been developed. A few critical areas have been identified and solutions to most of the problems found. Next step: Full PDES implementation Validation with a real simulator. ESA UNCLASSIFIED – For Official Use

Any Questions? Conclusion Thanks to: ESA UNCLASSIFIED – For Official Use