1. OPERATIONAL SIMULATORS GOING PARALLEL: From A Dream To A Concrete Concept
Vemund Reggestad DASIA 20/06/2011
Paper authors:
Vemund Reggestad (1), Nikolaos-A.I.Livanos (2), Pantelis Antoniou (3), Ioannis E. Venetis (4)
(1) ESA/ESOC, Robert-Bosch-Str. 5, D Darmstadt, GERMANY,
(2) EMTech, Vizantiou 58, Papagou, Athens, , GREECE,
(3) Antoniou Consulting, Imitou 56, Xolargos, Athens, , GREECE,
(4) RDTL, Technolog. & Educat. Inst. of Athens, Ag.Spiridonos, Aegaleo, Athens, , GREECE,
ESA UNCLASSIFIED – For Official Use

2. Table of Content Introduction Theoretical background
Discrete Event Simulations (DES)
Parallel Discrete Event Simulations (PDES)
Parallelization possibilities in Simsat
History
A Naive PDES proof of concept.
Compliance with SMP standard
Critical areas
Final Concept
Conclusion
ESA UNCLASSIFIED – For Official Use

3. Introduction
Outcome of TRP study: ”Linux & Multi Core Processor Technology for Simulators“
Timeframe Q to Q
Run by:
Prime contractor: EmTech – ExelMicroTechnologies, Greece.
Subcontractors:
Pantelies & Thomas Antoniou OE, Greece.
Technological & Educational Institute of Athens, Department of Electronics, Research and Development Telecom Lab – RDTL, Greece.
ESA UNCLASSIFIED – For Official Use

4. Study objectives
Development of the required prerequisites for a high performance simulator that can run efficiently on multi-core processor technology
Scheduler for multi-core CPU
Parallelization possibilities
Develop a test tool (PerfSim),
Based on the SMP-2 Reference Architecture
A performance validation of the suitability of this architecture
ESA UNCLASSIFIED – For Official Use

5. Theoretical background (DES)
Identified main concepts of a DES simulation
Model,
System State,
Event,
Current Event List,
Clock
Analyzed function of the main simulation engine
Remove event from the Current Event List
Update clock
Process Event
Update System State
Insert new events into Current Event List
ESA UNCLASSIFIED – For Official Use

6. Theoretical background (PDES)
Partition a simulation into a set of Logical Processes
Difficulties that arise when using them
Send/receive messages
Complicates programming
Large overhead
ESA UNCLASSIFIED – For Official Use

7. Theoretical background (PDES)
Last Processed Event
Unprocessed Events
Node 0
Node 1
New Event Scheduled
Erroneously Processed Events
The simulation system now contains multiple LPs and processors. Processors can process events at different rates.
Now, here is the potential problem.
It may easily happen that one processor is behind the other in simulated time. If an LP on this processor sends an event to any other processor the LP on the 2nd processor will receive an event in its past. Processing of this event will violate the causality constraint imposed upon the simulation. So, the following question rises? How can causality be maintained? To be more exact, how processes are to be synchronized in order to process events in the right sequence? There are two different approaches – Synchronization techniques: The Conservative simulation protocol and the Optimistic simulation protocol.
The last processed event on “Node 0” generates an event that must be processed on ”Node 1”.
Problem: The time-stamp of the generated event is smaller than the time-stamp of the event currently processed on “Node 1”.
How to overcome the problem? There are two approaches:
ESA UNCLASSIFIED – For Official Use

8. Theoretical background (PDES)
Conservative Mechanisms
Try to avoid causality errors in the first place
Execute events for which no other event with a smaller timestamp
Deadlock might occur
Add protocol to avoid deadlocks
Null messages
Add protocol to detect deadlocks and recover
Conservative Mechanisms examples:
Synchronous operation
Conservative Time Windows
Improving Look ahead by Precomputing Service Times
Conditional Knowledge
Conservative approach: Never allow events to be processed if it is possible for “strangler” messages to arrive.
However, this approach might get into a deadlock…. So, a protocol must be added in order to avoid it and recover.
Here are some of the conservative mechanisms (Perhaps john would like to add something)..
ESA UNCLASSIFIED – For Official Use

9. Theoretical background (PDES)
Optimistic Mechanisms
Allow causality errors to happen
Detect and recover from them
Requires saving of state
If a causality error happens restore state
TwLinux includes support for saving state in the OS itself
Node 0
Node 1
Strangler Message
Next Event to Process
Events Rolled Back
Anti-messages
Node 2
Last Processed Event
When using optimistic methods in a PDES simulation, it is allowed for causality errors to appear. Mechanisms are provided to detect and recover from them. In contrast to conservative mechanisms, optimistic mechanisms do not need to determine when it is safe to process an event. They process the event and determine when an error has occurred. In this case, they invoke a procedure to recover from the error.
The most well-known optimistic protocol is the Time Warp mechanism. A causality error is detected whenever an event message is received that contains a timestamp smaller than that of the process’s clock. The event causing a rollback is called a “straggler”. Recovery is accomplished by undoing the effects of all the events that have been processed and have time-stamps larger than that of the straggler.
An event may do two things that have to be rolled back: it may modify the state of the logical process, and/or it may send event messages to other processes. Rolling back the state is accomplished by periodically saving the process’s state, and restoring an old state vector on rollback. In order to undo the effects of a previously sent message, a negative or anti-message is sent, that annihilates the original message when it reaches its destination. Recursively repeating this procedure allows all the effects of the erroneous computation to eventually be canceled out.
ESA UNCLASSIFIED – For Official Use

10. Parallelization in Simsat - History
Traditionally Simsat supports a “Linear Scheduler”:
Typical DES scheduler.
A Concurrent scheduler was developed as part of Simsat 3:
Concurrent-with-all events
Concurrent-with-none events
Background events
Concurrency never used, all events “concurrent-with-none”.
Distribution was introduced by Galileo in Simsat 3:
Used by Galileo and Swarm
Allows distribution of models with clock synchronisation.
Does not allow model interactions.
ESA UNCLASSIFIED – For Official Use

11. Parallelization in Simulus A Naive PDES proof of concept
A conservative PDES selected
Multiple logical processes (LP)
One event lists per LP. (Simplification)
One local clock per LP. (Simplification)
Clock synchronization:
A sliding window protocol selected (simple)
Each LP synchronized against one master LP with a sliding window. (Simplification)
ESA UNCLASSIFIED – For Official Use

12. Compliance with SMP/ECSS-E-40-07
SMP does cover scheduling of events.
SMP does not cover parallelization or distribution, but does not prevent it either.
A new service for parallelization can be implemented as a SMP user defined service.
Ability to dynamically update the scheduling schema from the models.
Copying what has been done for the existing concurrent scheduler.
Enhancements to the SMP assembly to included mapping of models/events to Logical Processes could be done.
ESA UNCLASSIFIED – For Official Use

13. Critical areas Emulator as single source of CPU consumption.
Emulators are getting much faster.
Other area of S/C is getting more complicated.
Model re-entrance
None of the existing models are re-entrant.
Protection of internal variables?
Need for redesign
Non trivial task to optimize a scheduling for PDES.
A wrong scheduling schema would run slower than a DES.
Good tools are essential.
ESA UNCLASSIFIED – For Official Use

14. Critical area: Tools for optimization
An extensive set of tools has been prototyped as part of the TRP study called Performance Indicator Tools (PIT) covering:
Simsat Performance Metrics
Model Internal Performance Metrics
External (ie. OS related) Profiling Tools
Event Reporting System
Performance optimization and parallelization is an interactive process that needs tools
ESA UNCLASSIFIED – For Official Use

15. PIT tool example: Event Reporting
ESA UNCLASSIFIED – For Official Use

16. Final concept Final Concept:
Thread based Logical processes for PDES parallelization among cores.
Conservative PDES
Process based for distribution with Message passing interface.
ESA UNCLASSIFIED – For Official Use

17. Overall architecture
ESA UNCLASSIFIED – For Official Use

18. Overall architecture
ESA UNCLASSIFIED – For Official Use

19. Overall architecture
ESA UNCLASSIFIED – For Official Use

20. Overall architecture
ESA UNCLASSIFIED – For Official Use

21. Overall architecture
ESA UNCLASSIFIED – For Official Use

22. Conclusion
Exploitation of parallelization is required due to the evolution in modern computer systems.
A wide spectrum of theories exist for such systems, but no silver bullet.
Many theories are specific to domains/problems rather than generic.
A naïve PDES scheduler has been demonstrated.
A concept for a full PDES scheduler has been developed.
A few critical areas have been identified and solutions to most of the problems found.
Next step:
Full PDES implementation
Validation with a real simulator.
ESA UNCLASSIFIED – For Official Use

23. Any Questions? Conclusion Thanks to:
ESA UNCLASSIFIED – For Official Use