1. Simulation in a Distributed Computing Environment
S. Guatelli1, P. Mendez Lorenzo2, J. Moscicki2, M.G. Pia1
1INFN Genova, Italy
2CERN, Geneva, Switzerland
IEEE Nuclear Science Symposium
San Diego, 30 October – 4 November 2006

2. Speed of Monte Carlo simulation
Speed of execution is often a concern in Monte Carlo simulation
Often a trade-off between precision of the simulation and speed of execution
Typical use cases
Semi-interactive response
Detector design
Optimisation
Oncological radiotherapy
Very long execution time
High statistics simulation
High precision simulation
Fast simulation
Variance reduction techniques
(event biasing)
Inverse Monte Carlo methods
Parallelisation
Methods for faster simulation response

3. Requirements Architectural requirements
Transparent execution in sequential/parallel mode
Transparent execution on a PC farm and on the Grid
Semi-interactive simulation
High statistics simulation
e.g.Geant4 brachytherapy
Execution time for 20 M events: 5 hours
Goal: execution time ~ few minutes
e.g. Geant4 medical_linac
Execution time for 109 events: ~10 days
Goal: execution time ~ few hours
Reference: sequential mode on a Pentium IV, 3 GHz

4. Features of this study Geant4 in a distributed computing environment
Architecture
Implications on Geant4 simulation applications
Environments
PC farm
GRID
Real life use case
Geant4 brachytherapy Advanced Example
Test on a single CPU
Test on a dedicated farm (60 CPUs)
Test on a farm shared with other users
Test on the GRID (LCG)

5. Parallel simulation execution: local cluster / GRID
Both applications have the same computing model
A job consists of a number of independent tasks which may be executed in parallel
Result of each task is a small data packet (few kb), which is merged as the job runs
In a local cluster
Computing resources are used for parallel execution
Input data for the job must be available on site
Typically there is a shared file system and a queuing system
Network is fast
GRID computing: resources from multiple computing centres
Typically there is no shared file system
(Parts of) input data must be replicated in remote sites
Network connection is slower than within a local cluster

6. architectural pattern
Strategy to minimise the cost of migrating a Geant4 simulation to a distributed environment
Master-Worker
architectural pattern
DIANE
DIANE is a layer which provides applications with a easy and convinient way of execution in the distributed, cluster environment.
THE DESIGN GOALS:
- easy customization and adaptability to different needs
- hide details of underlying technology (allow for easy migration)
- location independant – accessible from anywhere in the network
- limited to Master-Worker model which covers most of the typical jobs and needs in HEP
The Geant4 application developer is shielded from the complexity of underlying technology via DIANE

7. Distributed Simulation
Interface class
which binds together
Geant4 application and
Master-Worker framework
UML
Deployment Diagram
for Geant4 applications
Original Geant4 application source code unmodified
G4Simulation class responsible of managing the simulation
random number seeds
Geant4 initialisation
termination

8. Practical example: Geant4 simulation with analysis
Each task produces a file with histograms
The job result is the sum of histograms produced by tasks
Master-worker model
client starts a job
workers perform tasks and produce histograms
master integrates the results
Distributed Processing for Geant4 Applications
task = N events
job = M tasks
tasks may be executed in parallel
tasks produce histograms/ntuples
task output is automatically combined (add histograms, append ntuples)
Responsibilities
Master steers the execution of job, splits the job and merges the results
Worker initializes the Geant4 application and executes macros
Client gets the results

9. Overhead at initialisation/termination
Test on a single dedicated CPU (Intel ®, Pentium IV, 3.00 GHz)
Study execution via DIANE w.r.t. sequential execution
run 1 event
Standalone application
4.6  0.2 s
Application via DIANE, simulation only
8.8  0.8 s
Application via DIANE,
with analysis integration
9.5  0.5 s
Overhead: ~ 5 s, negligible in a high statistics job

10. Farm: execution time and efficiency
Dedicated farm : 30 identical bi-processors (Pentium IV, 3 GHz)
Thanks to Regional Operation Centre (ROC) Team, Taiwan
Thanks to Hurng-Chun Lee (Academia Sinica Grid Computing Center, Taiwan)
Load balancing: optimisation of the number of tasks and workers

11. Optimizing the number of tasks
The job ends when all the tasks are executed in the workers
If the job is split into a higher number of tasks, the chance that the workers finish the tasks at the same time is a higher
Note: the overall time of the job is determined by the last worker to finish the last task
Worker number
Time (seconds)
Worker number
Time (seconds)
Example of a job that can be improved from a performance point of view
Example of a good job balancing

12. Farm shared with other users
Real-life case: farm shared with other users
Execution in parallel mode on 5 workers of CERN LSF
DIANE used as intermediate layer
Preliminary!
The load of the cluster changes quickly in time
The conditions of the test are not reproducible
Highly variable performance

13. Results in real life use case
Required production of Brachytherapy: 20 M events
20 M events in sequential mode:16646 s (~ 4h 38’)
on an Intel ® Pentium IV, 3.00 GHz
The same simulation runs in 5’ in parallel on 56 CPUs
appropriate for clinical usage

14. How the GRID load changes
Execution time of Brachytherapy in two different conditions of the GRID
Worker number
Time (seconds)
Very different result!
20 M events, 60 workers initialized, 360 tasks
The load of the GRID changes quickly in time
The conditions of the test are not reproducible

15. Test results (LCG) with/without DIANE
Execution on the GRID, without DIANE
Execution on the GRID through DIANE,
20 M events,180 tasks, 30 workers
Worker number
Time (seconds)
Without DIANE:
2 jobs not successful due to set-up problems of workers
Through DIANE:
- All the tasks are executed successfully on 22 workers

16. Farm/GRID execution Preliminary indication
Brachytherapy application, 20 M events, 180 tasks
Taipei cluster:
29 machines, 734 s ~ 12 minutes
GRID:
27 machines, 1517 s ~ 25 minutes
Preliminary indication
The conditions are not reproducible

17. Lessons learned DIANE as intermediate layer Load balancing
Transparency
Good separation of the subsystems
Good management of CPU resources
Negligible overhead
Load balancing
A relatively large number of tasks increases the efficiency of parallel execution Trade-off between optimisation of task splitting and overhead introduced
Controlled and real life situation is quite different in a farm
Need dedicated farm for critical usage (i.e. hospital)
Grid
Highly variable environment
Not mature for critical usage yet
Work in progress, details still to be understood quantitatively

18. Conclusions
General solution for Geant4 simulation in a distributed computing environment
transparent sequential/parallel application
transparent execution on a local farm or on the Grid
user code is the same
Quantitative results
on-going work to understand details
Acknowledgments to:
M. Lamanna, L. Moneta, A. Pfeiffer (CERN)
LCG teams at CERN
Hurng-Chun Lee (ASGC, Taiwan)
Regional Operation Centre Team of Taiwan