Benchmarks of medical dosimetry simulation on the grid S. Chauvie 1, A. Lechner 4, P. Mendez Lorenzo 5, J. Moscicki 5, M.G. Pia 6 G.A.P. Cirrone 2, G. Cuttone 2, F. Di Rosa 2, F. Foppiano 3, M. Piergentili 6, G. Russo 2 Thanks for contributing to the development of Geant4 medical physics Advanced Examples! 1 Santa Croce and Carle Hospital, Cuneo, Italy 2 INFN Laboratori Nazionali del Sud, Italy 3 IST, Genova, Italy 4 Technical University Vienna, Austria 5 CERN, Geneva, Switzerland 6 INFN Genova, Italy IEEE NSS 2007 Honolulu, HI, USA 27 October – 3 November 2007

Monte Carlo methods in oncological radiotherapy More precise than approximated analytical methods used in commercial treatment planning software systems BUT Too slow to be realistically usable in the clinical practice Variance reduction techniques (event biasing) Variance reduction techniques (event biasing) Inverse Monte Carlo methods Inverse Monte Carlo methods Analytical transport methods Analytical transport methods “Fast simulation” techniques “Fast simulation” techniques Parallelisation Parallelisation Reduce simulation execution time The GRID?

This project Practical approach to the problem Let’s take a few representative dosimetry simulation use cases Let’s run them on the grid Let’s evaluate (quantitatively) gains and drawbacks in realistic situations Three Geant4 Advanced examples of dosimetry simulation covering major techniques in oncological radiotherapy –Brachytherapy –Hadrontherapy –LINAC for IMRT (Intensity Modulated RadioTherapy) LCG infrastructure, Geant4 Virtual Organization –The exercise will be repeated one step further as Mr. Nobody No introduction on grid, EGEE and other flavours, LCG, medical-related grid projects, their impact etc.

brachytherapy Leipzig applicator MicroSelectron-HDR source Distance along Z (mm) Simulation Nucletron Data Exp. data: F. Foppiano et al., IST Genova Relatively “fast” simulation ~ 7 CPU hours on an “average” PC to produce meaningful statistics for clinical studies Italian National Institute for Cancer Research Geant4 brachytherapy Advanced Example

hadrontherapy Accurate reproduction of the CATANA hadrontherapy facility at INFN LNS, Catania, Sicily, Italy Geant4 hadrontherapy Advanced Example Geant4 geometry and beam primary generator implementation developed by INFN-LNS CATANA co-authors (Partial) code review by A.L. and MGP See related talk in Event Processing Session The experimental data shown in this plot have been taken at the CATANA hadrontherapy facility at INFN LNS, Catania by G.A.P. Cirrone, G. Cuttone, F. Di Rosa, G. Russo. No credit to other co-authors and their institutes for these data. Production aimed at validating physics modelling of the simulation ~ 150 CPU hours on an “average” PC to produce meaningful physics results ~ 16 hours for calibration results

Medical LINAC for IMRT High demand of CPU resources for meaningful statistics (e.g. for treatment planning verification) ≈ tens of CPU-days Geometry developed by M. Piergentili Italian National Institute for Cancer Research Exp. data: F. Foppiano et al., IST Genova Geant4 medical_linac Advanced Example

Requirements Submit jobs to a Grid system Plug-in Geant4 application Split into tasks (without biasing the results) Schedule tasks intelligently Hide the underlying complexity to the end user DIANE Ganga LCG Geant4 AIDA iAIDA ++ ++

DIANE Geant4 Interface Interface class which binds together Geant4 application and DIANE framework DIANE Master-Worker architecture

Computing environment Identification of resources –The site must support the Virtual Organization (Geant4) –Shared file system accessible through VO_GEANT4_SW_DIR environment variable –Operating system and architecture compatible with Geant4 Software installation –Geant4, CLHEP, AIDA, gcc –Problems encountered: missing write permission, user authentication etc. –Requires familiarity with grid middleware Requires experienced user One must do one’s own installation on the grid Interface to grid middleware (e.g. lxplus) must be set-up, otherwise additional (not trivial) installation needed (e.g. gLite)

Problem set-up Determination of the total # of events to be produced –Defined for each application on the basis of the desired statistical precision of dosimetric variables relevant to each use case Splitting into tasks –Task = smaller group of events to be produced Perform tasks sequentially on a worker node Use a range of worker nodes in parallel in a grid environment Critical issue

Task splitting Important for applications with short execution time –Time elapsed between submission of grid jobs and beginning execution on a worker node comparable to total duration High granularity → more effective use of CPU resources –Drawbacks: in general, larger output size to deal with Task size (execution time) # workers involved balance

Splitting optimization Goal: fastest simulation results Many small tasks –20000 events, ~ 25 s each 40 worker nodes Higher # nodes would not be more effective –Time for last registered workers to become active might be higher than total duration Goal: sensitive simulation study –beam energy calibration 1 task = 1 worker –Not the fastest option Compromise with large disk space requirements for output Simplify distribution of random number seeds passed to tasks “Fast” use case brachytherapy “Intensive” use case hadrontherapy “One size DOES NOT fit all”

Baby-sitting Selection of resources –Availability, priority Set-up a list of Computing Elements –Passed to Ganga through DIANE Criteria for selection –State of Computing Element (production, draining mode) –Number of CPU allowed to VO at a given site –Estimated waiting time in batch queue –Estimated additional workload on given nodes –CPU speed Relatively easy retrieval of information –Used for selection criteria Must be done on a regular basis Automated tools not always effective

Effects of delayed worker registration fast worker registration a slow worker registration b Unsatisfactory conditions: delayed worker registration, worker errors (crash, network problems etc.) c c b a brachytherapy Effects more relevant to “fast” use cases

Effects of slow worker nodes More visible when few worker nodes Similar execution time of all tasks on all nodes One slow node doubles the total simulation duration Both simulations executed in the same environment in identical Computing Elements Possible explanation: variable workload on the same nodes

“Fast” use case: brachytherapy Sequential 1 task = 25 ± 0.5 CPU s Total simulation = 417 ± 8 min on lcgui003 Task duration Total simulation duration ~ factor 6 between min and max duration ~ factor 4 between min and max duration 40 DIANE workers On the grid 64% runs terminated < 30 min 96% runs terminated < 40 min Ideal expectation with 40 workers: ~10 min for the whole simulation

More computationally intensive use case: hadrontherapy Sequential 1 task = 50’ ± 50 s CPU time Total simulation = 16.7 h ± 17 min 20 DIANE workers On the grid 84% runs terminated < 100 min Ideal expectation with 20 workers: ~50 min for the whole simulation

Large scale medical simulations Test case: hadrontherapy – Increase statistics by 2 orders of magnitude –10M events for high precision physics studies –Expected duration of whole production: 2-3 days 3 large statistics runs –2 did not complete (80% done) –1 terminated successfully Medical linac –Even more demanding than high statistics hadrontherapy –“Low” statistics demonstration OK –Realistic scale stable production does not look practical yet

Conclusion Politically correct Grid technology offers unprecedented opportunities to the medical physics community for fast, high precision Monte Carlo simulation Politically incorrect Grid technology is not mature yet for stable operation in a clinical environment by non-expert users Scientific Both statements above reflect reality Quantitative tests evaluate the extent of success and contribute to identify problems and bottlenecks Measurements in real-life use cases are the path to solve problems

Anton Lechner Austrian Doctoral Student CERN IT/PSS A BIG thank you! Jürgen KnoblochPatricia Mendez LorenzoKuba Moscicki Major contribution to the results presented! and Hurng-Chung Lee Alberto Ribon Alfonso Mantero Graduate Student at INFN Genova 1 st Geant4-DIANE project CERN IT/PSS

IEEE Transactions on Nuclear Science Prime journal on technology in particle/nuclear physics Rigorous review process  Associate Editor dedicated to computing papers Various papers associated to CHEP 2006 published on IEEE TNS Computing-related papers are welcome Manuscript submission: Publications on refereed journals are beneficial not only to authors, but to the whole community of computing-oriented physicists Our “hardware colleagues” have better established publication habits… Further info: