FESR Consorzio COMETA - Progetto PI2S2 Grid computing for medical Monte Carlo application Giorgio Russo 1, R. Barbera 2, G.A.P. Cirrone 3, G. Cuttone 3, F. Di Rosa 3, E. Giorgio 2, S. Lo Nigro 2, F. Romano 3, M.G. Sabini 4 1)Consorzio COMETA – INFN LNS 2)Università di Catania, INFN sezione di Catania 3)LNS-INFN 4)A. O. Cannizzaro di Catania Grid Open Day al LNS Catania,

Palermo, Grid Open Days all’Università di Palermo, Outline Monte Carlo Method – GEANT4 toolkit Our Applications: –HadronTherapy, CATANA beam line –GammaKnifeRS, Gamma Knife Monte Carlo Parallelization –How Grid works –Grid technology implementation –Grid problems –Our experiences Results Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Monte Carlo Method – GEANT4 toolkit In our applications we have used the GEANT4 Toolkit –Advantages:  Detailed geometry description  Precise magnetic/electric field description  Easier material composition  Variety of physics processes  Precise visualization  Extendibility  Open Source –Disadvantage:  Learning curve  It is not a black box to give correct results  Time run Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Simulation of CATANA beam line HadronTherapy is a GEANT4 Advanced Example, It is a free application downloadable from GEANT4 home page Grid Open Day al LNS, All components of CATANA beam line are simulated HadronTherapy can be used to simulate and to optimize a generic eye proton therapy beam line

Palermo, Grid Open Days all’Università di Palermo, HadronTherapy At the Isocenter a detector is simulated to reconstruct the Bragg Peak and the Spread Out Bragg Peak The simulation was used to validate the Treatment Planning Software, employed at CATANA facility EyePlan, in a clinical configuration Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Simulation of Gamma Knife Gamma Knife is a stereotactic radiosurgery for brain tumor Stereotactic radiosurgery is a technique for treating intracranial lesions with a single high dose of ionizing radiation It uses the convergence of 201 beams from 60 C0 radioactive sources Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Simulation of Gamma Knife The application has been validated comparing the results with the experimental ones The application can be used: –to verify the dose distribution inside the patient brain –to verify and understand the limitations of the Treatment Planning System (TPS) All physical and geometrical characteristics are simulated Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Clinical Treatment Simulation The TPS ignores tissue inhomogeneities by assuming the human head as water equivalent Grid Open Day al LNS, Bone Shell Air Cube Water Air Tumor Simulation of a cavernous sinus meningioma treatment It is simulated on an inhomogeneous phantom

Palermo, Grid Open Days all’Università di Palermo, Clinical Treatment Simulation Grid Open Day al LNS, The main drawback for a Monte Carlo application is the long calculation time –needed to acquire an adequate level of accuracy. For example: –HadronTherapy needs 10 hours to obtain a good information about dose distributions.  ( 1 Mega protons using a Pentium GHz ) –GammaKnifeRS needs 21 days to obtain a good information about dose distributions for a single shot.  ( 1 Billion photons using a Pentium GHz) Photon histories are completely independent from each other BUT Parallel computing is therefore particularly adequate SO

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Parallelization The large number of histories required is divided into shorter simultaneous subtasks The subtasks are generated with different seeds to ensure that the histories are statistically independent The reduction time is related to number of simultaneous subtasks A cluster could be used for this issue but GRID is the best solution !!

Palermo, Grid Open Days all’Università di Palermo, How Grid works Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Matchmaking & Job Run The Resource Broker –Looks for user requirements to run the job –Obtains the list of computing element (CE) matching user requirements –Chooses the best one and submits the job The Computing Element –Runs the job –Generates the output

Palermo, Grid Open Days all’Università di Palermo, Job Description Language A JDL file has attributes to specify: –the InputSandbox and the OutputSandbox, for the input and output files –how the job has to run setting “Executable” and ”Arguments” –Requirements –If there are some files stored in the SE Type = "Job"; JobType = "Normal"; Executable = "/bin/sh"; StdOutput = "hadrontherapy.out"; StdError = "hadrontherapy.err"; InputSandbox = {"hadrontherapy.sh"}; OutputSandbox = {"hadrontherapy.err","hadrontherapy.out","piccoGrid.dat","disAngGrid.dat","disEnGrid.dat"}; RetryCount = 7; Arguments = "hadrontherapy.sh"; Requirements = Member("GEANT4-6",other.GlueHostApplicationSoftwareRunTimeEnvironment); Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Job Status Submitted: the job is submitted from the user but it is not processed by RB yet Waiting: the job is in the RB and it is waiting to be processed Ready: the job is processed by RB and it is waiting to be sent to a CE Scheduled: the job is in the CE and it is waiting to be run Running: the job is running Done: the job is finished successfully Aborted: the job is aborted by the Workload Management System Cancelled: the job is cancelled by the user Cleared: the job is done and the output is recovered by the user Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Status of computing element (CE) and storage element (SE) **************************************************************** These are the related data for gridit: (in terms of queues and CPUs) **************************************************************** #CPU Free Total Jobs Running Waiting ComputingElement dgt01.ui.savba.sk:2119/jobmanager-lcgpbs-long grid010.ct.infn.it:2119/jobmanager-lcgpbs-long grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-long grid004.iucc.ac.il:2119/jobmanager-lcgpbs-infinite grid010.ct.infn.it:2119/jobmanager-lcgpbs-short …… ************************************************************** These are the related data for gridit: (in terms of SE) ************************************************************** Avail Space(Kb) Used Space(Kb) Type SEs grid005.iucc.ac.il cn02.be.itu.edu.tr egee016.cnaf.infn.it lxcde07.pd.infn.it …. List of available CEs for a VO with the number of CPU and the Job in running or waiting Example of a: List of available SE for a VO with the available and used disk space Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, The Grid Technology Implementation in a GEANT4 application does not need to modify any line code First step is to install in some CE the GEANT4 libraries –There are two possible solutions:  To make the RPM of the ToolKit, like Geant4  To import, using a tar file, only the libraries that the application calls during the run Second step is to publish the TAG, that you will use as requirements in your jdl file Requirements = Member("GEANT4CT“,other.GlueHostApplicationSoftwareRunTimeEnvironment); Grid Technology Implementation (1)

Palermo, Grid Open Days all’Università di Palermo, Third step is to make a bash script to distribute the simulation work in many jobs –Generally some hundred of jobs Fourth step is to make a bash script to get output automatically, it: –verifies the job status –counts the number of job “Done” –gets the output when the number of job done crosses the threshold chosen Grid Technology Implementation (2) Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Grid Technology Implementation (3) Grid Open Day al LNS, Fifth step is to make a C++ or Matlab application to sum the single job output, a matrix outputGRID/ russo_1ahuQWRpgMw4T4EZ8NHLRw russo_2kfjVOUYhz4l_xTradovew russo_8QJeNESyu18k_zZ3QUaoMA russo_96lwgOBuEhzj2mUcVad29g russo_aKw-LcyrVLA9Xw_N-0aJTw russo_B6dXBLBU0V04uMMYxnTZMQ russo_E14M-4BBQfPy2GYgCNG7Ww russo_EPxSVrT8VELFeYcHwDA2pw russo_GCoxwYVd1JWwkAJLeby_xw russo_HaxrPZOI6Xrg4v9LKAxzRQ russo_jGHzkOKq_dTI9muJ-FewNQ russo_JHRXWY7muzpEezwk6HPY9Q ……. ###Submitted Job Ids### ….. List of Job Submitted Identify Number List of Output Directories

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Grid Problems The implementation of the Grid technology needs to change the methodology to split the work The nodes that provide CPU cycles in a Grid system have more likely to vary greatly than a cluster system A node may go down or become inaccessible without notice while it is working on its task A slow node might become the bottleneck of the whole computation Each subtask is a “key” subtask

Palermo, Grid Open Days all’Università di Palermo, N-out-of-M strategy In a Monte Carlo application we must achieve a certain accuracy We do not much care which random sample set is estimated, provided that all the random samples are independent in a statistical sense The statistical nature of Monte Carlo applications allows us to enlarge the actual size of the computation Increasing the number of subtasks from N, classic method, to M Therefore it is not necessary to wait for all M subtasks to finish M value is chosen considering the Grid performance Grid Open Day al LNS,

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Our Experience - GammaKnifeRS We have used the COMETA resources We have divided the work into 350 subtasks ( M value ) The sufficient number of subtasks to obtain the required accuracy is 336 ( N value ) We have reduced the time run from several days to 16 hours In this way

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, We have compared the Monte Carlo dose distribution with the TPS one The comparison is been done using the gamma function The tumor is covered by the isodose curve 50%. On the boundary between the water and air areas there are the greater differences The gamma index is visualized in blue levels if gamma 1 When gamma index > 1 the dose distributions are not in good agreement Results - GammaKnifeRS

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Some articles claim that when any portion of the lesion do not receive the therapeutic radio-surgical dose, tumor recurrence within 3 years is very probable * Percentage difference between the isodose 50% areas Percentage of points that do not pass the  test –18,2 % Percentage difference of volumes covered by 50% isodose curves –5,7 % This is the percentage of tumor mass that is incorrectly covered Slices near the air volume * Masahiro Shin et al., “Analysis of treatment outcome after stereotactic radiosurgery for cavernous sinus meningiomas”, J. Neurosurg. 95/2001 Results – GammaKnifeRS

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Our Experience - HadronTherapy We have used a Testbed (GILDA) The run time using 20 CPU decrease by a factor 15 A web interface has been created using “genius” interface Easy to customize

Palermo, Grid Open Days all’Università di Palermo, Grid Open Day al LNS, Sending of any Geant4 application on GRID is easy and it do not needs any code modification The Grid Technology is a good solution to reduce the run time The Grid could be a good solution to routine dose distribution validation needed in some radiotherapy treatments like Intensity-Modulated Radiation Therapy (IMRT) showed in some articles, like “Routine IMRT verification by means of an automated Monte Carlo simulation system” Int. J. Radiation Oncology Biol. Phys. Vol. 56 Conclusions

Palermo, Grid Open Days all’Università di Palermo, Any Questions ? Thank you very much for your kind attention! Grid Open Day al LNS,