1. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney aka The Full Monte! Optimisation of Monte Carlo codes for High Performance Computing in Radiotherapy Applications Dr Iwan Cornelius, M.B. Flegg, C.M. Poole, Prof Christian Langton Faculty of Science and Technology Queensland University of Technology Queensland Cancer Physics Collaborative

2. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Outline Introduction Development of a LINAC Monte Carlo model using GEANT4 Optimisation Future Directions Conclusions

3. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Introduction: Radiotherapy LINAC: produce highly controllable source of MeV photons –Energy –Gantry angle –Patient position

4. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Introduction: Radiotherapy LINAC: produce highly controllable source of MeV photons –Multi Leaf Collimators (MLCs) to define arbitrary shaped fields

5. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Introduction: Radiotherapy Planning –Patient imaged –PTV OAR Contoured –Optimisation of fields to conform Dose to tumour and spare healthy tissue Delivery –Fractionated Based on analytical calculations –Can be inaccurate in regions of high heterogeneity

6. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Monte Carlo What is it? How is it used in radiotherapy? –Treatment plan verification –Support new dosimetry measurements used in QA What tools exist? –EGSnrc/BEAMnrc, PENELOPE, MCNPX, GEANT4 Challenges to overcome –Reduce Computation times (maintain accuracy) Code optimisation Variance reduction High Performance Computing (HPC) –Usability

7. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney High Performance Computing Monte Carlo: trivial to parallelise –Launch identical application with unique random number generator seed –Collate results Centralised Clusters –Multiple machines, Beowulf –Multiple CPU, Shared memory (SGI Altix) Cons –Look better on paper –Sharing resource with other users –Often limited to # of processors, wait in queue Single machine, multiple processors –Dual quad core –Hyperthreading can get 16 cores

8. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney High Performance Computing: GPGPU General Purpose Graphics Processing Units –hundreds of processors on a chip –NVIDIA Tesla C1060: PCIx 240 cores per card 4GB memory CUDA –Compute Unified Device Architecture –Write ‘kernel’ in ‘C for CUDA’ to run on the GPU –Copy from main memory to device memory –Kernel executes on GPU –Copies result back to main memory –Great for loops How to ‘Accelerate’ Monte Carlo codes with GPUs –Re-engineer entire code into C for CUDA kernels –Re-write computationally intensive portions of code into ‘kernels’ using CUDA –Calculation time doesn’t scale with # of processors

9. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney GEANT4 Toolkit of C++ classes –Primary beam, geometry, physics processes, scoring –User must create their own application based on these Very powerful general purpose Monte Carlo tool –High energy physics, space physics, medical physics, optics, radiation protection, astrophysics

10. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney GEANT4 Pros –Extremely flexible –Time dependent geometries –Radioactive decay, Neutron transport –Various visualisation tools Cons –Extremely flexible –Requires proficiency with C++ programming –Steep learning curve –Deterrent for first time users –Hospital based Medical Physicists with limited research time

11. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney The Full Monte! Create generic LINAC application using GEANT4 –Capable of modelling Elekta, Varian, Siemens LINACs –Do for GEANT4 what BEAMnrc did for EGSnrc (just text inputs) –Accurate. Verify against experimental data. Optimise for HPC environments (Desktop Supercomputer) –Distribute over available CPUs –Port to the GPU User interface –Simple text-file based interface –Graphical User Interface Interface with TPS –Able to routinely verify treatment plans

12. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Geometry Varian 2100 Clinac –Dimensions, material composition from Varian Docs Target Primary Collimator Vacuum window

13. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Geometry Flattening filter –Compensate for forward peaked distribution of bremsstrahlung photons Ionisation chamber –Monitor total Dose delivery

14. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Geometry Jaws –Define square fields

15. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Geometry Multi-Leaf Collimators (MLCs) –Interleaved Tungsten leaves –Varian Millenium –Brad Oborn (UoW)

16. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Primary Beam Monoenergetic electron beam Normally incident on target Gaussian spread radially

17. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Physics Photons –Photoelectric effect –Compton –GammaConversion Electrons –Multiple scatter –Ionisation –Bremmstrahlung Positrons –Ditto –Annihilation

18. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Scoring Water Phantom –50 cm x 50 cm x 50 cm –Score in voxelised geometry

19. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Validation / Commissioning Comparison with ionisation chamber measurements in a water phantom –Scanning with x,y,z Dose along beam axis

20. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Validation: Tune Electron Beam Energy Tuning of electron beam energy for best match –10 cm x 10 cm field –Compare between –10-30cm depths

21. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Results: Tune Electron Beam Energy Comparison with ionisation chamber measurements in water Tuning of electron beam energy for best match –10 cm x 10 cm field –Compare between –10-30cm depths

22. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Results: 5.85 MeV, 10 cm x 10 cm Within 2% agreement between 0.5cm and 38cm

23. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Results: 5.85 MeV, 10 cm x 10 cm Within 2% agreement between 0.5cm and 38cm

24. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Results: 5.85 MeV, 5 cm x 5 cm

25. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Results: 5.85 MeV, 20 cm x 20 cm

26. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Results: 5.85 MeV, 40 cm x 40 cm

27. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Optimisations No Optimisation –Many photons produced will never reach the sensitive region of the geometry

28. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Optimisations Kill zones –Nothing fancy-pants –Terminate histories that are unlikely to contribute to observable –Above target –Around primary collimator Relative Computation Time: 78 %

29. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Optimisations Phase space files –Some aspects of geometry don’t change –Create pre-calculated radiation field at plane –Sample this population to conserve computation times Relative Computation Time: 38 % 380 hrs, O(10 10 )

30. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney HPC: GPU/CPU Desktop Supercomputer Purchase of Xenon T5 Desktop Supercomputer –“The Terminator” –4 x C1060 Tesla card = 960 cores! –2 x quad core processors hyper-threading Linux ‘sees’ 16 processors NVIDIA Professorial partnership grant –Awarded 3 x C1060 Tesla cards Research team learning CUDA –Mark Harris, local CUDA guru

31. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Optimisations: Parallelise on CPUs Message Passing Interface (MPI) –Run identical simulation on different core with unique random number –Geant4 MPImanager class –Time scales roughly linearly with number of processors –Simulations in 24 hrs, O(10 10 )

32. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney The GPU Dilemma 1. Re-write entire code into C for CUDA? –C for CUDA doesn’t support sophisticated data types (classes) – O(10^6) lines of code, dozens of developers –Wait for CUDA to catch up (?) 2. Create C++ wrapper classes for certain methods –First step, random number generator –Incorporated into GEANT4 framework via inheritance –Implementing Mersenne Twister algorithm (hack example from CUDA SDK) to generate cache of random numbers –Improvement of only a few percent

33. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Profiling! Great first step when optimising code Linux gprof require to re-compile with flags set MacOSX –Profiling tool doesn’t require recompile

34. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Conclusions GEANT4 LINAC application has been developed –Specific to Varian Clinac –Many parameters hard-coded –Work commenced on textfile based UI commands –Preliminary validation promising Optimisation –Phase space files –Kill zones –MPI for parallel processing on CPUs –Porting random number generator to GPU

35. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Future Directions Validation –Verify dose distributions in heterogeneous phantoms –Verify model of MLCs (irregular fields) –Develop interface to Treatment Planning System Optimisation –Re-write part of GEANT4 to run on GPU Interface –User friendly text-file based commands Treatment Plan interface –Implement DICOM-RT interface

36. Queensland University of Technology CRICOS No. 00213J AstroMed09, 14-16th December, The University of Sydney Acknowledgements QUT –Scott Crowe, Tanya Kairn, Andrew Fielding discussion on Varian LINAC model, Experimental data –Mark Barry, Mark O Dwyer discussion on CPU optimisation, High Performance Computing Mater Hospital, Brisbane –Radiation Oncology Group UoW –Brad Oborn Millenium MLC model GEANT4 Collaboration –Joseph Perl (SLAC) discussion on visualisation / profiling NVIDIA –Mark Harris