1. Geant4 profiling performance for medical physics
The 22nd Geant4 Collaboration Meeting
University of Wollengong, Australia
Sep. 25~29, 2017
Geant4 profiling performance for medical physics
Kihyeon Cho and Insung Yeo
(KISTI)

2. Contents
1. Introduction 2. CPU time and Memory size 3. Profiling on Mesh size 4. Profiling on Version dependency 5. Summary

3. 1. Introduction

4. Introduction Why profiling? Diversity of physics applications
Evolving Computing
S/W development
Profiling tools
To draw community interests for collateral effort

5. There are many efforts to develop HEP software to save CPU Time and Memory.(
However, there are not enough codes for profiling system.
⇒ To use FNAL Geant4 computing performance profiling protocol and tools as a starting point

6. Current status High energy physics profiling (Fermilab)
SimpliCarlo (Sequential)
CMSExp (Multi-Thread)
Low energy physics profiling
Using Brachytherapy code

7. Brachytherapy
Using Brachytherapy simulation in advanced examples

8. Goal on Brachytherapy Low Energy Physics Profiling CPU/Memory Usage
Mesh Size
Version dependency
Scalability for new computing architecture (KISTI supercomputer)

9. Bench mark @ current machine
The 4th KISTI
Rpeak
300TFlops
Nodes Number
3,176
Processor
Intel Xeon X GHz
CPU Number
25,408
Memory
76.8 TB
Storage
1,061TB
Service date

10. Scalability to Future Hybrid machine
The 5th KISTI
100 X Scalability ?
Mini component supercomputer? 1PFlops
Rpeak
25.7PFlops
Price
54,700,000,000won ($50million)
Many core Processor
Intel Xeon Phi Knight landing processing
Multicore CPU
Intel Xeon Skylake process
Service date
1Q, 2018
Company
Cray

11. Methodology

12. Output of igprof and sprof
Machines
Machines OS
batch jobs
Input file
outputfile
Install Geant4
tachyon2
Linux
PBS -
Compile
Brachytherapy code
Igprof Run
kumac
IgProf_iodine.FTFP_INCLXX_HP.35.0_MEM_LIVE_1.txt
Sprof Run
g4profiling_1_X.tgz
Draw plot
hepkisti
Output of igprof and sprof
png
Web site
html

13. SimpliCarlo vs. Brachytherapy
Energy
More than TeV
Less than MeV
CPU time
Long
Too short
CPU time / event (sec)
0.01
0.0002
Snap shot time 30 5
Institute
Fermilab
KISTI

14. Physics List Name Content Energy QGS Quark Gluon String Model
> ~20 GeV
FTF
Fritiof string model
> ~ 5GeV
BIC
Binary Cascade
< ~ 10GeV
BERT
Bertini-style cascade
< ~10GeV HP
High Precision neutron model
>20MeV P
G4Precompund model used for
de-excitation

15. Results CPU time and Memory size Profiling on Mesh size
Profiling on Version dependency

16. 2. CPU time and Memory size

17. Settings Geant4 Version: 10.02 Brachytherapy Version: 10.02 Macro file
IodiumSourceMacro.mac (endocavitary brachytherapy)
IridiumSourceMacro.mac (interstitial brachytherapy)
LeipzigSourceMacro.mac (superficial brachytherapy)
Physics list
QGSP_BIC_LIV (Suggested by Sebastien)
QGSP_BIC_EMZ
QGSP_BIC_EMY

18. CPU Time in seconds/event
Sample
Physics List
Energy
Process
Iodine
QGSP_BIC_LIV
35 KeV
0.0026
QGSP_BIC_EMZ
0.0025
QGSP_BIC_EMY
Iridium
356 KeV
0.0092
0.0093
Leipzig
0.0031
0.0032
<Geant Brachy 01>
<Trial time : 528>
<Events : 10001>

19. Leipzig
QGSP_BIC_LIV

20. Total Memory in Counts/10,000
Memory Size
Total Memory in Counts/10,000
Sample
Physics List
Energy
Process
First Event
Last Event
Iodine
QGSP_BIC_LIV
35 KeV
QGSP_BIC_EMZ
QGSP_BIC_EMY
Iridium
356 KeV
146.72
Leipzig
163.89
<Geant Brachy 01>
<Trial time : 528>
<Events : 10001>

21. Leipzig
QGSP_BIC_LIV

22. Leipzig
QGSP_BIC_LIV

23. 3. Profiling on Mesh size

24. Settings Geant4 Version: 10.02 Brachytherapy Version: 10.02 Macro file
IodiumSourceMacro.mac (endocavitary brachytherapy)
IridiumSourceMacro.mac (interstitial brachytherapy)
LeipzigSourceMacro.mac (superficial brachytherapy)
Physics list
QGSP_BIC_LIV (Suggested by Sebastien)
QGSP_BIC_EMZ
QGSP_BIC_EMY

25. Comparing Profiling Mesh information Number of events: 10,000
CPU time: total sec
Trial time: 8
Mesh size: 30, 300, 900, 1500, 3000, 6000
Snap shot time: 5 sec

26. <Processing time>
Iodine
Iodine
QGSP BIC EMY
Chi = e-08
NDf =
p = / e-05
p = e / e-08
****************************************
QGSP BIC EMZ
Chi = e-08
p = / e-05
p = e / e-08
QGSP BIC LIV
Chi = e-08
p = / e-05
p = e / e-08
<Processing time> 30
300
900
1500
3000
6000 25
(min.) 35 40 55 85

27. Iodine QGSP BIC EMY Chi2 = 0.000122147 NDf = 4
p = / e-06
QGSP BIC EMZ
Chi =
p = /
p = / e-06
QGSP BIC LIV
p = /
Iodine
QGSP BIC EMY
Chi =
NDf =
p = /
p = / e-05
QGSP BIC EMZ
Chi =
p = /
p = / e-05
QGSP BIC LIV
Chi =
p = /
p = / e-05

28. Iridium Iridium QGSP BIC EMY Chi2 = 2.57939e-07 NDf = 4
p = e / e-08
****************************************
QGSP BIC EMZ
Chi = e-05
p = /
p = e / e-07
QGSP BIC LIV
Chi = e-05
p = /
p = e / e-07

29. Iridium QGSP BIC EMY Chi2 = 0.000116022 NDf = 4
p = / e-06
QGSP BIC EMZ
Chi =
p = /
p = / e-06
QGSP BIC LIV
Chi =
p = /
p = / e-06
Iridium
QGSP BIC EMY
Chi =
NDf =
p = /
p = /
QGSP BIC EMZ
Chi =
p = /
p = /
QGSP BIC LIV
Chi =
p = /
p = /

30. Leipzig Leipzig QGSP BIC EMY Chi2 = 7.64365e-07 NDf = 4
p = e / e-08
****************************************
QGSP BIC EMZ
Chi = e-07
p = /
p = e / e-08
QGSP BIC LIV
Chi = e-07
p = /
p = e / e-08

31. Leipzig ⇒ Linearity QGSP BIC EMY Chi2 = 0.00011575 NDf = 4
p = / e-06
QGSP BIC EMZ
Chi =
p = /
p = / e-06
QGSP BIC LIV
p = /
Leipzig
QGSP BIC EMY
Chi =
NDf =
p = /
p = / e-05
QGSP BIC EMZ
Chi =
p = /
p = / e-05
QGSP BIC LIV
Chi =
p = /
p = / e-05
⇒ Linearity

32. 4. Profiling on Version dependency

33. Settings Trial time: 528 Macro file Physics list
Content
Profiling
Reference
Geant4 Version
10.02
10.03
10.03p02
Brachy version
Trial time: 528
Macro file
IodiumSourceMacro.mac (endocavitary brachytherapy)
IridiumSourceMacro.mac (interstitial brachytherapy)
LeipzigSourceMacro.mac (superficial brachytherapy)
Physics list
QGSP_BIC_LIV
QGSP_BIC_EMZ
QGSP_BIC_EMY

34. CPU Time
QGSP_BIC_EMY

35. QGSP_BIC_EMY
Total Memory
First event
Last event
Brachy v10.2
Brachy v10.3
Brachy v10.2
Brachy v10.3
Version have new physics features such as pre-compound/
de-excitation physics parameters de-excitation parameter data.

36. <Trial time : 528> Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02
10,000
10,000
10,000

37. <Trial time : 528>
Geant4 version : / Brachy version: 10.03
10,000

38. QGSP_BIC_EMY Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02

39. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

40. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

41. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

42. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

43. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

44. Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02
G4Navigater::LocateGlobalPointAndSeup.C

45. Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02
G4Navigater::LocateGlobalPointAndSeup.C

46. Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02
libG4processes.so

47. Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02
G4RunManager::ProcessOneEvent(int)

48. 5. Summary

49. Summary Shown Future Low energy physics profiling code
CPU/Memory usage
Profiling on mesh size ⇒ Linearity
Profiling on version dependency ⇒ Geant4.10.3?
Future
To usage: Geant4 profiling test ⇒ G4 Q&A
Scalability for new computing architecture
KISTI supercomputer

50. Acknowledgement
Soon Yung Jun
Susanna Guatelli

51. Thank you!

52. Backup

53. QGSP_BIC_EMY Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02

54. QGSP_BIC_EMY Geant4 version : 10.02 Geant4 version : 10.03
Geant4 version : p02

55. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

56. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

57. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

58. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

59. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

60. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

61. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

62. Geant4 version : 10.02
Geant4 version : 10.03
Geant4 version : p02

63. Brachytherapy Comparison
g p01 sequential No
G mt
Yes G
sequential
G mt/sequential
G p01 sequential