1. Parallel 3D Finite Element Particle-In-Cell Simulations with Pic3P*
Arno Candel, Andreas Kabel, Zenghai Li, Cho Ng, Liequan Lee, Greg Schussman, Kwok Ko
Advanced Computations Department, SLAC
Ilan Ben-Zvi, Jorg Kewisch, BNL
Abstract
SLAC's Advanced Computations Department (ACD) has developed the parallel 3D Finite Element electromagnetic Particle-In-Cell code Pic3P. Designed for simulations of beam-cavity interactions dominated by space charge effects, Pic3P solves the complete set of Maxwell-Lorentz equations self-consistently and includes space-charge, retardation and boundary effects from first principles. Higher-order Finite Element methods with adaptive refinement on conformal unstructured meshes lead to highly efficient use of computational resources. Massively parallel processing with dynamic load balancing enables large-scale modeling of photoinjectors with unprecedented accuracy, aiding the design and operation of next-generation accelerator facilities. Applications include the LCLS RF gun and the BNL polarized SRF gun.
Overview
Causal Moving Window
SLAC is focusing on the modeling and simulation of DOE accelerators using high-performance computing
The performance of high-brightness RF guns operating at the space-charge limit is affected by retardation and wakefield effects which are omitted in standard static tracking codes
Conventional PIC codes based on Finite Difference methods have problems dealing with complex geometries, and are restricted to low particle-field coupling accuracy
SLAC has developed the first parallel 3D PIC code Pic3P that uses higher-order Finite Element methods on conformal unstructured meshes for unprecedented modeling accuracy
2+1D spacetime diagram indicating the causal field domain for an accelerated particle bunch inside the gun at a snapshot in time. Fields outside this causal domain can be neglected for the gun simulations; this leads to a significant speedup.
LCLS RF Gun
1.6 cell, GHz, 120 MV/m, 5.8 MeV, copper cathode, elliptical iris, high cylindrical symmetry of operating mode with dual-feed race track design, measured initial distribution, Q=1nC
Pic3P – Emittance Calculations for the
LCLS RF gun and the BNL polarized SRF gun
~ 20 ps
Initial spot size r ≈ 1 mm
Laser time profile
Parallel 3D Finite Element PIC Code Pic3P
Dynamic Load Balancing
Pic3P includes space charge, wakefield and retardation effects from first principles
Pic3P is supported by SciDAC and designed for massively parallel operation on leadership-class supercomputers
Pic3P solves the full set of Maxwell-Lorentz equations self-consistently in time domain with unconditional stability of the field solver in the time step
Field Partitioning
Particle Partitioning
Measured initial distribution at the cathode and laser profile (Courtesy LCLS Commissioning Team)
Realistic Particle Distributions
Pic3P obtains higher-order particle-field coupling accuracy by using higher-order Finite Element vector basis functions Ni
LCLS RF Gun: Pic3P 3D emittance results for a cylindrical bunch agree perfectly with 2D results obtained with Pic2P and MAFIA. PARMELA results differ since retardation and wakefield effects are neglected in the electrostatic model. N1 N2
For order p=2: 20 different Ni’s
For order p=6: 216 different Ni’s
curved tetrahedral element
Particle phase space at exit of the LCLS RF gun as calculated by Pic3P, starting from measured initial particle distribution shown above.
Snapshot of causal moving window PIC simulation of the LCLS RF gun. Colors indicate parallel partitioning of fields and particles to processors. Dynamic load balancing enables the solution of large problems for unprecedented modeling accuracy.
BNL polarized SRF Gun
½ cell, 350 MHz, 24.5 MV/m, 5 MeV, 7 Gauss solenoid, recessed GaAs cathode (~70 K) inserted via choke joint, cathode spot size 6.5 mm, Q=3.2 nC, initially ellipsoidal bunch
The operating mode of a RF gun is calculated with ACD’s parallel frequency-domain Finite Element code Omega3P to higher accuracy than machining tolerances and can be directly loaded into Pic3P for optimum field quality
Summary
SLAC is focusing on the modeling and simulation of DOE accelerators using high-performance computing
The Advanced Computations Department has developed the first parallel Finite Element PIC code Pic3P for self-consistent simulations of space-charge dominated beam-cavity interactions
Pic3P has been extensively benchmarked and delivers state-of-the-art modeling accuracy for simulations of next-generation accelerator structures
Pic3P has been applied to calculate emittance growth effects in the LCLS RF gun and the BNL SRF gun
Close-up of mesh in cathode region
Bunch distribution at gun exit, colored by energy
Left: Unstructured mesh model of the LCLS RF gun
Right: Drive fields and particles on symmetry plane
Bunch transit through the BNL polarized SRF gun as simulated with Pic3P, scattered self-fields are shown
*This work was supported by DOE Contract No. DE-AC02-76SF00515 and used resources of NERSC supported by DOE Contract No. DE-AC02-05CH11231, and of NCCS supported by DOE Contract No. DE-AC05-00OR22725.