Parallel 3D Finite Element Particle-In-Cell Simulations with Pic3P* Arno Candel, Andreas Kabel, Zenghai Li, Cho Ng, Liequan Lee, Greg Schussman, Kwok Ko Advanced Computations Department, SLAC Ilan Ben-Zvi, Jorg Kewisch, BNL Abstract SLAC's Advanced Computations Department (ACD) has developed the parallel 3D Finite Element electromagnetic Particle-In-Cell code Pic3P. Designed for simulations of beam-cavity interactions dominated by space charge effects, Pic3P solves the complete set of Maxwell-Lorentz equations self-consistently and includes space-charge, retardation and boundary effects from first principles. Higher-order Finite Element methods with adaptive refinement on conformal unstructured meshes lead to highly efficient use of computational resources. Massively parallel processing with dynamic load balancing enables large-scale modeling of photoinjectors with unprecedented accuracy, aiding the design and operation of next-generation accelerator facilities. Applications include the LCLS RF gun and the BNL polarized SRF gun. Overview Causal Moving Window SLAC is focusing on the modeling and simulation of DOE accelerators using high-performance computing The performance of high-brightness RF guns operating at the space-charge limit is affected by retardation and wakefield effects which are omitted in standard static tracking codes Conventional PIC codes based on Finite Difference methods have problems dealing with complex geometries, and are restricted to low particle-field coupling accuracy SLAC has developed the first parallel 3D PIC code Pic3P that uses higher-order Finite Element methods on conformal unstructured meshes for unprecedented modeling accuracy 2+1D spacetime diagram indicating the causal field domain for an accelerated particle bunch inside the gun at a snapshot in time. Fields outside this causal domain can be neglected for the gun simulations; this leads to a significant speedup. LCLS RF Gun 1.6 cell, 2.856 GHz, 120 MV/m, 5.8 MeV, copper cathode, elliptical iris, high cylindrical symmetry of operating mode with dual-feed race track design, measured initial distribution, Q=1nC Pic3P – Emittance Calculations for the LCLS RF gun and the BNL polarized SRF gun ~ 20 ps Initial spot size r ≈ 1 mm Laser time profile Parallel 3D Finite Element PIC Code Pic3P Dynamic Load Balancing Pic3P includes space charge, wakefield and retardation effects from first principles Pic3P is supported by SciDAC and designed for massively parallel operation on leadership-class supercomputers Pic3P solves the full set of Maxwell-Lorentz equations self-consistently in time domain with unconditional stability of the field solver in the time step Field Partitioning Particle Partitioning Measured initial distribution at the cathode and laser profile (Courtesy LCLS Commissioning Team) Realistic Particle Distributions Pic3P obtains higher-order particle-field coupling accuracy by using higher-order Finite Element vector basis functions Ni LCLS RF Gun: Pic3P 3D emittance results for a cylindrical bunch agree perfectly with 2D results obtained with Pic2P and MAFIA. PARMELA results differ since retardation and wakefield effects are neglected in the electrostatic model. N1 N2 For order p=2: 20 different Ni’s For order p=6: 216 different Ni’s curved tetrahedral element Particle phase space at exit of the LCLS RF gun as calculated by Pic3P, starting from measured initial particle distribution shown above. Snapshot of causal moving window PIC simulation of the LCLS RF gun. Colors indicate parallel partitioning of fields and particles to processors. Dynamic load balancing enables the solution of large problems for unprecedented modeling accuracy. BNL polarized SRF Gun ½ cell, 350 MHz, 24.5 MV/m, 5 MeV, 7 Gauss solenoid, recessed GaAs cathode (~70 K) inserted via choke joint, cathode spot size 6.5 mm, Q=3.2 nC, initially ellipsoidal bunch The operating mode of a RF gun is calculated with ACD’s parallel frequency-domain Finite Element code Omega3P to higher accuracy than machining tolerances and can be directly loaded into Pic3P for optimum field quality Summary SLAC is focusing on the modeling and simulation of DOE accelerators using high-performance computing The Advanced Computations Department has developed the first parallel Finite Element PIC code Pic3P for self-consistent simulations of space-charge dominated beam-cavity interactions Pic3P has been extensively benchmarked and delivers state-of-the-art modeling accuracy for simulations of next-generation accelerator structures Pic3P has been applied to calculate emittance growth effects in the LCLS RF gun and the BNL SRF gun Close-up of mesh in cathode region Bunch distribution at gun exit, colored by energy Left: Unstructured mesh model of the LCLS RF gun Right: Drive fields and particles on symmetry plane Bunch transit through the BNL polarized SRF gun as simulated with Pic3P, scattered self-fields are shown *This work was supported by DOE Contract No. DE-AC02-76SF00515 and used resources of NERSC supported by DOE Contract No. DE-AC02-05CH11231, and of NCCS supported by DOE Contract No. DE-AC05-00OR22725.