SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 1 Vay - APS-DPP 2011 Novel Simulation Methods in the Particle-In-Cell Framework Warp J.-L. Vay*, C.G.R. Geddes Lawrence Berkeley National Laboratory, CA, USA D.P. Grote, A. Friedman Lawrence Livermore National Laboratory, CA, USA 53 rd Annual Meeting of the APS Division of Plasma Physics Salt Lake City, Utah, USA – November 14-18, 2011 SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory Markers 2 Vay - APS-DPP 2011 Warp is a versatile parallel 3D Particle-In-Cell framework developed by the Heavy Ion Fusion Science Virtual National Laboratory Standard PIC Non-standard PIC Laboratory frameMoving windowLorentz Boosted frame Examples: Beam generation Neutralization in plasma Example: Beam transport Example: Laser plasma acceleration Steady flow Example: Fast injector design Quasi-static Example: electron cloud studies 2-D slab of electrons 3-D beam s SPS - CERN p+ bunches e- clouds

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory But short wavelength instability observed at front of plasma for large  100) 3 Vay - APS-DPP 2011 Conjectured that instability related to numerical dispersion, i.e. a kind of numerical Cerenkov. Warp 2D simulation 10 GeV LPA (n e =10 17 cc,  =130) Longitudinal electric field laser plasma Modeling of 10 GeV laser plasma accelerator stage is challenging 1 Vay, PRL 2007 laser  wake  L accelerate laser << wake << L accelerate Predicted speedup: >10,000 for 10 GeV stage; > 1,000,000 for 1 TeV stage. Boosted frame   wake Lab frame Calculation in boosted frame at  ≈  wake minimizes scale differences 1

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory An electromagnetic solver based on Non-Standard Finite-Difference (NSFD) was implemented in Warp NSFD: weighted average of quantities transverse to FD (  ). NSFD=FD if   =  0 YeeCole-Karkkainen (CK) Yee/CK allows for perfect dispersion along 3D/principal axes.  x=  y=  z) Cole 1 and Karkkainen 2 have applied NSFD to source free Maxwell equations Warp 3 : switched FD/NSFD to B/E. => FD on source terms, i.e. standard exact current deposition schemes still valid. * FD NSFD                   xx FD NSFD 3 J.-L. Vay, et al., J. Comput. Phys. 230 (2011)  x=  y=  z) Vay - APS-DPP J. B. Cole, IEEE Trans. Microw. Theory Tech. 45 (1997), J. B. Cole, IEEE Trans. Antennas Prop. 50 (2002). 2 M. Karkkainen et al., Proc. ICAP, Chamonix, France (2006). NSFD offers tunability of numerical dispersion perfect dispersion 2D diagonal isotropic

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory Example: Reflection of circular pulse using 5 cells PML with quadratic progression and standard coefficients or improved coefficients 2 5 Perfectly Matched Layer 1,2 (PML) implemented with NSFD solver - for absorption of outgoing waves - Same high efficiency as with Yee. 1 JP Berenger, J. Comput. Phys. 127 (1996) J.-L. Vay, J. Comput. Phys. 183 (2002) 367 NSFD FD Vay - APS-DPP 2011

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory After testing: instability mostly insensitive to numerical dispersion… …but very sensitive to time step! 6 Vay - APS-DPP 2011 Sharp decrease of instability level at c  t=  z/√2  Tunable NSFD solver allows c  t=  z/√2 time step for (near) cubic cells c  t=  z/√2 time step restricted to “pancake” cells in 3D using Yee FDTD solver  Use of special time step was helpful but not sufficient for large  boost Power spectrum (a.u.)

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 7 Vay - APS-DPP 2011 Digital filtering of current density and/or fields -- commonly used for improving stability and accuracy Multiple pass of bilinear filter + compensation routinely used 100% absorption at Nyquist freq. Bilinear (B) Bilinear (B) + compensation (C) 1/2 1/4 Bilinear filter Wideband filtering difficult in parallel (footprint limited by size of local domains) or expensive Example: wideband filters using N repetitions of bilinear filter 1×B + C 4×B + C 20×B + C 50×B + C 80×B + C

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 8 “Strided” bilinear filters enable efficient and versatile filtering 1 4×BC stride 1 (G1) 4×BC stride 2 (G2) 4×BC stride 3 (G3) 4×BC stride 4 (G4) Bilinear filter with stride 2 1/21/4 Using a stride N shifts the 100% absorption frequency to F nyquist /N Combination of filters with strides allows for more efficient filtering: G 1 G 2  20*B+C; speedup ×2 G 1 G 2 G 3  50*B+C; speedup ×3.5 G 1 G 2 G 4  80*B+C; speedup ×5.5 G1 × G2 G1 × G2 × G3 G1 × G2 × G4 20×B+C 50×B+C 80×B+C Vay - APS-DPP J.-L. Vay, et al., J. Comput. Phys. 230 (2011) Nice, but is wideband filtering possible without altering the physics?

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 9 Time Laser field Lab frame Wake frame Hyperbolic rotation from Lorentz Transformation converts laser… …spatial oscillations into time beating Vay - APS-DPP 2011

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 10 Vay - APS-DPP 2011 Lab frame Frame of wake (  =130) spectrum Spectrum very different in boosted and lab frames Time history of laser spectrum (relative to laser 0 in vacuum) Dephasing time Content concentrated around 0 00 Content concentrated at much larger More filtering possible without altering physics*. *J.-L. Vay, et al., PoP Lett. 18 (2011).

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 11 Vay - APS-DPP 2011 Controlling the numerical instability with tunable EM solver & filtering Laser injection Particle injection Diagnostics led to over 1 million x speedup 2 + new laser/particle injection and diagnostics through planes 1 : 2 J.-L. Vay, et al., PoP Lett. 18 (2011) & PoP (in press). 1 J.-L. Vay, et al., J. Comput. Phys. 230 (2011).

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 12 Vay – APS-DPP 2011  A. Friedman, D. P. Grote, and I. Haber, Phys. Fluids B 4, 2203 (1992) – code description, warped coordinates  J.-L. Vay et al., Phys. Plasmas 11 (2004) – mesh refinement  D.P. Grote et al., AIP Conf. Proc. 749, 55 (2005) – updated Warp description  R. Cohen et. al., Phys. Plasmas 12, (2005) – “Drift-Lorentz” particle pusher  J.-L. Vay, Phys. Plasmas 15, (2008) – ultra-relativistic pusher  R. Cohen et al. Nucl. Instr. & Methods 608, 53 (2009) – direct implicit Drift-Lorentz  For questions on Warp, to Warp contains a lot more not described here, see

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory BACKUP 13 Vay - APS-DPP 2011

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory 14 as well as Friedman damping algorithm - for noise control - B push modified to with where  is damping parameter. Yee-Friedman (YF)Cole-Karkkainen-Friedman (CKF) Dispersion degrades with higher values of  Damping more potent on axis and more isotropic for CKF than YF. Vay - APS-DPP 2011

SciDAC-II Compass SciDAC-II Compass The Heavy Ion Fusion Science Virtual National Laboratory Vay - APS-DPP 2011 BELLA Project: state-of-the-art PWfacility for laser accelerator science BELLA Laser Control Room Gowning Room Final focus < 100 cm e - beam ~10 GeV Laser Plasma 15