1. DOE/HEP SciDAC AST Project: “Advanced Computing for 21 st Century Accelerator Science and Technology” Impact of SciDAC on Accelerator Projects Across SC through Electromagnetic Modeling Kwok Ko Stanford Linear Accelerator Center * Work supported by U.S. DOE ASCR & HEP Divisions under contract DE-AC02-76SF00515 Rich Lee - Achievements in ISIC/SAPP Collaborations for Electromagnetic Modeling of Accelerators (Poster)

2. DOE Office of Science Accelerators LCLS PEP-II HEP ILC NP CEBAF Accelerators are essential tools for doing science in SC - close to 50% of the Facilities for the Future of Science involve accelerators BES LCLS PEP II ILC RIA

3. ILC LCLS Electromagnetic Structures in Accelerators  High resolution modeling and end-to-end simulation are being used to improve existing accelerators (e.g. PEP-II) and design future facilities (e.g. ILC).  Due to the large complex geometry and required accuracy Large-scale computation is absolutely essential FEL

4. Electromagnetic Modeling NLC Cell Design  CAD/Meshing PartitioningSolversAnalysis SciDAC has enabled collaborations with the ISICs and SAPP to advance the computational science needed for Large-scale electromagnetic modeling. Refinement Optimization Visualization Performance

5. AST- Electromagnetics Project Simulation and Modeling Computational Science Parallel Code Development H High Performance Computing (NERSC, ORNL) Accelerators SLAC DESY KEK Jlab ANL MIT PSI SLAC Accelerator Modeling Computational Mathematics Computing Technologies ISICs/SAPP LBNL LLNL SNL Stanford, UCD RPI, CMU Columbia UWisconsin SciDAC

6. UCD K. Ma, H. Yu Z. Bai AST Electromagnetics Team RPI M. Shephard, A. Brewer, E. Seol Stanford G. Golub SNL P. Knupp, K. Devine. L. Fisk, J. Kraftcheck LBNL E. Ng, W. Gao, X. Li, C, Yang P. Husbands, A. Pinar, D. Bailey, D. Gunter LLNL L. Diachin, D. Brown, D. Quinlan, R. Vuduc ISICs (TSTT, TOPS, PERC) and SAPP Columbia D. Keyes CMU O. Ghattas V. Akcelik Computational Mathematics L. Lee, L. Ge, E. Prudencio, S. Chen (Stanford), Accelerator Modeling K. Ko, V. Ivanov, A. Kabel, Z. Li, C. Ng,, L. Xiao, A. Candel (PSI) Computing Technologies N. Folwell, G. Schussman, R. Uplenchwar, A. Guetz (Stanford) SLAC/ACD UWisconsin T. Tautges, H. Kim,

7. Parallel EM Code Development Solve Maxwell’s equations in time & frequency domains using unstructured grid and parallel computing Generalized Yee Grid Omega3P Tau3P/T3P S3P Time Domain Simulation With Excitations Frequency Domain Mode Calculation Scattering Matrix Evaluation Finite-Element Discretization Track3P – Particle Tracking with Surface Physics V3D – Visualization/Animation of Meshes, Particles & Fields

8. Achievements in Accelerator Science/Design  PEP-II - IR Heating  - Vertex Bellows Heating  NLC – DDS Cell Design  - DDS Wakefields  - Dark Current Pulse  RIA - RFQ Design and AMR  LCLS - RF Gun Design

9. HEP PEP-II - IR Heating Omega3P/Tau3P calculated heat power distribution to help in the IR upgrade design to reduce beam heating TSTT generated good quality hexahedral meshes to enable Tau3P simulation with beam Power Distribution Total power = 17.2 kW for 3A (330 Modes) 15% higher beam current since upgrade led to higher luminosity/physics discovery PEP-II IR

10. Omega3P was used to evaluate the damping of localized modes by mounting ceramic tiles on the bellows convolution. Bellows modes were found to be damped to very low Qs Bellows Modes Bellows mode Ceramic tile absorber Dielectric loss HEP PEP-II – Vertex Bellows Heating

11. HEP NLC - DDS Cell Design +1MH z -1MHz Omega3P provided the dimensions for 206 NLC Damped Detuned Structure cells. Microwave QC of cells verified frequency accuracy of 1 part in 10,000 as targeted. SLAC/Stanford/LBL (SAPP/TOPS) developed faster and more scalable eigensolvers (ISIL & ESIL)  Potential savings of $100 million+ in NLC machine cost since DDS is 14% more efficient than standard cell design  New core accelerator design capability

12. Omega3P: Sum over eigenmodes Omega3P/Tau3P computed the long-range wakefields in the entire 55-cell DDS to assess the NLC baseline design in wakefield suppression. SLAC/SNL/LBL (SAPP) improved Tau3P speedup with alternate partitioning tools from Zoltan HEP NLC - DDS Wakefields Tau3P: Direct beam excitation

13. HEP NLC - Wakefields Comparison Omega3P Tau3P Mode Spectrum Wakefields behind leading bunch  1 st ever wakefield analysis of an actual DDS prototype  Provided benchmarking of Omega3P and Tau3P

14. Dark current @ 3 pulse risetimes Track3P Data -- 10 nsec -- 15 nsec -- 20 nsec Dark current pulses were simulated for the 1 st time in a 30- cell X-band structure with Track3P and compared with data. Simulation shows increase in dark current during pulse risetime due to field enhancement from dispersive effects. Track3P: Dark current simulation Red – Primary particles, Green – Secondary particles Track3P: Dark current simulation HEP NLC – Dark Current Pulse

15. NP RIA - RFQ Design Omega3P was used to model RIA’s low energy RFQ. With Adaptive Meshing Refinement accuracy in frequency and wall loss calculations improved by a factor of 10 and 2 respectively while using a fraction of CPU time required for no AMR case. SLAC\RPI (TSTT) developed AMR in Omega3P to speed up convergence, improve accuracy and reduce computing time. Wall Loss on AMR Mesh RFQ Frequency Convergence Qo Convergence More accurate f and Q predictions reduce the number of tuners and tuning range, and allow for better cooling design

16. Omega3P/S3P provided the dimensions for the LCLS RF Gun cavity that meet two important requirements:  minimized dipole and quadrupole fields via a racetrack dual-feed coupler design,  reduced pulse heating by rounding of the z coupling iris. Quad  (  βr)/mm A new parallel Particle-In-Cell (PIC) capability is being developed in T3P for self-consistent modeling of RF guns needed for the LCLS upgrade, future light sources and FELs. Quad BES LCLS – RF Gun Cavity

17. Projects in Progress  ILC – Cavity Design  Eigensolvers  Visualization  Shape Optimization  Parallel Meshing  Performance

18. HEP ILC – Accelerating Cavity The ILC, highest priority of future HEP projects, will be the most costly accelerator (many billion $) and is being designed by an international team Europe, Asia and North America. An international collaboration (KEK, DESY, SLAC, FNAL and Jlab) is assessing a Low-Loss (LL) design as a viable option for the ILC superconducting RF accelerating cavity.

19.  HEP ILC – Low-Loss Cavity Design The LL Cavity uses a cavity shape that has 23% less cryogenic loss. The design challenge is to ensure damping of the HOMs over a broad band of frequencies via the two HOM couplers while maintaining the efficiency of the fundamental mode. Two variants of the LL cavity are being considered, the DESY LL design and the KEK Ichiro design. SLAC is simulating both designs.

20. Partitioned Mesh of LL Cavity Q ext With new advances in eigensolvers under SciDAC, Omega3P can now compute the complex frequency or Qe =  r /  i of HOMs as a result of damping by the HOM couplers HEP ILC – HOM Damping DESY KEK Qe

21. Advances in Eigensolvers i Omega3P Lossless Lossy Material Periodic Structure External Coupling ESIL ISIL w/ refinement Implicit Restarted Arnoldi SOAR Self-Consistent Loop WSMPMUMPSSuperLUKrylov Subspace Methods Domain-specific preconditioners Cavity designs with coupling to external waveguides require solutions to a nonlinear eigenvalue problem as the boundary conditions also depend on the eigenvalue. Two solvers, SOAR and SCL have been applied successfully to the ILC cavities. (SLAC, TOPS/SAPP - LBL, Stanford, UC Davis)

22. Advances in Visualization New graphics tools for rendering LARGE, multi-stream, 3D unstructured data have been developed, to be supported by a dedicated visualization cluster to help in analyzing cavity design, such as mode rotation in the ILC cavity. Rendering of LARGE, 3D multi-stream, unstructured data is essential to the analysis of accelerators, such as mode rotation exhibited by the two polarizations of a damped mode in the ILC cavity as they are coupled through resonance overlap. (SLAC, SAPP - UC Davis)

23. R b b1 a1 L a2 b2 Advances in Shape Optimization A parallel shape optimization capability is under development in Omega3P for optimizing the ILC cavity end cells to damp trapped HOMs so the cavity can meet beam stability requirements. Omega3P Sensitivity Optimizer Geometric model Omega3P Meshing (SLAC, TOPS – CMU, LBL, Columbia, TSTT – SNL, LLNL)

24. Advances in Meshing To model a chain of cavities within a cryomodule a parallel meshing capability has been developed to overcome the single CPU memory limitation of standard meshing software. Processor: 1 2 3 4 4 cavity cryomodule at STF (KEK) (SLAC, TSTT – U Wisconsin, SNL) CAD-based Partitioning for Parallel Meshing

25. Advances in Omega3P Performance (SLAC, PERC – LBL, LLNL) Breakdown of Solve & Postprocess Speedup after code optimization LLNL Latest communication pattern study on NERSC LBL

26. AST- ESS under SciDAC  Support Accelerator Science/Design across SC  Advance Computational Science through ISICs and SAPP BPM & Wakefields in LCLS Undulator ILC XFEL SC RF Gun MIT PBG Cavity for Jlab 12 GeV Upgrade ILC LL Cavity & Cryomodule