1. Frank SchmidtICE 5th of December 20121 1) Development of MAD-X with Space Charge and preliminary Application to the Fermilab Booster 2) Benchmarking of PTC_ORBIT & SYNERGIA with Emphasis on Micro-Scale Stability Visit at Fermilab (Aug – Nov 2012) F.S.

2. Frank SchmidtICE 5th of December 20122 1)MAD-X development and applications Valery Kapin. 2)Orbit issues and possible remedies are being pursued together with Leonid Vorobiev. Continuous discussions with Valery Kapin and Giuliano Franchetti from GSI. 3)SYNERGIA installation on my Laptop and getting applications to work Eric Stern. 4)Collaboration with Sasha Valishev to introduce a the exact Hamiltonian into LIFETRAC (not discussed here). Acknowledgements

3. Frank SchmidtICE 5th of December 20123 Development of MAD-X with Space Charge and preliminary Application to the Fermilab Booster References: 1.V. Kapin & Yu.Alexahin, “Space Charge Simulation Using MADX with Account of Synchrotron Oscillations”, Proc. RuPAC-2010, Protvino, Oct 27, 2010, pp. 204- 206. 2.V. Kapin, “Space Charge Simulation Using MADX with Account of Longitudinal Motion”, FNAL Beamsdoc- 3582 ‑ v2, Apr. 2011. 3. Y.Alexahin et al., “EFFECTS OF SPACE CHARGE AND MAGNET NONLINEARITIES ON BEAM DYNAMICS IN THE FERMILAB BOOSTER”, Proceedings of PAC07, Albuquerque, New Mexico, USA. Task I

4. Frank SchmidtICE 5th of December 20124 MAD-X V3 with Space Charge for debuncher (Valery Kapin 2010) Utilizing the BB elements to create a frozen space charge model but adapting emittances and Twiss parameters for the sigma determination. Using the MAD-X Macro technique for all operations Phase-I: Lattice Preparation Splitting the elements (at least once) and introduce SC kicks in between Making sure that the lattice is stable including SC and proper sigma values (  next slide) Transferring thick lattices into thin ones (symplectic and only way to track in MAD-X) Phase-II: Running (turn-by-turn due to Macro procedures) Time varying fields Emittance and TWISS calculation during run Output (losses, emittances, etc)

5. Frank SchmidtICE 5th of December 20125 Procedure to determine Beam Sizes with large SC Tune Swings TWISS fails for integer and/or half integer tunes  Trick to avoid it: turn on SC tune-shift in small steps and let tunes converge at each step Lines represent Laslett formula Reference: Valery Kapin talk at GSI 2009

6. Frank SchmidtICE 5th of December 20126 Lattice with variable Parameters M.J.Syphers, “Status of Mu2e Operating Scenario”, Feb, 2010 Beams-doc-3556  Mu2e experiment  Accelerator issue: 8 GeV Accumulator and Debuncher storage rings are used  Compact bunches (<200ns,  <0.025) are prepared in Accumulator ring  Bunches are transferred to Debuncher ring and extracted out to the experiment using 3-rd order resonant extraction with spill times ~150ms. A stable, slow spill with a very intense proton beam is a big challenge.

7. Frank SchmidtICE 5th of December 20127 The MACRO technique of MAD-X is working fine but typically one ends up with very complex logic so that even Valery had to work hard to re-understand them after a delay of 2 years!  Therefore quite unpractical for new applications. Macros are inherently slow! Therefore the idea was to reduce the use of Macros as much as possible and to modify the code to do most of the work directly within MAD-X. In detail: Time varying multipoles Time varying phase trombone (Forest/Schmidt: Don’t use them!) Time varying cavity voltage (application for Booster lattice) Include the sigma determination at each turn Allow for stop&go for the MAD-X tracking routine to allow intermediate TWISS calculation at start and the location of the SC elements Input via several TFS tables. Task for MAD-X V5sp for the Booster as a new Application

8. Frank SchmidtICE 5th of December 20128 I have left the MAD-X project 18 months ago. Laurent Deniau at CERN has taken over till then. I have been a manager of the code with limited involvement in the actual code writing other than the PTC part. In particular, the core of the code in C has been written exclusively by Hans Grote. In a state of emergency I had to be ready to fix the code quickly. MAD-X is not well suited for this task I since deferred evaluation does not easily allow to put element parameters back into the database. Various C routines were needed but Hans did not have to help! MAD-X Status

9. Frank SchmidtICE 5th of December 20129 N_macro_surv vs Turn Number for the Debuncher Timing on CERN computer Macro version ~20-24h MADX-5sp ~2-4h

10. Frank SchmidtICE 5th of December 201210 Ex,y_rms vs Turn Number for the Debuncher

11. Frank SchmidtICE 5th of December 201211 Longitudinal Emittance vs Turn Number for the Debuncher

12. Frank SchmidtICE 5th of December 201212 MAD-X clean-up changes of the lattice. A small horizontal non- closure is tolerated for the time being. The cavities had to be replaced by zero-length cavities. The first order edge focusing of Bends has to be simulated with the DIPEDGE elements. The booster has a vertical bump and I had to fix a bug to allow for the vertical DIPEDGE element in MADX-5sp. Phase-I has been passed including the very large SC tuneshift of ~0.4 and a successful crossing of the half integer (0.48). Phase-II has been started and some turns have been tracked. However, it has become clear that for such a large SC tuneshift one has to carefully set the initial parameters like tunes etc. Progress for the Booster

13. Frank SchmidtICE 5th of December 201213 Benchmarking of PTC_ORBIT & SYNERGIA with Emphasis on Micro-Scale Stability References: 1.J.A. Holmes et al., “ORBIT: BEAM DYNAMICS CALCULATIONS FOR HIGH- INTENSITY RINGS”, Proceedings of EPAC 2002, Paris, France. 2.E.Forest, A.Molodozhenstev, A.Shishlo, J.Holmes, “Synopsis of the PTC and ORBIT Integration”, KEK Internal, 2007-4, November 2007. 3.L.G. Vorobiev and K.Hirata, “A symplectic Poisson Solver Based on Fast Fourier Transformation”, KEK Report 95-12. 4.L.G. Vorobiev, “Space Charge Templates for High-Current Beam Modeling”, FERMILAB-PUB-08-236-APC. Task II

14. Frank SchmidtICE 5th of December 201214 H-DETUNING GSI SIS18 Giuliano Franchetti Valery Kapin (2008) Shinji Machida Giuliano Benchmark WEB page: http://web-docs.gsi.de/~giuliano/

15. Frank SchmidtICE 5th of December 201215 H-DETUNING PTC-ORBIT

16. Frank SchmidtICE 5th of December 201216 Fake Tune Evolution PTC-ORBIT Turns Qx

17. Frank SchmidtICE 5th of December 201217 Amplitude blow-up PTC-ORBIT Turns X [0.13  ] RED: zero Amplitude GREEN: 0.1 sigma Amplitude

18. Frank SchmidtICE 5th of December 201218 Small Amplitude blow-up SYNERGIA Turns X [0.13  ]

19. Frank SchmidtICE 5th of December 201219 SYNERGIA Code Synergia is an accelerator simulation package for combining advanced single-particle optics with computationally-intensive collective effects. https://compacc.fnal.gov/projects/wiki/synergia2 Synergia is designed to take advantage of computing resources ranging from desktops to supercomputers.

20. Frank SchmidtICE 5th of December 201220 Synergia Personnel Synergia is developed and maintained by the Computational Physics for Accelerators group in Fermilab's Scientific Computing Division James Amundson, Paul Lebrun, Qiming Lu, Alex Macridin, Leo Michelotti (CHEF), Chong Shik Park, Panagiotis Spentzouris and Eric Stern With development contributions from Tech-X: Steve Goldhaber

21. Frank SchmidtICE 5th of December 201221 Physics Of Synergia Single-particle optics are provided by CHEF  Direct symplectic tracking Magnets, cavities, drifts, etc.  (and/or) arbitrary-order polynomial maps Apertures Collective (single and multiple bunches)  Space charge (3D, 2.5D, semi-analytic, multiple boundary conditions)  Wakefields Can accommodate arbitrary wake functions

22. Frank SchmidtICE 5th of December 201222 Synergia Features Python-driven simulations  Pure C++ simulations are possible  All computationally intensive calculations in C++  One example: arbitrarily complex ramping, including feedback, is straightforward Checkpointing  Simulations can be stopped and restarted arbitrarily Parallel scalability  Has run efficiently on 2^17 = 131,072 cores(!), simulating over 10^10 particles. Can be extended in either Python or C++  User-defined Diagnostics  New collective operators  Etc.

23. Frank SchmidtICE 5th of December 201223 Plans for Synergia More collective effects  Electron cloud  In progress  Beam-beam? Cern related Features  Double RF implementation  Time varying fields  Acceleration  Foils  Observer Particles Parsers for more lattice formats Mad8 finished MadX close to done others as requested Full documentation End-user, API and examples Fixed API Very close

24. Frank SchmidtICE 5th of December 201224 “Unfair” speed comparison between PTC-ORBIT and SYNERGIA I have made a speed comparison of the SIS18, which is just a linear lattice plus space charge. For the same number of space charge kicks, the same grid bins and the same number of macro-particles. I find Synergia ~5 times faster than PTC-ORBIT. To avoid to be “unfair” one has to make a more careful analysis for various cases and with non-linearities. PTC allows a very general treatment of space charge without any need to split elements. This might be inherently inefficient. It also could be that up to now PTC-ORBIT is not yet fully optimized. Presently, we have started an attempt to profile the code to understand where the time is being spent. To identify remedies will take some time.

25. Frank SchmidtICE 5th of December 201225 H-DETUNING SYNERGIA 32 Turns 0.1  Qx

26. Frank SchmidtICE 5th of December 201226 H-DETUNING SYNERGIA 1024 Turns 0.1  Qx

27. Frank SchmidtICE 5th of December 201227 Sudden shift in Tune SYNERGIA Sudden shift in Tune SYNERGIA Turns X[m]

28. Frank SchmidtICE 5th of December 201228 Evolution of Linear Invariant SYNERGIA Courant Snyder Invariant

29. Frank SchmidtICE 5th of December 201229 Analysis of the situation with PIC codes Originally, I had planned to follow the complete benchmarking procedure for PTC- ORBIT as prescribed by Giuliano. In view of a possible upgrade of ORBIT to PTC- ORBIT at Fermilab. Following this surprising results I checked out with Leonid every possible source of error that I might have done. In fact, he repeated the same studies with plain ORBIT and we find the same results. The conclusion is: that for PTC_ORBIT the benchmarking can simply not be done. We then contacted the author Jeff Holmes and his team and let them be the first to be informed about this problem. A couple of weeks later I have received an Email by Jeff with a confirmation of our finding: “I have been doing a bit of work on the individual particle tracking with space charge. I set up a uniform focusing channel and a KV distribution. I observe the same numerical diffusion behavior seen by you and Leonid. If I symmetrize the distribution in the x-y planes, the diffusion is much slower and delayed, but eventually it does occur (due to roundoff?).” Lastly, Synergia behaves much better but is also suffering from this instability.

30. Frank SchmidtICE 5th of December 201230 Remedies and Discussions During the discussions with Jeff we have agreed to investigate possible remedies within the next 6 months. Independently, Leonid is reviving his decade old very careful “clouds in PIC” analysis and he also has a much improved Poisson solver. The much smoother distribution (see for comparison next 2 slides) did not in itself lead to a substantial improvement. But we still are hopeful that the full replacement of the space charge classes in ORBIT will bring the desired improvement. To a lesser extend SYNERGIA suffers also from this problem and we should be looking into that as well. Leonid also has a radically different approach that is to use the so-called space charge templates (see reference above), i.e. NOT based on PIC. It remains to be seen how this will work out! Following all these findings and the excitement I am happy to invite you to a space charge workshop to be held in April 2013 at CERN where we will discuss these issues in depth and hopefully with a better understanding! For me the crucial question is if PIC codes can be reliably used for 1’000 - 10’000 turns or more despite this micro-scale instability?

31. Frank SchmidtICE 5th of December 201231 Shapes of Distributions (Leonid)

32. Frank SchmidtICE 5th of December 201232 Original ORBIT Density

33. Frank SchmidtICE 5th of December 201233 Reversed Parabolic Density (#3 in Shape Slide)

34. Frank SchmidtICE 5th of December 201234 MADX-5sp is going to be used at CERN as our frozen model tool for space charge studies. Any developments, bugfixes etc will be provided to Fermilab. Most probable Valery Kapin and Yuri Alexahin will use it here and we will certainly help in case of problems and/or further developments. Optimizing the ORBIT code: Leonid Vorobiev has started to replace the space charge classes in ORBIT. Our plan is to continue our collaboration remotely. But there has been discussions with his future employer to invite Leonid for an extended period to CERN to finish this work in the most efficient way. In fact, since we are investigating a possible speed-up of PTC-ORBIT we could at the same time make his implementation compatible with MPI. Planned Collaborations I

35. Frank SchmidtICE 5th of December 201235 ORBIT authors: Leonid and myself have approached Jeff Holmes and his team to present out findings about the unstable particle motion on the micro-scale level. We have agreed with them to find a solution or at least achieving a dramatic improvement within the next 6 months. It could very well be that the MPI version of Leonid’s implementation will be the best that can be done. SYNERGIA: We have prepared in writing a program for a long- term collaboration for a possible use of this code for the LIU (LHC Injector Upgrade) at CERN. Missing code features have been identified and have been discussed with the code developers. Special attractions are the modern structure with a nice PYTHON interface and the high performance compared to other codes. We have also agreed to perform the full benchmarking cycle that has been prepared by Giuliano Franchetti at GSI. Planned Collaborations II

36. Frank SchmidtICE 5th of December 201236 Reserve

37. Frank SchmidtICE 5th of December 201237 Synergia Personnel Synergia is developed and maintained by the Computational Physics for Accelerators group in Fermilab's Scientific Computing Division James Amundson, Paul Lebrun, Qiming Lu, Alex Macridin, Leo Michelotti (CHEF), Chong Shik Park, Panagiotis Spentzouris and Eric Stern With development contributions from Tech-X: Steve Goldhaber

38. Frank SchmidtICE 5th of December 201238 Physics Of Synergia Single-particle optics are provided by CHEF  Direct symplectic tracking Magnets, cavities, drifts, etc.  (and/or) arbitrary-order polynomial maps Apertures Collective (single and multiple bunches)  Space charge (3D, 2.5D, semi-analytic, multiple boundary conditions)  Wakefields Can accommodate arbitrary wake functions

39. Frank SchmidtICE 5th of December 201239 Synergia Code structure A simulation consists of propagating a Bunch (or bunches) through a Lattice. Inputs: machine lattice, initial bunch parameters Outputs: user-selected Diagnostics. 6d means 6d std deviations 6x6 covariance matrix 6x6 correlation matrix Individual particle tracks Dump of all particles Losses with locations in lattice Some available Diagnostics Actions may specify Diagnostics every n steps, every m turns, specified sets of turns or sets of steps, etc. User-defined logic is also possible.

40. Frank SchmidtICE 5th of December 201240 Synergia Features Python-driven simulations  Pure C++ simulations are possible  All computationally intensive calculations in C++  One example: arbitrarily complex ramping, including feedback, is straightforward Checkpointing  Simulations can be saved and restarted arbitrarily Parallel scalability  Has run efficiently on 2^17 = 131,072 cores(!), simulating over 10^10 particles. Can be extended in either Python or C++  User-defined Diagnostics  New collective operators  Etc.

41. Frank SchmidtICE 5th of December 201241 Status of Synergia Synergia is in production use at Fermilab  Main Injector space charge  Mu2e resonant extraction  Booster with SC and wakefields End-user release in near future

42. Frank SchmidtICE 5th of December 201242 Potential with zero Boundary Condition Assumption (Leonid)