2. A Monte Carlo Model of Tevatron Operations Elliott McCrory Fermilab/Accelerator Division 13 October 2005

3. Elliott McCrory, Fermilab/AD 3/70 13 Oct 2005 Where is Fermilab?

4. Elliott McCrory, Fermilab/AD 4/70 13 Oct 2005 ~50 km

5. Elliott McCrory, Fermilab/AD 5/70 13 Oct 2005 Fox River

6. Elliott McCrory, Fermilab/AD 6/70 13 Oct 2005 Fermilab Overview Main Injector & Recycler Tevatron Linac Booster Pbar Source

7. Elliott McCrory, Fermilab/AD 7/70 13 Oct 2005 Outline  Overview of the Operations Model  Monte Carlo ≡ Randomizations  SDA  Sequenced Data Acquisition  Shot Data Analysis  Model Observations and Predictions  Effects of Future Improvements  Note:  Several “extra” concepts relating to current Tevatron performance. May have to skip some.

8. Elliott McCrory, Fermilab/AD 8/70 13 Oct 2005 Definitions of “Model”  Curiously subtle shades of meaning! 1.An example for imitation or emulation “My brother is a role model for my son” 2.Person who serves as a subject for an artist or a fashion designer 3.A Structural Design “We need a business model” 4.A type or design of a product “I own a Volkswagen.” “Which model?” “Jetta.” 5.Something built to represent reality in a simplified way “Model Airplane, 1:32 scale”  Here: 5

9. Elliott McCrory, Fermilab/AD 9/70 13 Oct 2005 Fermilab Terminology  Stack  The antiprotons in the Accumulator  Stash  The antiprotons in the Recycler  Shot  The process of transferring antiprotons to the Tevatron  Done during a “Shot Setup”  Store  Proton/antiproton collisions in the Tevatron  Begins at the end of the Shot Setup  Often used interchangeably with shot  Transfer  Accumulator  Recycler antiproton transfer and its associated setup time

10. Elliott McCrory, Fermilab/AD 10/70 13 Oct 2005 Fermilab Operations: Basics  Stacking  Antiproton production in Debuncher  Accumulator Every 2 to 3 seconds 15E10 per hour  Acc  Recycler transfers Three or four time per store, today –Depends on stacking rate  Shot Setup 100 to 200 minutes Each step is 10 to 60 minutes  Tuning  Transfer protons into Tevatron  Transfer antiprotons into Tevatron  Accelerate  Squeeze/scrape  Collisions  20 to 40 hours  Between stores …

11. Elliott McCrory, Fermilab/AD 11/70 13 Oct 2005 The Recycler  An antiproton Storage Ring  Main bends are permanent magnets  Transfers into Recycler every few hours Offloading antiprotons from Accumulator  Advantages over Accumulator  Electron cooling and Stochastic cooling  Emittances are better 4π smaller transverse emittance Longitudinal emittances are consistent and smaller  Transfers into Tevatron are better Transmission efficiency is higher Brighter antiprotons bunches at collisions  Can hold more antiprotons  Stacking rate into Accumulator is better at smaller stacks

12. Elliott McCrory, Fermilab/AD 12/70 13 Oct 2005 Fermilab Operations: Update  Averaging 18 pb -1 delivered per week to our two experiments  107 ± 27 hours/week in collisions  10 E10 antiprotons/hour  Initial luminosity world record set on 4 October  1.42E32 [cm -2 sec -1 ]  Main Injector  Slip stacking  Recycler  Electron cooling achieved on July 9  Implemented for >50% of transfers to Tevatron in last 4 weeks Cool by 70 eV-sec in 80 minutes, 250E10 particles  I am, by no means, an expert on these topics!  An intelligent observer, perhaps

13. Elliott McCrory, Fermilab/AD 13/70 13 Oct 2005 Recycler Electron Cooling Beam current: 250E10 Longitudinal emittance 53 min Cool 70 eV-sec in 80 minutes Transverse emittance

14. Elliott McCrory, Fermilab/AD 14/70 13 Oct 2005 Tevatron Operations Status  BPM Upgrade completed  New lattice implemented last month  28 cm beta-star Practical understanding of coupled machine Partially equalized luminosity at 2 experiments Reduced beta-beating in arcs between 2 experiments Increase luminosity by ~15%  Previous lattice change  December 2004  30% improvement in luminosity  Extra  Orbit stabilization  Crystal Collimator demonstration?

15. Elliott McCrory, Fermilab/AD 15/70 13 Oct 2005 Orbit Stabilization EXTRA

16. Elliott McCrory, Fermilab/AD 16/70 13 Oct 2005 Crystal Collimator Study EXTRA

17. The Operations Model

18. Elliott McCrory, Fermilab/AD 18/70 13 Oct 2005 One Week of Operation Recycler Stash Luminosity Accumulator Stack

19. Elliott McCrory, Fermilab/AD 19/70 13 Oct 2005 One Simulated Week of Ops Hours Blue: recycler Stash [E10] Red: Luminosity [1/(cm 2 sec)] Green: Accumulator Stack [E10] Recycler Stash Luminosity Accumulator Stack

20. Elliott McCrory, Fermilab/AD 20/70 13 Oct 2005 Basic Idea  Phenomenological representation of the Tevatron Complex  Mostly non-analytic  Monte Carlo (randomizations)  Complexity is replaced by randomizations  Downtime  For the Tevatron, stacking, PBar Source, etc.  Real data: Match model to reality  This model’s genesis:  To develop intuition and provide guidance for optimizing luminosity  Now:  Extrapolations/”What If”, based on today’s performance The effect of Recycler improvements

21. Elliott McCrory, Fermilab/AD 21/70 13 Oct 2005 Complexity  Randomness  Variations in all realistic parameters  For example Transmissions during a shot, Luminosity lifetimes, Extraction efficiency from antiproton sources, Shot setup time, Downtime for each sub-system, Etc…  Model Assumptions  Performance does not improve Random fluctuations around a specific set of parameters Performance determined largely by these parameters Better performance? Change parameters and run again.  No shutdown periods

22. Elliott McCrory, Fermilab/AD 22/70 13 Oct 2005 Luminosity Characterization  One average proton & 36 antiprotons are tracked  Proton bunches are all the same  Recycler & Accumulator antiproton bunches are different  L i (t=0) = K H N p (0) N PBar, i (0) [ є p (0) + є PBar, i (0)]  L (t) = L (0) e -t/τ(t)  τ(t) = τ(0) + C 1 t C 2 τ(0) depends on L (0) and is adjusted to fit Real Data C 1 = 3 ± 2 C 2 = f(C 1 ) ≈ 0.5

23. Elliott McCrory, Fermilab/AD 23/70 13 Oct 2005 Match Model to Reality  Goal  Appropriate range of values for important parameters  Correlations among the parameters  Data Sources  SDA The “Supertable” Other data tables  Data loggers  Weekly summaries from operations

24. SDA

25. Elliott McCrory, Fermilab/AD 25/70 13 Oct 2005 SDA: Overloaded Acronymn  Sequenced Data Acquisition  Defines alternate “clock” for recording data  Extends definition of what can be stored  Shot Data Analysis  Look at Sequenced Data Acquisition database  Look at conventional data loggers  Create summaries  Do certain types of calculations More complicated (transmission efficiencies) Time dependent (Emittances)  Observe/alert

26. Elliott McCrory, Fermilab/AD 26/70 13 Oct 2005 Sequenced Data Acquisition  More relevant “clock”  Shot/store number Today: store # 4440  Case Collider shot: 15 main cases 1.Proton Injection Porch 2.Proton Injection tune up 3.Eject Protons 4.Inject Protons 5.Pbar Injection Porch 6.Inject Pbars 7.(Defunct) 8.Before Ramp 9.Acceleration 10.Flattop 11.Squeeze 12.Initiate Collisions 13.Remove Halo 14.HEP 15.Pause HEP  Set Each case may have one or more sets  For example:  “What happened at 4401, Inject Protons, second bunch injection [a.k.a. Set 2]?”  Other common processes use this clock abstraction  Accumulator  Recycler transfers  Pbar Transfers to Tevatron

27. Elliott McCrory, Fermilab/AD 27/70 13 Oct 2005 Sequenced Data Acquisition  Data collection abstraction  All types of data can be acquired  Implemented as a Java interface  SDA Database  Detailed information 36 bunch data Raw data from front ends  Indexed by Store Number Accumulator to Recycler Transfer Number  30 GB today  Data Loggers  Not strictly part of this, but very relevant  Store pairs in relational DB Essentially Unix + milliseconds timestamp  70+ instances at Fermilab O(100 GB)

28. Elliott McCrory, Fermilab/AD 28/70 13 Oct 2005 Shot Data Analysis  Data mining applications  Example  Sequenced Data Acquisition cross-checks  Summary tables on the web  The Supertable  A summary of key information, mostly from SDA database  Excel, HTML, AIDA/JAS  One row = one store  224 columns for each store  http://www-bd.fnal.gov/sda/supertable

29. Elliott McCrory, Fermilab/AD 29/70 13 Oct 2005 SDA Database Example

30. Elliott McCrory, Fermilab/AD 30/70 13 Oct 2005 http://www-bd.fnal.gov/sda/supertable

31. Elliott McCrory, Fermilab/AD 31/70 13 Oct 2005 Supertable Example

32. Elliott McCrory, Fermilab/AD 32/70 13 Oct 2005 SDA Examples Relevant to Model  Using Excel  Initial Luminosity versus Number of Antiprotons  Initial Luminosity versus Initial Luminosity Lifetime  Antiproton Emittances  Uncertainty at the IP Beta-star changing??  Extras  Lifetime fits Record luminosity vs. record integrated luminosity?  Antiproton Burn Rate  Tevatron failure rate Not strictly SDA

33. Elliott McCrory, Fermilab/AD 33/70 13 Oct 2005 Initial Luminosity vs. # PBars

34. Elliott McCrory, Fermilab/AD 34/70 13 Oct 2005 Initl Lum Vs. Init Lum Lifetime

35. Elliott McCrory, Fermilab/AD 35/70 13 Oct 2005 PBar Emittance at Extraction Accumulator Recycler

36. Elliott McCrory, Fermilab/AD 36/70 13 Oct 2005 PBar Emittance at Extraction Model generated Emittances Real Emittances from Recycler Number of Antiprotons Removed [E10] Emittance

37. Elliott McCrory, Fermilab/AD 37/70 13 Oct 2005 Luminosity Decay Fits http://mccrory.fnal.gov/tevatronDecayFits  Three types of fits  e (-t/tau) over first 2 hours  e (-t/tau(t)) like in the model  1/t EXTRA

38. Elliott McCrory, Fermilab/AD 38/70 13 Oct 2005 Fit results for Stores 4332 τ(t) L(t) = L(0) exp(-t/τ(t)) τ(t) = τ(0) + c1 × t c2 Fourth best initial luminosity Second Place for Integrated Luminosity EXTRA

39. Elliott McCrory, Fermilab/AD 39/70 13 Oct 2005 Fit results for Stores 4332 & 4431 World record Initial Luminosity, 1.43E32 Third Place for Integrated Luminosity L(0)HoursIntegTau(0)C1C2 43321.2830.85329.4.391.270.541 44311.4233.35265.2.642.220.495 EXTRA

40. Elliott McCrory, Fermilab/AD 40/70 13 Oct 2005 Antiproton Burn Rate  Calculated numerically using fitted results  Removes data noise  Cut: χ 2 /DOF < 30 (error bars fixed: 0.05E9)  Luminosity Burn Rate [R lum (t)]  dN(A) / dt [E9 particles/hour] = −0.252 × ( L CDF + L D0 ) [E30/cm 2 sec]  CDF & D0 Luminosities taken from SDA,  Assumptions: Emittances, tunes, orbits, etc. are bunch independent (?!)  See Beamdocs # 1408  http://beamdocs.fnal.gov EXTRA

41. Elliott McCrory, Fermilab/AD 41/70 13 Oct 2005 Burn Rate [E9 pbars/hour] Non-Luminosity Burn Rate Luminosity Burn Rate Total Burn Rate Summary: 9/36 Bunches in Store 3744 Hours into the Store EXTRA

42. Elliott McCrory, Fermilab/AD 42/70 13 Oct 2005 Uncertainty at the IP Luminosity / (all known factors) β* = 28 cm  Better emittance measurements  Better lattice understanding  Better instrumentation EXTRA

43. Elliott McCrory, Fermilab/AD 43/70 13 Oct 2005 = 0.975 / hour Tevatron Failure Rate f(t) = e - t σ = = 1/ Time Between Tevatron Failures; Real Data R ≈ 1 - Δt Δt = 42 hours e - t Model data for Tevatron Failures

44. Elliott McCrory, Fermilab/AD 44/70 13 Oct 2005 Failure Rate: Interpretation  is Tevatron “Up Time”  is measured directly from real data  = σ = 1/  Probability of having stores of:  1 hour: 0.975  2 hours: (0.975) 2 = 0.951  10 hours: (0.975) 10 = 0.776  20 hours: 0.603  30 hours: 0.459  Failures are Independent of Time  This is a random process!!

45. Elliott McCrory, Fermilab/AD 45/70 13 Oct 2005 Reliability of Tevatron Today  Tevatron is two machines  Low beta: Higher reliability ~ 0.988 –20 hours: 0.785 –30 hours: 0.696  Injection and ramping: Lower reliability ~ 0.88 to 0.95  Recovery time  Severe for superconducting machine  Classes of failures?  Beyond the scope of this talk!  ∴ Longer stores  Tevatron is more reliable in collisions

46. Model Details and Predictions

47. Elliott McCrory, Fermilab/AD 47/70 13 Oct 2005 State Machinery  All machines are implemented as Finite State Machines  Vary in complexity  Proton source: 5 states Ready, Down, Sick, Studies, Access  Accumulator/Debuncher: 7 states ReadyStacking, ReadyShot, ReadyRecTransfer, Down, Recovery, Sick, Studies  Tevatron: 17 states Ready, 7 shot-setup, 4 luminosity, Failure, Studies, Access, Recovery, Turn-Around  Recycler: 12 states Ready, 4 transfers (2 in, 2 out), 2 down, recovery, 2 studies, access, cooling, turn-around.

48. Elliott McCrory, Fermilab/AD 48/70 13 Oct 2005

49. Elliott McCrory, Fermilab/AD 49/70 13 Oct 2005 Program Structure  C++/Linux  800 weeks/minute On 1.8 GHz Celeron  220+ parameters  How does this work?  Step size = 0.1 hours  “Listeners” provide connections among State Machines  Main program guides time progression & venue for main decisions Stack –Do transfer to Recycler? “End-store” criterion satisfied? Start shot setup.  Repeat for N weeks, dumping lots of relevant data.  Input parameters  Over200 input parameters to a model run  Output handler  Lots of data files can be dumped

50. Elliott McCrory, Fermilab/AD 50/70 13 Oct 2005 Random Numbers RandomLikely(-2, 12, 8) Product of these two distributions RandomLikely(0, 5, 2)  Linux drand48( ) “RandomLikely”

51. Elliott McCrory, Fermilab/AD 51/70 13 Oct 2005 Decisions  Same as reality  Store  When to end the store  When to begin a store after a failure Answer: Wait for accumulation of antiprotons  Antiprotons  When and how much to transfer from Accumulator to Recycler  Combination  How many antiprotons to get from two sources Recycler only, Accumulator only, Combined Source

52. Elliott McCrory, Fermilab/AD 52/70 13 Oct 2005 Some End-Store Criteria  Store Duration  Integrated Luminosity to experiments  Number of Antiprotons we have available  How low L can the experiments use  Best: Combination of last two:  Np  expected luminosity  R = Expected Luminosity / Actual Luminosity  This criterion works very well algorithmically, but there are other considerations in Real Life Nowadays, the Run Coordinator ends a store based on this factor and many other factors, e.g., time of day.  If Model is believable  Can change the performance  See how the End-Store criteria respond  Find the Best criterion for ending stores for lots of parameters

53. Elliott McCrory, Fermilab/AD 53/70 13 Oct 2005 End-Store Criterion  How to decide which is the “Best” criterion?  It integrates lots of luminosity  It insensitive to natural fluctuations in parameters Some of these changes may be unnoticed Random fluctuations or improvements?!  It is simple Everyone can understand it! –Some effective but complex schema have been rejected  Look at two end-store criteria  Integrated Luminosity  Ratio Of Expected luminosity from available antiprotons to the luminosity now

54. Elliott McCrory, Fermilab/AD 54/70 13 Oct 2005 Integrated Luminosity Criterion Number of Antiprotons [E10] Or Luminosity [E30/cm 2 /sec] Hours from start of simulated “Run”

55. Elliott McCrory, Fermilab/AD 55/70 13 Oct 2005 Optimization of “Integ Lum” End store when Integrated Luminosity reaches this value [nb -1 ] Average Integrated Luminosity for the Week [nb -1 ]

56. Elliott McCrory, Fermilab/AD 56/70 13 Oct 2005 Store Duration: Integ Lum Stop at Integ=3000 nb -1 4000 nb -1 5000 nb -1 6000 nb -1 7000 nb -1 Duration of stores ended intentionally [hours]

57. Elliott McCrory, Fermilab/AD 57/70 13 Oct 2005 Target Ratio Criterion Number of Antiprotons [E10] Or Luminosity [E30/cm 2 /sec] Hours from start of simulated “Run”

58. Elliott McCrory, Fermilab/AD 58/70 13 Oct 2005 Ratio: End at R>6; R(t) Hours from start of simulated “Run”

59. Elliott McCrory, Fermilab/AD 59/70 13 Oct 2005 Ratio Criterion: Optimization End store when Ratio reaches this value Average Integrated Luminosity for the Week [nb -1 ] 22700 for integ

60. Elliott McCrory, Fermilab/AD 60/70 13 Oct 2005 Ratio: Store Duration Store Duration [hours] dN/dt [stores/1 hour bin] End store when Ratio=2 4 6 8 10

61. Elliott McCrory, Fermilab/AD 61/70 13 Oct 2005 Decisions Involving Recycler  More decisions with Recycler  When to shoot from Accumulator to Recycler? How much to take?  Whence do we get pbars for Tevatron?  Long story short …  Shoot to Recycler when stack reaches 40 to 80 E10  Get pbars mostly (all?) from Recycler Presently, want to get all pbars from Recycler But luminosity lifetime may be diminished because of brighter pbars –Does integrated luminosity suffer??? “use it or lose it” –Ignores antiprotons in Accumulator

62. Elliott McCrory, Fermilab/AD 62/70 13 Oct 2005 Ongoing Studies on Optimum Recycler  Some crucial dependencies  Time required to transfer into Recycler Currently, 0.75 to 2 hours, most likely=1 hour. Plan: 15 minutes or less  Transmission efficiency Now ~90%  Emittance from Recycler is ~4π less than Accumulator Improved L (0), but diminished initial lifetime; –∫ L dt ?

63. Elliott McCrory, Fermilab/AD 63/70 13 Oct 2005 # Pbars Avail vs. Transfer Time Analytic calculation by D. McGinnis, BeamDocs # 1948

64. Predictions for Future Performance

65. Elliott McCrory, Fermilab/AD 65/70 13 Oct 2005 Predictions on Future Performance  Recycler improvements  It may be able to hold 6E12 antiprotons  Transfers from Accumulator should eventually take a minute or less  Recycling?????? Collecting spent antiprotons from the Tevatron and re-cool them with Electron Cooling  Accumulator improvements  Today’s goal: 24E10 antiprotons per hour

66. Elliott McCrory, Fermilab/AD 66/70 13 Oct 2005 Accum: 24E10/hr; Recy:6E12 Number of Antiprotons [E10] Or Luminosity [E30/cm 2 /sec] Hours from start of simulated “Run” Stack Size Stash Size Luminosity

67. Elliott McCrory, Fermilab/AD 67/70 13 Oct 2005 Optimization of 24 mA/hr: 6E12 Average Integrated Lum/week [1/nb] End-of-store Ratio Best today Future Performance

68. Summary & Conclusions

69. Elliott McCrory, Fermilab/AD 69/70 13 Oct 2005 Conclusions  This Operations Model has helped us understand how to operate the Complex  SDA has been a crucial element to understanding the Tevatron and making this model work  The clock abstraction created by SDA has been key

70. Elliott McCrory, Fermilab/AD 70/70 13 Oct 2005 Lessons for LHC?  Reliable, redundant, easily accessible performance data are crucial to understanding how you are operating  Shot/Case/Set clock  Complexity of LHC loading may be better to model  Reliability matters (duh!)

71. Fin! A Monte Carlo Model of Tevatron Operations Elliott McCrory Fermilab/Accelerator Division 13 October 2005