1. CEBAF Accelerator Status
Arne Freyberger
Operations Department
Accelerator Division
CLAS Fall 2017 Meeting

2. Availability Challenges
Range of CEBAF Availability during 6GeV Operations

3. CEBAF System Availability
Weekly availability reviews at the weekly scheduling meeting
Early identification of issues
Monthly availability report
Presented and discussed on the first Friday of a new month at the daily 8am meeting
Look for trends
Increased statistics for better information
Annual availability review is used to set priority for support groups annual work plan (AWP).
Feb shown
A good month

4. DTM Tracks all downtime events Downtime: Period longer than 5minutes
Requires manual entry by operator
Data quality may not be the top priority when things are not going well
Operability reviews and presents every Wednesday
Useful interpretation of this data requires that system owners help Operability in QA the data.
Dealing with system downtime during non-beam operations not trivial
Dealing with stops/starts of program not trivial
Downtimes that result in program and/or schedule change may not be correctly tracked

5. Availability Data: DTM
Failure Rate: 𝜆= 1 𝑀𝑇𝐵𝐹
MTBF: Mean Time Between Failure, desire this to be infinite
MTTR: Mean Time to Repair, desire this to be zero
𝐴𝑣𝑎𝑖𝑙𝑎𝑏𝑖𝑙𝑖𝑡𝑦= 𝑀𝑇𝐵𝐹+𝑀𝑇𝑇𝑅 𝑀𝑇𝐵𝐹
lim 𝑀𝑇𝑇𝑅→0 𝐴𝑣𝑎𝑖𝑙𝑎𝑏𝑖𝑙𝑖𝑡𝑦 = 100%
Count: Number of failures in a given period
System Repair Time: 𝐶𝑜𝑢𝑛𝑡 ∗𝑀𝑇𝑇𝑅

6. It’s not personal, it’s data

7. System Availability

8. System MTBF (Large # is good)
End of life
WTF!!!!!!

9. System MTTR (Small # is good)

10. Cryogenics
Data highly skewed by significant events that resulted in programmatic change
MTBF mostly due to CHL1->SC1 faults, classic end of life.

11. RF MTBF: about two events per day
MTTR: less than 1h, good but that’s two derailments per day.
Would better training of operators improve MTTR even more? Improved controls?

12. CEBAF Performance Plan
Define the performance goals
Availability/Reliability
Beam Parameters (emittance, energy spread, bunch charge, beam energy)
Peak Conditions (Hall multiplicity, operating weeks,…)
Perform gap analysis to identify where present CEBAF performance does not achieve the performance goals
Develop a plan to close the gaps

13. CEBAF Performance Goals

14. CEBAF 20 year view

15. Critical Spares Top Priority
Spring 2014: ZA magnet coil and vacuum failure; 3 week interruption to replace damaged coil and repair the vacuum chamber. This failure consumed the existing spare coil; the next failure will take much longer for repair and recovery.
Spring 2015: Cold compressor 4 failure in 2 K cold-box, SC1; No spare at JLab, consumed the SNS cold compressor spare. Program change required: half design energy after 5 week down.
Fall 2015: YR coil on 3-pass extraction generated a spontaneous leak. No 3-pass program for FY16, repaired Summer 2016, consumed a YR coil spare. Required Hall-A DVCS experiment to re-arrange its run plan.
Fall 2016: Arc7 box supply failure, no spare, program change required: to single hall operation until supply repaired.
Fall 2016: 5th pass separator vacuum leak, program change required, could not support 5th pass beam to Hall-A simultaneously with 5.5 pass beam to Hall-D.
Spring 2017: Cold compressor 5 failure in 2 K cold-box, SC1; on-going root cause investigation, might be repairable. Scheduled program terminated.

16. Energy Gap and Plan Energy Gap:
Presently CEBAF cannot support design energy (FY17/FY18 operations at 4% below design)
Annual gradient loss exceeds the gain from one C50/year
One C50/year insufficient to maintain energy reach; gap is growing!
Energy Plan:
C75 Upgrade: replace the C20 cavities with high current 5-cell cavities
Re-use as much of the cryomodule as possible
Total cost in terms of MV/$ lower than C50 or C100 options
Upgrade LLRF to digital controls (similar to C100)
Target specifications:
Gradient: 18.5 MV/m
Q0 > 8e9
Upgrade warm region vacuum and clean (SRF quality processing) all warm vacuum regions
two warm girders/year (continues to end of 12GeV/era)
Upgrade analog LLRF with digital LLRF in all zones
two zones per year (continues to end of 12GeV era)

17. Energy Gap and Plan (C75 Upgrades)
Proposed path to design energy (1090 MeV/linac)
Comparable to C50 program post hurricane Isabel
If annual degradation is mitigated: might achieve 1090 MeV/linac by FY21 and with fewer C75 modules

18. Preliminary Performance Plan M&S Cost Estimate
C75s
C100-Refurbs
Preliminary
Near term goal is to close the gaps by FY23
Plan emphasizes improving CEBAF availability/reliability first:
Energy Reach second
Operating at optimal weeks third
All dollar values in FY17 direct costs

19. July : CHL1 Event
Modules that warmed above 50 K require a thermal cycle to room temperature and a controlled cool-down to 4K
All the modules in these plots had temperature excursions above 50 K (or were already at room temperature for maintenance.
Significant He contamination and loss of inventory.
As of today all modules, except one, are full of LHe 4K.
This remaining module will be cooled and filled during the transition to 2 cryo plants

20. 12GeV CEBAF Beam Parameters
Obsolete
At the request from Physics Division the beam parameter table created in 2001 for the4-6 GeV program is being updated to reflect 12GeV CEBAF capabilities.
Please work with your APEL (Michael Tiefenback) in developing a table that will define the beam parameters appropriately for Hall-B experiments.

21. 4-Hall Constraints In order to deliver beam to all 4-halls:
At least one original end-station (A,B or C) must be on 5th pass
The last 4 days in Dec as published violate this rule.
Hall-B will be at 5th pass for all of Dec (unless Hall-D is offline)
Lower passes will be arrange in the Jan/Feb/Mar beam delivery
End-station cryogenic load can exceed the ESR capacity
End-station magnet/target cool-downs must be carefully scheduled.
High power target operations in one or more halls must be carefully scheduled.

22. The Good News
CHL1 has been cleaned and is back in service: Presently maintaining 4K
All 5 cold compressors in SC1 have been operated up to 20% of operating value.
Full SC1 spin up and 2K pump-down attempts (the first since March 10th) will take place today.
Once CEBAF 2K; assessment of the SRF gradient capabilities will take place.
Transition from CHL1->SC1->CEBAF to 2 cryo-plants maintaining 2 schedule for Nov. 12.
RF assessment + Magnet soak continues through Thanksgiving
Re-visiting Cryo configuration during down-times: CEBAF/Cryo will over Winter break
Pro: Fewer transitions on aging 2K cold-boxes
Pro: Beam Operations can be extended to Dec. 21 (NPES schedule has Dec. 18th)
Con: Increased cost (power bill)
SRF OPS has been producing field emission free cavities for LCLS-II; 7 out of 8 cavities FE free.
4-Hall system (lasers, separators, operations) ready for integration and first 4-hall operations.
The CEBAF Performance Plan has been developed and reviewed. Presently working for Lab leadership to find funding opportunities.
Funding for a new 2K cold-box (replace SC1) has been secured in FY year effort, $8M.

23. Backup

24. Downtime Count

25. System Repair Time

26. Optimizing 12GeV Operations
Reliability/Availability Focus
4-Hall Operations
Improved Linac/Cryogenic procedures: Reduce risk of cryoplant contamination due to He connections/disconnections
Linac LHe Pressure optimization
Address end-of-life issues:
SRF C20 Warm poly RF windows
SRF C20/C50 end-can O-rings
Improve C100 warm girder tolerance to radiation environment
Identify and correct errant magnets
West Arc to East Arc extension cable, allows for West Arc power supply to power East Arc magnets. Provides CEBAF the ability to support a full ABC program in the event of an East Arc box power supply failure.
Energy Reach/Gradient Maintenance
Linac energy setting: margin -> reliable operations
Identify source of SRF Particulate
Cleaning/processing warm girder regions
Improved tunnel vacuum procedures

27. CEBAF Weeks of Operation Delivered
Optimal
# Weeks
Accelerator Shutdown

28. From 2012 S&T Presentation

29. FY17 Joule Metric
FY17 first year of 12 GeV operations reporting DOE metrics
Beam operations for FY17 complete, with the exception of a few days of restoration at the end of FY17 (Sept 2017).
Measured Reliability 76.3%
Exceeds our expectations of 65%
Below nominal DOE goal of 80% for all facilities

30. Availability/Reliability Gap & Plan
Reliability and Availability inadequate for robust effective operations.
Critical system failures have forced program/schedule changes to almost every run period to date.
Availability/Reliability Plan:
Purchase CEBAF and Cryogenic critical spares
Build and commission a new 2K cold-box
Purchase System immediate needs
Shore up the depleted spares
Purchase 6.5 kW and 13 kW klystrons
Address immediate obsolescence issues

31. Operations Gap & Plan Operations Gap (Support 4-Hall/optimal weeks):
Operator/CrewChief Staffing inadequate for 4-hall operation
Group is presently 4 heads below historical 6GeV levels
Technical and MCC staffing is inadequate to support optimal weeks (37 weeks/year)
ESR capacity insufficient to support robust 4-hall operations
Operations Plan:
Increase staffing
Execute projects that will reduce time required for maintenance
Spare warm girders
Redundant CHL carbon beds
Transition from 2 PSS certifications/year to 1 certification/year
Build a new ESR for robust 4-hall operations