1 BROOKHAVEN SCIENCE ASSOCIATES Lessons Learned from NSLS-II Linac Commissioning Ray Fliller and Emil Zitvogel Linac Manager and Beam Operations Group Leader Workshop on Accelerator Operations 2014

2 BROOKHAVEN SCIENCE ASSOCIATES National Synchrotron Light Source II NSLS-II is a state of the art 3 rd generation light source at Brookhaven National Laboratory. Beam energy 3 GeV, 1 nm horizontal emittance, 500 mA stored current. The $912 million project will be complete this year. First light occurred at the CSX beamline last Thursday, October 23. Commissioning continues for the remaining 5 project insertion devices and damping wigglers.

3 BROOKHAVEN SCIENCE ASSOCIATES Linac Mis-Steer Event On May 29, 2012, a beam mis-steering event took place at the BNL National Synchrotron Light Source II Linac during high intensity commissioning activities to accept the machine Operator conducting beam loading studies with 100 MeV beam energy at 15 nC/s current with dipole set at maximum current This combination caused the beam to bend 4 times more than designed and missed a down stream shadow shield and hit the wall between linac and booster This resulted in elevated radiation levels in a radiation monitored and Controlled Area within the booster tunnel, which is a technical equipment installation area No one was injured and no significant personnel radiation exposure have been identified The maximum dose that one could have received would be 38 mrem. Linac commissioning was suspended This event was not adequately identified or addressed in the Linac Commissioning Safety Assessment Document (SAD)

4 BROOKHAVEN SCIENCE ASSOCIATES Injection Building Local Control Room Linac Tunnel Klystron Gallery Equipment Racks for Linac and LTB line Booster (only arc 1 installed)

5 BROOKHAVEN SCIENCE ASSOCIATES Close-up of Linac and Booster Tunnel Installation Areas

6 BROOKHAVEN SCIENCE ASSOCIATES Linac Tunnel (As seen from the top of the dipole) Simulation of miss-steered beam and impingement on Linac bulk shield wall

7 BROOKHAVEN SCIENCE ASSOCIATES Booster Tunnel Simulation of 1700 mr spot on Booster wall

8 BROOKHAVEN SCIENCE ASSOCIATES Results of the Event An investigative committee was formed to determine why the incident occurred. The committee membership was internal to the laboratory and external to our directorate The result was a report listing 12 “Judgments of Need” The report was reviewed by a “red team” consisting of members internal and external to the laboratory. The response was a Corrective Action Plan to address all of the needs This plan affected the entire the directorate, and in one instance the laboratory as a whole. Completing the CAP was a requirement prior to restarting the linac and commissioning the booster. I’ll discuss those things that affect operations directly.

9 BROOKHAVEN SCIENCE ASSOCIATES Impacts on Operations Organizational Changes in the control room Procedures Training Shielding configuration controls Area Radiation Monitors Configuration control Response to alarms Accelerator Readiness Review Process

10 BROOKHAVEN SCIENCE ASSOCIATES Control Room Prior to the event: Control room had an operator and a “knowledgeable person”. The knowledgeable person was to respond to alarms in the absence of the operator and to act as a second set of hands. The Operator was in charge. There could be confusion about who was in charge if two operators were assigned a shift. Operator reported to commissioning coordinator Commissioning coordinator made the shift plan After the event Two flavors of operator. –Lead Operator (operations personnel) –Scientific Operator (scientists) At least one lead operator of shift. Someone is always THE lead operator – the person ultimitely responsible for the shift. This is recorded in the log. The Lead Operator is responsible for the control room and all machine operations. All other personnel in the control room report to Lead Operator. The scientific operator can fill in for lead if lead needs to step out of control room. Lead Operator reports to Beam Operations Group Leader only. Beam Operators Group Leader approves all shift plans, even if written by a machine manager

11 BROOKHAVEN SCIENCE ASSOCIATES Procedures All procedures are now in accordance with the “Procedure on Procedures” which specifies: Common format Review process Review Panel (QA, training, ESH, USI, upper management minimum) Procedure validation (can the procedure be done as written by someone who may do the procedure) Minor Changes Minimum Review Frequency This was somewhat ad hoc before. This has improved the quality and usability of the procedures This is a good thing since the number has grown as well! There are occasional audits from Quality Assurance to see that procedures are followed. This helps find: Lapses in following procedures Procedures that need revision. This is not a perfect process, but it is a good start.

12 BROOKHAVEN SCIENCE ASSOCIATES Training All procedures get trained on by all persons who may have a role in a procedure Lab wide courses may be classroom (RadWorker) or web course (Fire Extinguisher, LOTO) Read and acknowledge is the minimum required training on all procedures. Complicated procedures (startup/ shutdown) or safety related (Response to Radiation Alarms, Emergency Response) require more extensive training. This is usually classroom, or sometimes performing the procedure. Training is tracked in the lab’s training database. This means: You are notified when your training is to expire. When a new procedure issued. Weekly reminders to you AND your supervisor of out of date training. –One weakness is that the Beam Operations Group Leader does not get automatic notificationn of the scientific operators. Some of the operator training includes a test. Mostly on safety related procedures. There are still weaknesses here: There are 100’s of procedures – does the training “stick”? In my opinion, the most important thing one learns is that the procedure exists. To quote a professor of mine, “The most important thing I learned in school is where to look up the answer.” If a person writes, reviews, or even trains people on a procedure, they are not considered trained until they complete the training. I wrote the linac startup procedure, and trained operators on it, however I needed to attend my own class! Retraining on a procedure is necessary for the smallest of changes. I once trained on a procedure where the change log on the procedure was “grammatical changes”. To be fair, these weaknesses are recognized.

13 BROOKHAVEN SCIENCE ASSOCIATES Shielding One of the outputs of the report, and the immediately obvious cause is that the local shielding was not adequate. The responsibility for shielding was spread out among groups within the directorate and did not report to a single line of management. In my opinion, the largest flaw in shielding design itself prior to the event is that the folks involved considered “operations” and not commissioning or other “special configurations” of the machine. (guilty) A working group was formed evaluate the shielding facility wide. The working group reported to the Accelerator Division Director. Our philosophy is that if the beam could be steered to a location, no matter how improbable, and a shield was needed, that location got a shield. So when reviews said, “Would the operator really do that?” The answer was, “The operator CAN do that, even if they don’t intend too. Yes, we agree the chances of this happening may be very small, and requires a unique configuration of magnets that one would ‘never’ encounter in operators. It is a valid configuration of the magnets.” Now the operator has more confidence that the shielding is adequate. Shielding is reviewed by the Radiation Safety Committee. An internal committee with some external members. Included are accelerator physicists, beamline physicists, Beam operations group leader, ESH, Radiation controls, Radiation Safety Committee chair in C-AD, among others. It is a very broad committee. Shield construction is tracked via traveler, so that there is verification that the stack matches the design. The shielding is under configuration control, and prior to running the shields are checked to be in place in accordance with a posted photograph. The checklist is posted in the control room. Operators or machine coordinators perform this task. Work requiring moving of a shield is governed by a “Safety System Work Permit”. A lead operator signs this permit and it is posted in the control room. This permit usually requires some sort of LOTO on a device which inhibits the beam. Operators is usually not on the LOTO, but ESH is always the last on, first off.

14 BROOKHAVEN SCIENCE ASSOCIATES My Favorite Shield

15 BROOKHAVEN SCIENCE ASSOCIATES Area Radiation Monitors Prior to the event, we had two ARMs which were not part of the interlock system, and were “only” for local warning to those working near them. The controls system readback of these alarms was not originally envisioned and had a software bug which limited the maximum dose rate readback. Note: They did work as originally designed during the event. There were 2 other ARMs in the personnel protection system, but they did not see dose during the event. Now all ARMS are interlocked unless specifically placed for fault studies. ARMs were always routinely calibrated and tested in the PPS system, but now the controls system readback is tested. Their configuration control is handled exactly like shielding. Inspection Photo Checklist ARMs have three alarm levels, both of which alarm through the PPS system. Each has an immediate action. Low Alarm – inhibits the linac gun via the controls system. High Alarm – disables the election beam via a number of devices. Fail – something is wrong with the monitoe Operator response is governed via procedure. Response is different depending on the alarm level. Make sure the beam is off. Find out why the alarm occurred and mitigate the condition Send notifications to machine managers Determine if personnel are in the area In case of high alarm, obtain approval of Beam Operations Group Leader prior to restarting

16 BROOKHAVEN SCIENCE ASSOCIATES Accelerator Readiness Review Process The Dept. of Energy requires a review prior to startup of an accelerator to ensure that the organization is ready to start. This is the Accelerator Readiness Review (ARR) Prior to the event BNL’s process (SBMS) specified that the following were necessary: Safety Systems Documentation and Procedures Trained Operators. Notice how the actual accelerator is missing…… The original ARR committee was solely ESH people, who reviewed documentation and saw presentations in one day. The event led to changes in the ARR process, starting with SBMS Now the process really requires readiness (including safety system, documentation, trained operators and the accelerator). The committee is very broad, even including an accelerator physicist. The review is almost a week long. The committee is “hands on” One member prided himself on touching every shield in the linac and booster. The committee interviews staff, especially operators. In the latest review, operators were interviewed on 3 separate occasions.

17 BROOKHAVEN SCIENCE ASSOCIATES Conclusion On May 29, 2012, a beam mis-steering event took place at the BNL National Synchrotron Light Source II Linac during high intensity commissioning activities. This event led to a number of changes to how we operate NSLS-II. Linac restart and Booster Commissioning started November 27, 2013 Booster Commissioning and Storage Ring Commissioning are complete. Insertion Device and Front End commissioning started October 3, First light was on October 23, 2014 to the CSX beamline. Commissioning continues on the remaining Insertion Devices and Damping Wigglers.