2016 Operations StayTreat 4-Hall Ops and Operations: What is the Control room vision for 4-Hall Ops?

4-Hall Ops Vasilauskis 2016 Ops StayTreat 2 2

4-Hall Ops Vasilauskis 2016 Ops StayTreat 3 Can we support running 4 halls simultaneously? Yes, we're Operations, we're accustomed to doing more with less! Will it be easy? Depends. On....? The phase of operations, whether it is: Initially starting up 4 halls and the accelerator at the very same time (worse case) vs 4 halls up and running while handling normal shiftly items like RF trips, the occasional Controlled Access... 3

4-Hall Ops Vasilauskis 2016 Ops StayTreat 4 4 So it can be done? True, but one can not simply increase the number of customers by 33% and not expect repercussions. Something somewhere has to give. Such as... Staffing levels are set at two operators and a Crew Chief. Each operator is generally only able to work one hall at a time. At the start of a four hall setup, there will be two halls waiting while the other two halls get setup.

4-Hall Ops Vasilauskis 2016 Ops StayTreat 5 5 Can't the Crew Chief help out and set up a hall? The Crew Chief is responsible for overall operations and safety. They will be watching over both operators, providing guidance, keeping things on track, interfacing and coordinating with other groups... in short, No! Can't you bring on more staff from your off-shift operators to help in the crunch? It may appear to be option, but operators already work a 70% on-shift, 30% off-shift schedule. That 30% is already used to cover for those on vacation, out sick, on travel as well as those missing from the rotation due to attrition or replacements not yet fully qualified.

4-Hall Ops Vasilauskis 2016 Ops StayTreat 6 6 What would you suggest? It might be best if a hall or two anticipated this in their scheduling and used the time to their advantage instead of waiting for us to get to them. i.e. don't schedule 4 halls starting at the same time and then expecting it to happen without some delays. You know that ain't gonna happen. Well, when I researched for this presentation I compared the Physics schedule to actual logbook entries and it showed that most times at least one hall was not scheduled or was not able to come up. It wasn't until 26 Oct that I could find 3 halls being restored together from being completely down.

4-Hall Ops Vasilauskis 2016 Ops StayTreat 7 7 So that's worse case, what about normal 4 hall operations? Quiet operations, the “Steaming along, fat, dumb and happy” just resetting RF trips type of operations?, Those don't generally see significant change whether you have 1, 2, 3 or 4 halls running. However, going beyond that, when things go down or the optics needs tuning and it starts to get busy, that's when the lower operator to hall ratio will get noticed.

4-Hall Ops Vasilauskis 2016 Ops StayTreat 8 Why is that? With four halls and two operators, only one half of the halls can receive any special attention at a time. This special attention can be in the form of optics tuning, harp scans, current calibrations, energy measurements, Moller measurements, surveys for access... The procedure itself may not be invasive to the other halls but the occupying of a limited operators time may be. 8

4-Hall Ops Vasilauskis 2016 Ops StayTreat 9 When there is a problem with the main accelerator and the beam is down, the users generally want to take advantage of the time and will ask for a Controlled Access. With one operator as SSO and the other as ARM, only one hall can be surveyed at a time. Full surveys take time, escorted surveys monopolize the solo ARM the whole time. Even if one considers Hall B out of the picture with its fully operational Rapid Access system and thus no need for a survey. Hall D makes up for that by being on the opposite side of the accelerator and having two locations that could be in need of a survey. 9

4-Hall Ops Vasilauskis 2016 Ops StayTreat 10 Can't the SSO be another ARM and the Crew Chief be the SSO? Again, Crew Chiefs should be just that, Crew Chiefs. We have had Notable Events when Crew Chiefs got directly involved and played SSO instead of being a safety observer. Besides, if both operators are ARM and the Crew Chief is SSO, who's working the problem causing the accelerator to be down? 10

4-Hall Ops Vasilauskis 2016 Ops StayTreat 11 What about hiring additional operators? Adding one additional operator per shift would raise the Ops staffing level to 25. Wow, that's a lot! Not really. Back in 1995 Operations had 26 staff members. Interestingly three hall, 5-pass 4 GeV operations didn't happen until Operations historically has an average staff level of That has level not been seen since In 2010 Operations began a decline in staffing ending at the current level of 19. (this is where the more with less I opened with came from)

4-Hall Ops Vasilauskis 2016 Ops StayTreat 12

4-Hall Ops Vasilauskis 2016 Ops StayTreat 13 So what does adding one more operator per shift get us? You double the ARM support for surveys from one to two while still leaving an operator as SSO and the Crew Chief as the Crew Chief. For set ups you increase the number of halls being set up at the same time by 50% That “last hall waiting” now only has to wait for only one hall to finish being set up, where as before with four halls and two operators, that last hall had to wait for two halls to finish. 13

4-Hall Ops Vasilauskis 2016 Ops StayTreat 14 Do we get anything else? Having an extra operator also sets us up for simultaneous CEBAF and LERF operations. This is a concern because a CEBAF Operator can not be a LERF Operator at the same time. Running LERF and CEBAF forces a 3 rd operator to be on shift. If a LERF Laser Operator is required, the LERF Operator can be the Laser Operator but only if they qualified, if not, now we need a 4 th operator. (5 staff total with the Crew Chief) Then there is the possibility of UITF operations someday. 14

4-Hall Ops Vasilauskis 2016 Ops StayTreat 15 Why don't we just wait and see if we need more operators? It takes time to built an operator. First is the overhead of creating the position and having it approved, posting it, waiting for applications, going through the interview process and the hiring process. Operators have the most safety training of anyone on site. In addition there is their own online training that is continuously growing. This all takes ~2 months. To be considered an operator they need to qualify ARM and/or SSO, both of which take several months of OJT. This is just the bare minimum. To become a comfortable, competent operator takes 1 to 2 years. 15

4-Hall Ops Vasilauskis 2016 Ops StayTreat 16 Besides people, any other 4 Hall Ops concerns? The display wall is only half of what it used to be in the 3 hall 4 to 6 GeV era. We have yet to determine the full impact this will have. 4 laser setup, new screens, new controls, current locks, 4 lasers through 3 slits, 750MHz separator ops... Anything else? Not that I can think of, so I guess this is the point where we begin a discussion. 16