Maintenance and Efficiency Paul W. Sampson Maintenance and Accelerator Support Group Leader: CAD/RHIC Brookhaven National Laboratory

Short Background CAD at BNL in Upton New York, USA RHIC Large superconducting collider 500GeV cm for polarized proton running Species to Au (U this year?) Dissimilar species Many “parasitic” users on injectors: LINAC Tandem Van de Graff EBIS Booster AGS April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

Maintenance Support Group Role Plan and coordinate work throughout the accelerator complex. Emphasis on operations and uptime. Integrate Professional Trades, Technical, Engineering and Physics staff activities during shutdown, failure (unscheduled maintenance) and scheduled maintenance periods. April 13th, 2011 Paul W. Sampson

Goals: Maximize uptime and optimize accelerator output Integrated Luminosity is king? STAR, PHENIX, Overall integrated luminosity. What cost for improvement? Parameters vary for varying species in RHIC Use and maintenance of different systems. Differing beam parameters. Safety/Radiation issues- affects on access and equipment availability. New systems commissioning. Internal goals: Beam parameters. Available time at store, setup minimization Calendar time at store. April 13th, 2011 Paul W. Sampson

Reliability formula complicated End of life. Installation and commissioning. Infant mortality “Roll up period” Legacy systems. April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

Shutdown Annual occurrence. Duration factors: Budget. Major work schedule. Experiment upgrades/readiness. Climate. Documented utilizing data base tool. Progress tracked and Project published. Goals Complete shutdown work. Prepare for structured pre-startup checkout. Minimize start up (time to Physics). April 13th, 2011 Paul W. Sampson

Unscheduled Maintenance Major failure or other prolonged downtime. Goal of maximizing use of time and access while minimizing the recovery. Coordinated by Maintenance Support Group. Schedule are created as necessary, depending on length of downtime. Each day run similar to a scheduled maintenance day. April 13th, 2011 Paul W. Sampson

During the run: “Minor” Failure (>4hours) Small low impact jobs to be performed. “Fast” experimenter access Coordinated by on duty Operations Coordinator, with assistance and advice from Maintenance Support Group as needed. Documented by standard methods. April 13th, 2011 Paul W. Sampson

During the run: Scheduled maintenance days. Usually 8-12 hours, biweekly. Driven by experimental access. Accelerator access for: Preventative Maintenance. Repair Modification Installation Commissioning. Shutdown planning Inspection/work observation Continuation of shutdown/installation work. April 13th, 2011 Paul W. Sampson

How maintenance days directly impact availability… Scheduled maintenance days demand that the program is off. Many required tasks require prolonged access. Some systems have long lead time before work can be performed. Source cool down. Cryogenic warm up. Radiation cool down periods. Setup and recovery after maintenance period. Human error: Systems left in inoperable state following maintenance period. Failure of improperly maintained equipment or systems. April 13th, 2011 Paul W. Sampson

Reliability and Maintenance days: Preventative maintenance periods should be focused on reducing failures. Repairs to systems and infrastructure. Accelerator tunnel/Radiation area access: PM and repair Installation, testing and commissioning of new and modified systems. Low intensity test cycles following maintenance period: Used for machine development. Tests and implementation modified and modes before they are adapted as operational. Scheduled as part of the maintenance period. April 13th, 2011 Paul W. Sampson

Reliability and maintenance days Maintenance day goals: Complete tasks necessary to maintain current mode of running at the accelerator complex. Prepare for any new running conditions expected. Minimize the total duration and the time from end of maintenance to a return to smooth running. Some tasks performed can cause unstable running and increased failure in the days that follow maintenance periods. Vacuum bleed-up. Ion source maintenance. Cryogenic. RF systems. Super Conducting power supply systems. April 13th, 2011 Paul W. Sampson

Reliability after maintenance: New running modes can cause increased failure for various reasons: More aggressive magnetic cycles. Increased high voltage. Additional interlocks. Re-integration of repaired systems: Feedback elements. Interlocking devices- loss monitors, chipmunks/radiation monitors, beam permit or inhibit systems. Failure of new or un-commissioned systems: Apertures in the beam pipe. Improperly grounded devices. Miss-configured interlocks or inhibits Miss-configured devices (current loops). April 13th, 2011 Paul W. Sampson

Maintenance methods for increased reliability Scheduled Maintenance Days Job Requests Benefit and impact analysis Prioritization Individual work planning Overall work planning and scheduling System recovery Accelerator recovery Return to physics April 13th, 2011 Paul W. Sampson

Before maintenance days Determine which tasks are actually necessary. Avoid low return, high risk jobs. Carefully document accelerator status before turn off. Define parameters for recovery. Schedule accesses for efficiency- 4 hours to secure accelerators. Communicate with all groups Scheduling meeting, web, CCTV, face to face … April 13th, 2011 Paul W. Sampson

Accelerator Physics/Administration Experimental Support and Facilities Maintenance Support Accelerator Physics/Administration Experimental Support and Facilities Accelerator systems Integrated Facility Management

Job Request: Create April 13th, 2011 Paul W. Sampson

Pic of list April 13th, 2011 Paul W. Sampson

Work planning: Individual jobs are planned by cognizant techs or engineers. Job request is submitted to the MSG for review. Primary approval given and schedule created. Final review with upper management, experimenters and run coordinators. Staff briefing. Schedule posts. April 13th, 2011 Paul W. Sampson

Schedule: Time Activity Personnel By 0600 Turn down OPPIS source for maintenance (After store to last to 0800hrs has been established) MCR 0600 Apply CA LOTO to AGS AGS Maintenance Begins (items outside the ring) CAS CAD 0800 RHIC areas to RA, RHIC Maintenance Begins LINAC Maintenance begins MSG LINAC 0900 CA Access into the AGS MCR/CAS 1100 AGS Access complete 1130 RHIC sweeps begin in select areas 1400 PHENIX tunnel sweep 1430 Hysteresis ramp, PHENIX IR sweep 1500 Physics setup in RHIC MCR/AP… April 13th, 2011 Paul W. Sampson

Submission: Individual Jobs Initial review: Task Analysis, job approval (denial) Initial Schedule Proposal Final Schedule publication, review with staff Execution of scheduled and Emergent tasks System Recovery, Test ramp Return to Physics April 13th, 2011 Paul W. Sampson

Execution Complex formally handed over to Maintenance Support Group MCR staffing: Maintenance Coordinator (MC) to run activities, normally from MCR. Utilize Operations Coordinator, Operators and support staff. Leads keep MC informed of progress, emergent issues and job completion. MSG leader appraised of major change, facilitates review, approval and schedules changes as necessary. Maintenance coordinator executes changes. April 13th, 2011 Paul W. Sampson

Communication: During the maintenance, it is communication is essential. CCTV Web, elogs, Job request system. Radio, pagers and telephone… April 13th, 2011 Paul W. Sampson

End of maintenance: Job closeout statements reviewed by MC before final closure. Tiered access allows for recovery of systems to begin as soon as they are ready. Injectors RF Power supply Vacuum Instrumentation Specialist confirms with MSG that systems have been returned to operational condition. Specialist remains on alert until program is recovered April 13th, 2011 Paul W. Sampson

Return to Physics Accelerators are returned to operating condition as available. Schedule work accordingly. After injectors have recovered, RHIC test ramp performed. Structured hand off to MCR for routine operation. Maintenance Coordinator’s Journal closeout. Operations Coordinator’s journal entry. Maintenance day summary/closeout. April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

Results Since structured systems introduction in 2005: The methods described have produced a marked improvement in performance on and after maintenance periods. Return to productive running has steadily improved. In many instances, machine performance is actually better immediately following maintenance periods. Could maintenance have actually helped??!! April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

Maintenance Effectiveness Trend April 13th, 2011 Paul W. Sampson

Other comments: Maintenance and repairs done behind physics stores. Software releases. Systems maintenance: BPMs. Systems commissioning, improvement. Studies following Maintenance days. Dual mode days- Maintenance on one machine and science on another. April 13th, 2011 Paul W. Sampson

Acknowledgements: Availability, “Top Ten” and integrated luminosity plots adapted from presentation “Operating Statistics” by P. F. Ingrassia, Brookhaven National Laboratory. “RHIC from space” courtesy of BNL Photo archives. April 13th, 2011 Paul W. Sampson

Thank you April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson

April 13th, 2011 Paul W. Sampson