1 MICE Running L. Coney – UCR MPB – 31 Oct 2012

2Coney - MPB - 31 Oct 2012Outline Current Running Current Running Shift personnel training Shift personnel training Organization & Preparation Organization & Preparation Moving toward Step IV Operations Moving toward Step IV Operations Organization Organization Operational plan Operational plan Online systems Online systems Conclusions Conclusions

3Coney - MPB - 31 Oct 2012 Leading Into Step IV Current running experience feeds into Step IV/Step VI preparation Current running experience feeds into Step IV/Step VI preparation Characterize muon beam & Understand detector operation Characterize muon beam & Understand detector operation Gain experience operating beam line and Hall equipment Gain experience operating beam line and Hall equipment Current run personnel Current run personnel MOM (MICE Operations Manager) MOM (MICE Operations Manager) Rolling monthly appointment Rolling monthly appointment Responsible for meeting scientific goals of MICE Responsible for meeting scientific goals of MICE Safety responsibility delegated from Project Manager Safety responsibility delegated from Project Manager BLOC – BeamLine On Call expert BLOC – BeamLine On Call expert Trained member of MICE collaboration Trained member of MICE collaboration 2 shifters for ~9 hour shift 2 shifters for ~9 hour shift Trained member of MICE collaboration Trained member of MICE collaboration SOC – Software On Call expert – often remote SOC – Software On Call expert – often remote Procedures Procedures Run during ISIS User Runs Run during ISIS User Runs Normal working hours (plus weekends) Normal working hours (plus weekends) Preparation Preparation

4Coney - MPB - 31 Oct 2012 Shifter Training Implemented training regime (May) Implemented training regime (May) Systematic, consistent, good coverage for all shifter responsibilities Systematic, consistent, good coverage for all shifter responsibilities Record of training kept by MOM Record of training kept by MOM MOMs lead/arrange training MOMs lead/arrange training Overview & introduction – given by MOM Overview & introduction – given by MOM Introduces responsibilities and role of Shifters Introduces responsibilities and role of Shifters Notification of safety training requirements Notification of safety training requirements Description of shifter duties Description of shifter duties Practical training – ends with practical test Practical training – ends with practical test Work with the equipment Work with the equipment List of tasks must learn how to complete List of tasks must learn how to complete Specific individuals are certified to train shifters on systems Specific individuals are certified to train shifters on systems PPS, Target, Beamline magnets, C&M, DAQ, Online Reconstruction and Monitoring, Datamover, Computing, Documentation PPS, Target, Beamline magnets, C&M, DAQ, Online Reconstruction and Monitoring, Datamover, Computing, Documentation

5Coney - MPB - 31 Oct 2012 Organization & Preparation Redmine project management tool Redmine project management tool Micemine Micemine Provides useful & necessary Operations info Provides useful & necessary Operations info Organize running Organize running Schedule shifters, SOC, BLOC Schedule shifters, SOC, BLOC Develop Run Plan Develop Run Plan Daily to-do Daily to-do Beamline configurations Beamline configurations ac.uk/projects/operati ons/wiki ac.uk/projects/operati ons/wiki ac.uk/projects/operati ons/wiki ac.uk/projects/operati ons/wiki

6Coney - MPB - 31 Oct 2012 Regular Running Implement changes Implement changes Operate equipment long idle Operate equipment long idle Practice  learn Practice  learn Train shift personnel Train shift personnel NEED coordination of vital personnel NEED coordination of vital personnel MAUS software expert prepare control room branch for running MAUS software expert prepare control room branch for running ‘experimenters’ and associated personnel available ‘experimenters’ and associated personnel available Shake down of systems in advance Shake down of systems in advance Software/on call experts/remote availability Software/on call experts/remote availability If responsible for a system – must be contact-able If responsible for a system – must be contact-able

7Coney - MPB - 31 Oct 2012 Moving Forward.. Shift focus from basic functionality to reliability and efficiency Shift focus from basic functionality to reliability and efficiency Continue: Continue: Pre-run preparation Pre-run preparation Further simplify running Further simplify running Improve: Improve: Automate procedures Automate procedures Use ConfigDB to full capability Use ConfigDB to full capability Increase Online Reconstruction capabilities Increase Online Reconstruction capabilities Improve stability in Online Systems Improve stability in Online Systems MAUS – already using ‘release’ system of changes MAUS – already using ‘release’ system of changes C&M – Implement more formalized release for C&M code C&M – Implement more formalized release for C&M code DAQ – staged upgrades – no major changes planned DAQ – staged upgrades – no major changes planned Prepare for long term running in Step IV Prepare for long term running in Step IV

8Coney - MPB - 31 Oct 2012 MICE Step IV Change operational mode Change operational mode Define for rest of experiment (Step IV, Step VI) Define for rest of experiment (Step IV, Step VI) Equipment: Equipment: Both Spectrometer Solenoids Both Spectrometer Solenoids Two trackers installed in the SS magnets Two trackers installed in the SS magnets One AFC (Focus Coil magnet & LH2 system) One AFC (Focus Coil magnet & LH2 system)

9Coney - MPB - 31 Oct 2012 Step IV: New Equipment… We will have much more in the Hall! We will have much more in the Hall!

10Coney - MPB - 31 Oct 2012 Step IV: Operations Moving toward Step IV running Moving toward Step IV running Many configurations to test/run Many configurations to test/run Long term running/every opportunity taken Long term running/every opportunity taken All ISIS cycles – every day All ISIS cycles – every day Develop idea of not-so-remote control room Develop idea of not-so-remote control room Develop robust system of Operations Develop robust system of Operations Includes all facets of Online Group Includes all facets of Online Group DAQ, C&M, Online Reconstruction/Analysis, Infrastructure DAQ, C&M, Online Reconstruction/Analysis, Infrastructure Develop plan for cooling channel commissioning Develop plan for cooling channel commissioning Develop operational support plan with new personnel Develop operational support plan with new personnel

11Coney - MPB - 31 Oct 2012 Step IV Operations Organization Overall Operations Manager Overall Operations Manager Understands STFC safety and operational environment Understands STFC safety and operational environment Provide oversight, link between MOMS, safety responsibility from project manager Provide oversight, link between MOMS, safety responsibility from project manager Local – full time – not necessarily STFC employee Local – full time – not necessarily STFC employee Position does not currently exist Position does not currently exist Monthly MOMs Monthly MOMs Continues current role – rolling appointment Continues current role – rolling appointment Responsible for daily experimental planning & running duties Responsible for daily experimental planning & running duties On call for 1 month On call for 1 month MICE Shift personnel MICE Shift personnel Trained MICE collaborators Trained MICE collaborators Two shifters for each ~8 hour shift Two shifters for each ~8 hour shift Current on-call experts continue Current on-call experts continue BLOC (BeamLine), SOC (Software), TROC (Tracker) BLOC (BeamLine), SOC (Software), TROC (Tracker) Covers conventional magnets, detectors, and PPS Covers conventional magnets, detectors, and PPS

12Coney - MPB - 31 Oct 2012 Step IV Operations Organization STFC organizational and safety requirement that qualified experts are on premises when hazardous operations are carried out STFC organizational and safety requirement that qualified experts are on premises when hazardous operations are carried out New system-specific experts New system-specific experts Minimum of three needed Minimum of three needed Not necessarily current MICE Not necessarily current MICE Positions do not currently exist Positions do not currently exist Cryogenic support Cryogenic support Magnet system maintenance, management of cryogens, vacuum pumping rigs, instrumentation Magnet system maintenance, management of cryogens, vacuum pumping rigs, instrumentation Superconducting magnet diagnostics Superconducting magnet diagnostics Cooling & powering expert, quench behavior expert Cooling & powering expert, quench behavior expert LH2 expert LH2 expert Control engineering expertise, safety procedures Control engineering expertise, safety procedures During LH2 system commissioning – 3 experts rotated shifts During LH2 system commissioning – 3 experts rotated shifts

13Coney - MPB - 31 Oct 2012 Step IV Operations Experimental program preliminary – 1 full year data-taking for Step IV Experimental program preliminary – 1 full year data-taking for Step IV Includes time for commissioning SS1/FC/SS2 together Includes time for commissioning SS1/FC/SS2 together Not need 24-hour coverage  need on-call magnet expert Not need 24-hour coverage  need on-call magnet expert Run with empty FC  solid absorbers  liquid hydrogen absorber Run with empty FC  solid absorbers  liquid hydrogen absorber For LH2 – need 24-hour support For LH2 – need 24-hour support Given current MICE schedule for Step IV equipment and installation..We must be ready for 24/7 running in order to get Step IV data before ISIS shutdown in August 2014 Given current MICE schedule for Step IV equipment and installation..We must be ready for 24/7 running in order to get Step IV data before ISIS shutdown in August 2014 Prepare: Prepare: Train shift personnel Train shift personnel Develop plan to staff multitude of shifts Develop plan to staff multitude of shifts Can our systems handle this? Can our systems handle this? Target ok? Magnets ok? Target ok? Magnets ok? DAQ ok? Controls ok? Datamover ok? DAQ ok? Controls ok? Datamover ok?

14Coney - MPB - 31 Oct 2012 Step IV & VI – Operations Safety Safety Understand implications under STFC rules of new equipment Understand implications under STFC rules of new equipment Operations review of LH2 system under way Operations review of LH2 system under way Understand from experts what each system entails/requires for commissioning & operation Understand from experts what each system entails/requires for commissioning & operation PPS PPS Understand implementation of system with SC magnets Understand implementation of system with SC magnets Step VI – understand implementation with RF Step VI – understand implementation with RF Hall Access – limited Hall Access – limited Magnet commissioning period differs from running Magnet commissioning period differs from running Step VI – includes everything from Step IV + RF Step VI – includes everything from Step IV + RF Need full time RF engineer Need full time RF engineer Daresbury interviewing for this position Daresbury interviewing for this position

15Coney - MPB - 31 Oct 2012 Step IV: Operations & Online Next: focus  Functional, Safe/Secure, Reliable, Easy to use Next: focus  Functional, Safe/Secure, Reliable, Easy to use Functional Functional DAQ – incorporate all detectors – improve rate – DAQ upgrades DAQ – incorporate all detectors – improve rate – DAQ upgrades All C&M for each element All C&M for each element Commissioned LH2 controls & monitoring Commissioned LH2 controls & monitoring Fully commissioned SS controls system Fully commissioned SS controls system Fully commissioned FC controls system – incorporate with LH2 system Fully commissioned FC controls system – incorporate with LH2 system Alarm Handler values set appropriately Alarm Handler values set appropriately Add capability to Online Reco/Analysis Add capability to Online Reco/Analysis Automate run infrastructure – Run Control Automate run infrastructure – Run Control Automate read/write to Configuration Database Automate read/write to Configuration Database Automate write to EPICS archiver Automate write to EPICS archiver Safety/Security Safety/Security Incorporate fully commissioned PPS Incorporate fully commissioned PPS Implement formal shifter training Implement formal shifter training Update Operations/Online documentation & instructions Update Operations/Online documentation & instructions Develop comprehensive list of safety-critical maintenance Develop comprehensive list of safety-critical maintenance Update safety paperwork – develop overall system for MICE operations Update safety paperwork – develop overall system for MICE operations New target controller with all interlocks & Beam Protection System (BPS) New target controller with all interlocks & Beam Protection System (BPS) Online system security reviewed – improvements made Online system security reviewed – improvements made Access limited to micenet Access limited to micenet

16Coney - MPB - 31 Oct 2012 Step IV: Operations & Online Easy to use Easy to use Simplified Ops procedures – single key exchange ISIS Simplified Ops procedures – single key exchange ISIS MICE control access to Hall MICE control access to Hall Improving Online Reconstruction/Data Quality plots Improving Online Reconstruction/Data Quality plots Add Online Reco for KL/EMR/accelerator analysis Add Online Reco for KL/EMR/accelerator analysis Run Control Run Control Mature Alarm Handler Mature Alarm Handler Reliable Reliable New OS implemented – automatic updates New OS implemented – automatic updates Automatic/systematic backups in place Automatic/systematic backups in place New DAQ and C&M computers installed New DAQ and C&M computers installed UPS coverage for critical systems UPS coverage for critical systems Develop “stockroom” of spares – crates/computers/etc Develop “stockroom” of spares – crates/computers/etc Automatic write to EPICS Archiver Automatic write to EPICS Archiver Extensive remote monitoring of equipment/computers/hardware/environment in EPICS Extensive remote monitoring of equipment/computers/hardware/environment in EPICS Archive of all software – Launchpad repository Archive of all software – Launchpad repository Implement MAUS releases for Online Reconstruction Implement MAUS releases for Online Reconstruction Implement formal releases of C&M code Implement formal releases of C&M code

17Coney - MPB - 31 Oct 2012Conclusions Step I running has been good preparation/practice for future Step I running has been good preparation/practice for future Many changes in store for Step IV Many changes in store for Step IV New equipment, new procedures, long term running New equipment, new procedures, long term running Many challenges remain before ready for Step IV running Many challenges remain before ready for Step IV running Implementing personnel requirements Implementing personnel requirements Staffing shifts Staffing shifts 24/7 running – equipment & personnel 24/7 running – equipment & personnel

18Coney - MPB - 31 Oct 2012

19Coney - MPB - 31 Oct 2012 Shifter Training Trained collaboration members to be qualified shifters (intro role) and Lead Shifters (more advanced) Trained collaboration members to be qualified shifters (intro role) and Lead Shifters (more advanced) Recent running has made the need for more qualified/trained individuals very clear Recent running has made the need for more qualified/trained individuals very clear Even more true for Step IV running Even more true for Step IV running Difficult to do while running – difficult to properly train w/o running Difficult to do while running – difficult to properly train w/o running Develop & implement training schedule Develop & implement training schedule Corollary – schedule training updates when new equipment arrive Corollary – schedule training updates when new equipment arrive