Streamlining CDF Shift Operations Peter Wilson Mary Convery CDF Detector Operations All Experimenters Meeting February 26, 2007

Peter Wilson - All Expt Meet February 26, 2007 slide - 2 Challenge: collect high quality data with high efficiency as the pool of physicists shrinks!  LHC will turn on in FY2008  Many people working on CDF also on ATLAS or CMS  Expect fewer Postdocs and students in 2008/2009 l However, we don’t need as many people as in 2001:  Experiment is mature  Systems are well understood  Procedures are well defined  Upgrades/changes are minimal l Effort initiated by previous Ops heads (Willis Sakumoto and Pat Lukens) to re-organize shift operations to use fewer people while maintaining or increasing operational efficiency Running CDF thru FY2009

Peter Wilson - All Expt Meet February 26, 2007 slide - 3 CDF Shift Crew - Ops Man Operations Manager (Ops Man) - On call 24/7 l Makes daily operational plan l Typically experienced postdoc - 1 wk on/2 wks off for 6 months CDF Control Room

Peter Wilson - All Expt Meet February 26, 2007 slide - 4 CDF Shift Crew - SciCo Scientific Coordinator (SciCo) - 8/7 x 3shifts l Leader of the shift crew l Responsible for implementing daily plan l Communicate with MCR l Summon expert help l Provide food for the crew! l Faculty, lab scientific, staff or senior postdoc CDF Control Room

Peter Wilson - All Expt Meet February 26, 2007 slide - 5 CDF Shift Crew - DAQ Ace DAQ ACE - 8/7 x 3 shifts l “Runs” the experiment l Controls the data acquisition system  Setup run configurations  Start/stop DAQ runs l Monitor DAQ status, diagnose and fix DAQ problems Run calibrations CDF Control Room

Peter Wilson - All Expt Meet February 26, 2007 slide - 6 CDF Shift Crew - Mon Ace Monitoring ACE - 8/7 x 3 shifts l Detector HV monitoring and control. Diagnose/fix HV problems l Monitor status of low voltage power supplies l Responsible for safety of Si in the event of a cooling alarm l Monitor beam conditions for detector safety l Twelve Students and postdocs cover DAQ & Monitoring Shifts in 10 week rotation CDF Control Room

Peter Wilson - All Expt Meet February 26, 2007 slide - 7 CDF Shift Crew - CO Consumer Operator - 8/7 x 3 shifts l Data quality monitoring for all detector subsystems l Ten “Consumers” each receiving a few Hz (of ~100Hz) of events from L3  Detector hit occupancies and average pulse-heights in minimum bias data  Trigger rates and trigger algorithm validation (compare emulation to data)  DAQ performance and errors l Compare results to references l Check all routine detector calibrations Any CDF physicist CDF Control Room

Peter Wilson - All Expt Meet February 26, 2007 slide - 8 CDF Shift Crew - Cryo Process System (Cryo) Operator - 12/7 x 2 shifts l Solenoid and Cryogenic System operation l Gas systems l Cooling systems for electronics and detectors l Electronics rack protection Four PPD technicians on rotation, one other tech fills in for sick leave/vacations CDF Cryo Control Room (1st Floor)

Peter Wilson - All Expt Meet February 26, 2007 slide - 9 Streamlining the Shift Crew l Consumer Operator Shifts taken remotely  Pilot operation at INFN-Pisa  Reduce need for travel for overseas members of CDF  “OWL” shift is 7:00-15:00 in Pisa, Italy or 15:00-23:00 in Tsukuba, Japan l Reduce the number of Aces from 2 to 1  Consolidate responsibilities with the remaining ACE  Transfer some monitoring tasks to Cryo Operator  SciCo trained on some tasks of monitoring Silicon detector - Safety of detector if Ace is not in control room (eg high beam losses)  Automate tasks to allow shift crew to focus on decision making

Peter Wilson - All Expt Meet February 26, 2007 slide - 10 Remote CO l Effort started early in 2006:  All monitoring plots available via Web  Tools for remote starting and stopping of Consumer programs  CO shift room in Pisa: monitoring workstations and video conferencing  CDF  Pisa Video conference for SciCo-CO communication  Remote CO does all of the same functions as local CO  Experience shared with and gained from US CMS remote operations  Team from Fermilab, Pisa, Tsukuba, UCLA and Waseda led by Kaori Maeshima and Fabrizio Scuri l First official remote shift: week of November 10  Commissioning in October 2006: simultaneous Remote and local CO  Through June 2007: schedule 1week/month of remote CO from Pisa  Easy to extend to other locations: planning to start shifts in Japan starting July 2007  Goal: all OWL shifts covered remotely in

Peter Wilson - All Expt Meet February 26, 2007 slide - 11 Single ACE Operation l Planning started in Summer 2006 l Implementation starting mid-October  Re-arranged control room computer consoles - Added ACNET and HV control next to DAQ consoles - DAQ and Monitor functions from one work area  Automated recovery of Silicon HV crate communication failures, Ace previously left the control room (Hockerization  Auto-Hockerization)  Automated generation of Hourly ACNET plots of beam losses  Updated monitoring pages and documentation for Silicon Cooling and LV PS monitoring  Cryo Operator  Add audible alarm for Silicon detector PS current monitoring  SciCo  Improved SciCO training  Team from Silicon detector group, CDF Ops, CDF DAQ and Accelerator (Ron Moore and Brian Hedricks) led by operations heads l First shift with single Ace Dec 29  Followed 10 days where 2nd Ace acted only as backup

Peter Wilson - All Expt Meet February 26, 2007 slide - 12 CDF Owl Shift - Feb 15 SciCo - John Yoh CO in Pisa (Simone Donati) Ace (Michal Kreps)

Peter Wilson - All Expt Meet February 26, 2007 slide - 13 Owl Shift in Pisa - Feb 15

Peter Wilson - All Expt Meet February 26, 2007 slide - 14 Efficiency High with Reduced Crew Monthly average good runs w/ Si

Peter Wilson - All Expt Meet February 26, 2007 slide - 15 Experience with Reduced Shift Crew l No significant problems with shift crew changes, biggest challenges have been substitutions for shifters who are ill!  New Years weekend needed to find substitute Ace  Remote CO needed to leave after 6 hours of shift l Operational efficiency has not dropped l With single Ace, crew is operating even more as team l Experience from three weeks of remote CO: as effective as local CO l Automation is reducing downtime:  Recovery of Si HV Hockerization took 10-15min  Auto-Hockerization now takes about 4 mintues  Typically about once per store

Peter Wilson - All Expt Meet February 26, 2007 slide - 16 Additional Improvements Coming l In case of dangerous beam conditions, detector HV must be set to Stand-By:  CDF TEVMON tool monitors losses, abort gap beam, TevRF, etc based on ACNET variables, warns shift crew of dangerous conditions  If shift crew is occupied might be delayed in responding  New TEVMON based on an Open Access Client (OAC) by Ron Moore - ACNET variables indicate danger level and provide permit for operation of detectors - Automatically set Silicon HV to standby in dangerous conditions  Final testing underway l HV problems sometimes require Ace to leave control room:  Plug calorimeter HV regularly requires procedure to re-establish communications  move control PC and interface hardware from 3rd floor to control room (2nd flr)  Other HV systems need to have programs restarted or PCs to be rebooted on 1st floor  working on remote access to HV PCs from control room

Peter Wilson - All Expt Meet February 26, 2007 slide - 17 Some Key Members of the Team

Peter Wilson - All Expt Meet February 26, 2007 slide - 18 Members of the Team Thank you to the many people who helped make this happen: l Fermilab: Bill Badgett, Kurt Biery, Ray Culbertson, Steve Hahn, Brian Hendricks, Jonathan Lewis, Kaori Maeshima, Ron Moore, JJ Schmidt, Peter Wilson l CDF Technical support team: Dervin Allen, Del Allspach, Jim Loskot, Lew Morris, Bill Noe, George Wyatt l Duke: Dean Hidas l INFN Pisa: Dario Fabiani, Luciano Ristori, Fabrizio Scuri, Jose Enrique- Garcia l Tsukuba: Taichi Kubo, Koji Nakamura l Waseda: Junji Naganoma, Tetsuo Arisawa l Rochester: Jennifer Gimmel, Willis Sakumoto l Rockefeller: Mary Convery l UCLA: Charles Plager l Yale: Ulrich Husemann

Peter Wilson - All Expt Meet February 26, 2007 slide - 19 Summary l CDF has successfully reduced the shift crew from 4 physicists to 3.  25% of Owl shifts have only 2 at B0 l Process of reducing shift crew has improved the tools for shift takers l Operational efficiency has been maintained or improved during the transition  Additional improvements are coming which should result in more efficient shift operation l We are working on other improvements to increase our operational efficiency and prepare for a smaller physicist support  For example, adapting Electronic Worklists from AD for scheduling accesses and tests