Final Commissioning Plan and Schedule Cheng-Ju Lin Fermilab Installation Readiness Review 09/27/2004 Outline: - Work to be done during shutdown - Work to do after shutdown - Some thoughts on operational issues

Work to be Done During Shutdown Firmware Developments: -Firmware for most data paths are in good shape. -Cluster board still has low level data corruption either due to upstream data protocol violation and/or Pulsar firmware issue. Top priority item for Vadim and Wojtek. - ShowerMax board also has low level data corruption. Problem not understood. Top priority item now for Chris. - Implement ShowerMax zero-suppression firmware to reduce latency Firmware at simulation stage Critical items:

Active splitter for ShowerMax fibers  need all 48 fibers split PC/Control Node : -Integrate into the current trigger table building framework -Interface with ScalerMon (pre-scale changes) -TL2D building (in processing node) -Infrastructure code for L2 algorithm -Need to start purchasing 2 more PC soon -All above items are software work, should be able to complete during shutdown Misc. Software: - PulsarMon (online monitoring for Consumer Operator) - FER code for TL2D and ScalerMon support Documentations !! To the committee: anything else we’ve missed? Work to be Done During Shutdown Critical items:

Shutdown Work Schedule Integrating into current trigger table building framework (Daniel, Kristian) Finish on Nov 1, tested and debugged by Dec 1 Interface with ScalerMon for prescale changes (Daniel): Finish on Sept 15 TL2D Building in node (Kristian): Complete by Oct 15 L2 algorithm infrastructure code: Complete by Nov 15 PC purchase (Kristian): Start paperwork Oct 15, expected arrival date Dec 1. Debug Cluster firmware (Vadim): fix the problem during the shutdown and spend the the first two weeks of beamtime (parasitic) validating the fix. Debug ShowerMax data corruption (Chris): middle of October Active splitter working for 48 SMXR fibers (Chris): complete by Nov 15 FER code changes for TL2D and ScalerMon (Cheng-Ju, with help from DAQ experts): complete by Nov 15. Testing zero-suppression ShowerMax firmware and merger optimization (Sakari): before shutdown PulsarMon code (Vadim, Wojtek and other subsystem experts): Full blown version running by the middle of November. Clean up cabling in the Pulsar crate (Cheng-Ju with help from JDL): Before shutdown. Documentation (all): continuous updates. Names in parentheses are folks in charge of the work Completion dates are estimated by folks working hard at the task.

Summary of Shutdown Work Schedule We should be able to to meet the timeline for most of the items mentioned before. Firmware related issues  Making Cluster and ShowerMax firmware robust. Since some of the problems are still not fully understood, solving the problem may take longer than expected.  People are working hard at it. Hardware related: ShowerMax active splitter fully working. Crucial for final commissioning

Work to be Done After Shutdown We would like to come out of the shutdown with a full system Some pieces may need a week or two of stand-alone testing first. Initial work  parasitic commissioning of the L2 decision trigger: - Check for data corruption for the various data paths - Compare L2 trigger decision (based on upstream simulation and alpha) - Verify ScalerMon, TL2D, PularMon infrastructure in beam running In the past we have used very little dedicated beam time for Pulsar work. We will need dedicated beamtime (~few hours/week) after the shutdown to fully validate the Pulsar system: - Look for system wide problems with Pulsar driving L2, - Study error handling recovery implementation, - Study timing performance of the Pulsar driven system + optimization, - etc… We will do our best to minimize dedicated beamtime.

Validation criteria: - Two weeks of stable running with Pulsar driving L2 and no trigger decision discrepancy based on upstream simulation, - No significant downtime caused by Pulsar system during the two weeks of trial, - Pulsar system performance (deadtime) meet and exceed alpha system at any L1A rate and luminosity. To the IRR committee: are the above conditions sufficient to meet the technical requirements? Our goal is to achieve the above criteria + any other technical requirements recommended by the IRR committee by March After we have met the technical requirements, we’ll have another review before officially switching over. At this review, we’ll have to settle the MOUs, manpower, and other issues. Work to be Done After Shutdown

Thinking Ahead on Operational Issues Overhead for switching between the Alpha and Pulsar driven system: Switching from Alpha to Pulsar driven trigger should be seamless. All the reconfiguration can be done within minutes. Hardware maintenance support: - Pulsar boards, custom mezzanine cards and transition boards  PREP - Control and processing PCs  UPENN - Input data splitting infrastructure  Chicago, Fermilab, UPENN. Pager carrier rotation: All Pulsar group members have expressed strong commitment to carry out pager responsibility. Similar to the current L2 page, we will maintain Pulsar webpages for CO, shiftcrew, and pager carriers. Some questions from IRR committee members:

Hot spares: at any given time, we plan to maintain a copy of the L2 decision system in the trigger room. However, the system may be used for developmental work. We will also set aside hot spares. Pulsar group will maintain the PulsarMon package. However, we do not intend to assume responsibility for TrigMon, XMON, nor the trigger database. All the above issues (and more) will have to be fully resolved by the next review. Thinking Ahead on Operational Issues Some more questions from IRR committee members:

Summary As Ted mentioned in the beginning, the purpose of this review is to help us get ready for the final commissioning. We believe that we have a well defined plan, but no plan is perfect. We welcome any recommendations (or even criticisms) of our plan. Lastly but not least, the Pulsar group would like to thank the Committee for taking your time to review the Pulsar upgrade project.