6/19/2014 PHENIX WEEKLY PLANNING June 19, 2014 Don Lynch 1
6/19/2014 This Week Run 14 Continues Maintenance Access 6/16: VTX monitoring work MPC-Ex board replacement VTX/FVTX Shutdown work review meeting (Monday) Plan for 2014 Shutdown Tech Support for Run 14 as required Support for sPHENIX efforts as required 2
6/19/2014 Next Week Run 14 Continues No scheduled access for remainder of Run 14? VTX/FVTX Shutdown work review meeting part 2 (Monday) Plan for 2014 Shutdown Tech Support for Run 14 as required Support for sPHENIX efforts as required 3
6/19/ planned Technical Support & 2014 Shutdown Support for run 142/3-6/30/2014 Procure & Fabricate parts for MPC-Ex North and South1/1/2014-6/30/2014 Set up Physics lab for FVTX/VTX east6/15/2014 End of Run Party 6/27/2014 MuID Efficiency Measurement (Itaru, requires cooling water & isobutane)7/7-7/10/2014 Start of Shutdown Tasks (purge flammable gas, disassemble and stow shield wall, remove collars, move EC to AH, Move MMS south, etc.)7/14 – 7/25/2014 Remove FVTX/VTX East to PHYSICS?, repair and reinstall7/14 – 9/15/2014 Remove MMS east vertical lampshade 7/28-7/30/2014 Troubleshoot intermittent water leak in MMS7/30- 8/8/2014 Other Maint. In MMS TBD Reinstall MMS east vertical lampshadeTBD Summer Sunday prep AH, tours and restore AH7/30-8/15/2014 Install scaffolding in Sta 1 South7/28/2014 Remove MPC-Ex prototype7/28-8/1/2014 MuTr Sta 1 South troubleshooting and repairs7/28-8/1/2014 Maint. & Repairs for MPC South, BBC South, RPC1 South17/28-8/1/2014 Assemble & test MPC-Ex North, ready for installation8/1-9/5/2014 Remove scaffolding from sta 1 south, Move CM South8/4-8/5/2014 Install scaffolding in Sta 1 North8/6-8/8/2014 MuTr Sta 1 & Sta. North troubleshooting and repairs8/11-9/5/2014 Prep MPC-Ex North installation area8/11-9/5/2014 Install new MPC-Ex North9/8-9/26/2014 Assemble & test MPC-Ex South, ready for installation9/2-10/3/2014 4
6/19/2014 Remove Sta 1 N scaffolds, Move CM North, Install scaffolding in Sta 1 S9/29-10/3/2014 Install MPC-Ex South10/6-10/24/2014 Other detector supportTBD Infrastructure Maintenance and ImprovementTBD Decommissioning of obsolete PHENIX detector equipmentTBD sPHENIX Supporton-going End of Shutdown Tasks (Move MS north, roll in EC, install collars, remove 10 ton cart, plates and manlifts, build shield wall, etc.)10/27-11/26/2014 Pink/White/Blue Sheets12/1-12/19/2014 End of Shutdown Party???? Start Flammable gas flow???? Close shield wall, install radiation interlocks and prepare for run 1412/31/2014 Start run 151/2/ planned Technical Support & 2014 Shutdown (cont’d) 5
Muon Tracker Shutdown Work List – summer 2014 testing as MPC-EX installed, particularly before closing Sta-1’s fix North Arcnet – N.2.7.1, North Sta-2 Oct-7 Chassis-1 (bad cable?) fix packets that were disabled for Run ,36 – South Sta-1 Quad-4 Chassis ,68 – North Sta-2 Oct-7 Chassis-2 replace boards for most frequent FEM problems from run North Sta-1 Quad-3 Chassis-3? might have already done this; check history (changed RX 3/14/12) – South Sta-2 Oct-3 Chassis-3 - unreachable N341 HV trip problem? auto-reboots of ArcNet and iocondev’s for calibration? Access needed: South & North Sta-1 Inside North Sta-2 on bottom Main Issue – Manpower 6/19/2014 6
VTX/FVTX east repairs/upgrades required West to remain installed ?? Need to get PHYSICS FVTX/VTX lab ready by ~ mid June Meeting Monday 6/16 to discuss shutdown task details: follow up meeting next Monday 6/23 1:00 PM
6/19/ Action Items: 1. FVTX group prioritize list of IR work based on work that needs to be done with detector “cold” and what can be done when detector is “warm”. 2. M. Wysocki: Discuss needs for IR electronics testing with Miljko and report back to VTX group. 3. R. Pisani: Estimate cost and time to add a 4th chiller and split FVTX disks from VTX StriPixel Ladders 4. E. Mannel: Estimate heat load on cold loop by FVTX disks and VTXS ladders 5. E. Mannel/R. Pisani/S. Boose: Investigate adding additional temperature/humidity sensors inside the (F)VTX enclosure. 6. Mike L./ Rob P.: Look at ways of creating an “dry” zone around the (F)VTX enclosure using a portable AC unit. End of run is scheduled for July 7, It is expected that the shield wall will remain up for at least a couple of days (until July 10). Once the wall comes down, the IR will be subject to higher levels of humidity and larger temperature variations. Work can continue on the detector after the wall comes down, but most likely will have to be at higher temperatures to avoid condensation issues. Current plan is to remove the East Detector only. De-cableing of the east half will begin sometime after July 14, but can be delayed if needed for additional IR studies by the (F)VTX group. VTX StriPixels may request west half to be removed depending on understanding of bias issue in the east half and IR studies immediately after the end of Run.
6/19/ Post-Run Studies in the IR: VTX Pixels: None VTX StriPixels: Bias studies. Requires bias current measurements varying the N2 flow and warming the detector up. Warm up studies will be last after all “cold” work by (F)VTX groups is complete in order to avoid thermal cycling of the detector. Expect the work to take or order 1 day (half day cold/half day warm). Unclear if any IR work is requested by ORNL group for readout work. M. Wysocki will discuss it with ORNL group. FVTX: FVTX group has a number of problems observed only in IR (see FVTX WIKI page). Group needs to prioritize the list based on what needs to be done “cold” and what can be done once the detector is “warm” and estimate the amount of time for the work. Physics Lab Work: General: Mike L. has requested a couple of days once the detector is in the lab to look at the detector in order to see what can be done to improve the flow of N2 and better seal against condensation. VTX Pixel: Investigate one bad ladder on east detector (B1-L11). Will concentrate on one end first. Will require access to the SPIRO Board. This requires removal of FVTX 1⁄2 cage and Bigwheel, and a couple of Pixel Bigwheels. Depending on what the problem is may require access to SPIRO board on opposite end. VTX StriPixels: Investigate one bad ladder on east detector (B3-L16). Requires access to LDTB. This requires removal of south FVTX 1⁄2 cage and south Pixel Bigwheels. Depending on outcome of IR bias studies FVTX: Number of ROC issues. Can be done with 1⁄2 cages installed or removed IR Infrastructure after East detector is removed Chillers: general repair and maintenance of the 3 chillers. Requires tech time and support for BNL HVAC group. Cooling lines: Replace Teflon tubing in the vicinity of the detector with stainless steel tubing. Look at any rerouting of stainless steel lines that were added during the run. Interlocks: Review flow interlock hardware and update/modify as needed. Provide additional interlocks and monitoring for 4th chiller if added (see next item)o The VTX StriPixel group is requesting that the FVTX and StriPixel ladders be put on separate chillers. This will require purchasing an additional chiller, running additional cooling lines to the IR and installing additional manifolds at the detector. Need to estimate cost of materials and labor required to add 3rd chiller loop. N2: Look at ways of better sealing up the enclosure to reduce the leakage of N2 and migration of humidity into the enclosure. Concern that sealing the detector too tight will impact the cooling of electronics. Right now there are significant gaps at the BigWheels in part due to the age of the foam seal. Need to also re-evaluate the N2 distribution inside the detector. There was a suggestion to look at ways of creating a low humidity air volume around the enclosure. Idea would be to put some kind of “tent” around the (F)VTX and pump dry air from an portable AC unit into the enclosed volume. Region would not have to be air tight, but provide a region of drier air around the enclosure. Mike L and Rob P. will investigate options.
6/19/2014 MPC-Ex N & S Final Installations This summer Working on North BP support design Tungsten plates received, QC acceptable Additional parts ordered from CS due end of June Rescheduling meeting yesterday to update shutdown schedule 10
6/19/2014 (1) Stony Brook in charge of MPC-EX testing (2) Tom Hemmick is leading a test beam effort at SLAC June th : one MPC-EX hemisphere with 8 tungsten layers and 4X and 4Y carrier boards one micromodule on each layer, partial assembly to be returned to BNL by July 7. (3) Current Parts Dispositions: Sent to Stony Brook for tests: 8 (4 x and 4 y) carrier boards laminated to 'W' plates (new plates). 3 loose carrier boards for testing (one missing a connector) 50 brass spacer nuts 4 SS 1/ x 4" studs 4 rapid prototyped spacers 1 micromodule Mike Lenz office: 1 Assembly fixture 3 Delrin covers (in the shape of the 'W' plates) 6 'W' plates (new plates) 50 brass spacer nuts At Central shops (due 6/27/14): 5 mounting tabs 50 brass spacer nuts 20 FEM mounting brackets 20 FEM mount Isolation Boards Jim LaBounty’s office: Installation assembly parts (to be itemized) (Additional parts in currently installed partial South prototype to be itemized) MPC-Ex Project Summary 11
6/19/2014 Current Parts Dispositions (cont’d): Sensors Sensor testing at Yonsei is proceeding well, and we should have the remaining 200+ sensors from Yonsei by the end of June. We already have almost all the sensors we need for the south MPC-EX. Micromodules: 450 ROC’s ordered: 100 received and used in building initial micromodules 100 in transit to BNL 250 rejected by Sierra (bad solder mask) and being remanufactured expect shipment in ~ 1 week The electronics for the FEM is designed and at Sierra awaiting a BNL PO for manufacture and assembly. The FEMs have four readout inputs, so there will be eight FEMs per arm. two FEMs per box, four locations on the magnet for each arm. (4) Assembly Plan After receipt at BNL, ROC’s are inspected and sent to Quik Pak with SVX4 chips for wirebonding the chips to the ROC ROCs returned to BNL and inspected BNL wirebonds the sensors and assembles the micromodules Micromodules sent to Stony Brook for testing Final assembly into MPC-Ex N & S at Stony Brook, PHENIX tech to participate and assure QC Return assemblies to PHENIX for installation in accordance with current shutdown schedule MPC-Ex Schedule considerations, Cont’d 12
Getting Ready For DOE Science Review 7/1-7/2 Outer HCAL Cryostat Inner HCAL EMCAL VTX 6/19/2014 6/12/
Basic sPHENIX model cutaway (updated) 6/19/2014 Design efforts over the next few months will include several areas of emphasis: (a) incorporating a more detailed flux return (cap) design 6/12/
sPHENIX open, with racks 6/19/2014 (b) A more detailed structural support design, with support structure envelope details to be included in detector envelope 6/12/
(c) Initial concept for external support, racks and upper bridge platform to be more detailed and conform to envelope restrictions 6/19/2014 6/12/
6/19/2014 (d) “Innie” / “Bothie” tile configuration to be updated 17
Magnet (BABAR) (Magnet Div) Acquire rightsDone Evaluate TransportDone Design Transport fixtures Done Fabricate Transport Fixtures 6/15/14 Evaluate BNL test facility requirements 8/1/14 Prepare test facility9/1/14 Install Transport fixtures9/1/14 Transport to BNL 9/15/14 Install in test facility10/1/14 Design tests/test equipment10/1/14 Fabricate/procure test equipment1/1/15 Run tests6/1/15 Design Magnet modifications12/1/14 Fabricate magnet modifications6/1/15 Install magnet modifications8/1/15 Evaluate final installation9/1/15 Design installation support equipment12/31/2015 Fabricate/procure support equipment7/1/2018 Final test before transport to 10085/1/2019 Transport to 1008 IR6/10/2019 Install8/30/2019 Test 10/25/2019 Commission11/22/2019 6/19/2014 6/12/
6/19/2014 Babar Magnet Schedule: Mike Anarella 6/12/
6/19/2014 ePHENIX 2ft high x 1 ft wide clearance needed for e-ring components 20
6/19/ From Ray Karol: Lessons Learned: Door switches fail due to improper installation Discussion: During a Search and Secure procedure for the BNL, NSLS-II storage ring, two adjacent door switches were found in the closed position when they should have been open. By the switches remaining in the closed position, the monitoring system "thinks" the door is closed when in fact, it could be open and access could be gained to the accelerator enclosure. Note that the locks and a second set of switches remained fully active during this period and the safety of staff was maintained at all times. Investigation of the event revealed that the installation of the switches was inadequate resulting in improper strike angles and/or interferences, which ultimately caused damage to the switch bodies and the premature failure of the switches. To ensure uninterrupted protection, the door switches are installed in two independent logic chains and are also redundant within that chain (i.e., two switches per chain). Due to the specific design of the switches and the method of installation, both switches failed in one of the two chains. Protection of staff was maintained throughout the period through the second chain. Findings: -Inconsistent installation. -Varying strike angles. -Interference between the switch body and the face of the door.
6/19/ Analysis: A "5 Whys" analysis was performed to assess the cause of the failure: 1) Two PPS Switches failed simultaneously by getting jammed in the closed (made) orientation (why?) 2) Either (a) the door struck the leading edge of the limit switch-body causing the internal components to be damaged or (b) the door struck the switch lever-arm while the arm was at a near-zero strike angle with respect to the door, which caused the internal components to be damaged (why?) 3) Prior to installation, a thorough review of the mechanical elements of the PPS Door/Gate Interlock System was not conducted by an experienced mechanical engineer (why?) 4) The PPS interlock system is mostly electrical in nature. The mechanical interaction between the door/gate, limit switches and "Mag Lock" appear to be straight forward but in actuality, there are many things to consider that would normally be reviewed by a mechanical engineer. The participation of a mechanical engineer was not considered in the scope of resources required to design and implement the PPS Interlock system (why?). 5) The PPS design and installation practices have not included a review by a cross-cutting team for this system. In summary, the mechanical assessment that took place on January 20, 2014, identified two probable causes for the switch failure: inadequate clearance and improper strike angle. Recommended Actions: 1) Review the Design Review process to ensure all systems are reviewed by an appropriate team of engineers and/or technical staff. All installations should be reviewed by a mechanical engineer or other qualified person prior to placing them into operation to ensure installation meets manufacturer requirements. 2) Periodically inspect all switch installations to ensure proper clearance and strike angles are maintained.
6/19/2014 PHENIX Safety and Security 23
6/19/
6/19/2014 Where To Find PHENIX Engineering Info Run 14 Continues! Just over 2 weeks to go ! 25
6/19/