1. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov RPC Detector Review Hardware Status and New Technical Procedures

2. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov CMS RPC EQUIPMENT  Detector and Service EQUIPMNET  Known Problems  Procedures

3. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov DETECTOR AND SERVICE EQUIPMENT BARRELENDCAPTRIGGERSPARE UXC Spare USC Spare UXCUSC Spare UXCUSC Trigger EndcapBarrel  EQUIPMENT ALREADY UPLOADED in EMDb  EQUIPMENT TO BE UPLOADED in EMDb – till end of June 2010

4. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov KNOWN PROBLEMS Chambers in SG Slow ControlMasked Rolls Threshold Control TPH Sensors PROBLEMS by TYPE PROBLEMS by SEVERITY 1) Problems for long shutdown/CMS opening 2) Problems for NEXT UXC ACCESS 3) Reappearing Problems

5. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Technical Procedures  How to deal with a new equipment problems?  How to create a work package when access is announced?  How to replace equipment in UXC?  How to replace equipment in USC?

6. How to deal with an equipment problem Operation Manager P5 SHIFTER RPC Equipment Manager HARDWARE EXPERT elog Savannah Automatic mail Savannah

7. ACCESS CMS RC&TC, GLIMOS SLIMOS, RFM, CSL Operation Managers Subdetector RFM (Operation Managers) WP EMDB System Experts EMDB Manager RP Technical shifter BUFFER ZONE EMDB EQUIPMENT UXC Announce Access RP Sweep Access Open Authorize Request/Modify Open/Close Control Give Access/Key Change Data (Full System Rights) Supervise (Full Rights) OperationalSpare Operational Defective Check Equipment Repair Still Defective Tested Waste Broken Take after RP check Use WP name to check in equipment

8. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Work Package List # WP name DateDuration Task Description probllocation 1 RB LV board inspection 14.06.101.1h W+1/S7 board replaced 1X2A11 2 RB HV test 31.05.101.9d RPC-DT noise test 1 X1 PP 3 RE HV checkout 31.05.102d Investigate RE+1/2/21_22 1 X1 PP 4 RE LV connect. replacement 31.05.102.1h RE-3/2,3/25 Anderson changed 1X3S51 5 RB LV connector check 31.05.102.2dW-1/S5/RB3-1X4S11 6 RB LV module check 20.05.101.1hW-1/S5/RB3-1X4S11 7 RE LV module replacement 20.05.101.1h Due to channel RE-3/2,3/25 1X3S51 8 RPC LV connector check 19.05.1039min RE-3/2,3/25 & W-1/5/RB3- 2 X4S11, X3S51 9 RB HV check out 04.05.102.1h W+2/8/RB2in & W+1/7/RB3+ 2X1PP 10 RPC HV system inspection 26.04.102.1d 4 spare RB cables for SVmax 3 RE probl (2 3polar changed) 7 X1 PP 11 RB HV check out 08.04.105.2h Trip channel at W0 PP 1X2N39 12 RB HV chain check out 16.03.101.9d 7 HV prob(W+2,W+1,W-2,RE-1) 7 X1 PP

9. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Work Packages by system System # WP Duration # Problems InsideMTAOutsideMTA RB HV 5 6.2 days 13103 RB LV 4 2.3 days 211 RE HV 3 6 days 660 RE LV 3 2.2 days 110 RG Trigger 0 (2) shared220 Total15 RB 8.5d RE 8.2d 24204

10. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov BARREL HV INTERVENTIONS Wheel No. Problems No. Components changed RB+276 RB+183 RB012 RB-100 RB-211 TOTAL1712

11. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Barrel HV System: Summary by Components II Trimester Wheel HV Board A3512N Short red cable Brd -> Y Y Long black cable US-UX Beyond PP SVMax < 9600 i1 settings > 10uA RareProblem RB+2-132---- RB+11-1141- RB0--11-1-- RB-1-------- RB-21------- Total21441510

12. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Barrel HV System Status II trimester  Broken Y in Distributor Boxes: 6/480; ~ 1.2% (0.6%*)  Broken long black cables (USC – UXC): 11 (7*)  Spare black cables used 17/480; ~ 3.5% (2.7%*)  473** RPC in DG mode 473/480; ~ 98.54% (98.75%*)  5 RPC in SG mode 10/1020; ~ 1% (1%*)  1 RPC totally dead 1/480; ~ 0.21% (0.21%*) N.B. * => Barrel HV system status for I trimester 2010 ** => 1 RPC OFF due to LV problem (next slide)

13. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Barrel Low Voltage/Slow Control Status II Trimester 1 RPC LV OFF 479/480 ~ 0.21% (0%*) 18 RPC I2C lines (in 17 chambers) not working in the Barrel 18/800 ~ 2.25% (2.13%*) Redundant DT lines working for 8/18 ~ 44% (9/17~53%*) 10/800 I2C lines in the Barrel not controlled at all ~ 1.25% (8/800 ~ 1%*) Vth or Vmon values not settable on chip basis in 4 cases in the Barrel (no change*)

14. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Barrel Threshold Control History Date Problematic FEB/chip No THR Control DT line Working No redundant Control 22.01.200861275 21.05.200981385 23.11.200961486 16.03.201041798 15.06.2010418810

15. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov RE+2/2, 3/11- Tripping: Problem localized in HV distributor so fixed it by using the spare channel. (On April08) RE+2/2, 3/11- Tripping: Problem localized in HV distributor so fixed it by using the spare channel. (On April08) RE-1/3/31, 32- Spikes: Problem localized in RE-1/3/31 (TOP) so disconnect this gap and run rest of three gaps. (On April08) RE-1/3/31, 32- Spikes: Problem localized in RE-1/3/31 (TOP) so disconnect this gap and run rest of three gaps. (On April08) RE-1/3/05, 06-Tripping: Localized ~ at PP so fixed it by changing the tripolar connector in UXC. (Reported on April10) RE-1/3/05, 06-Tripping: Localized ~ at PP so fixed it by changing the tripolar connector in UXC. (Reported on April10) RE-3/2, 3/26-Tripping - Fixed by changing connector in UXC at PP. (On April 28) RE-3/2, 3/26-Tripping - Fixed by changing connector in UXC at PP. (On April 28) RE-1/3/35, 36-Tripping – Actually problem was in RE-1/2/01,02 because channels were swapped. Put back the channels at original position. Problem do not reproduce. (On April 28 RE-1/3/35, 36-Tripping – Actually problem was in RE-1/2/01,02 because channels were swapped. Put back the channels at original position. Problem do not reproduce. (On April 28 RE+1/2/21, 22- Tripping – During TA31052010 switch on back this channel so it start working without any intervention and it is working up to now.(Reported on May06) RE+1/2/21, 22- Tripping – During TA31052010 switch on back this channel so it start working without any intervention and it is working up to now.(Reported on May06) RE-2/2,3/11-Tripping- During TA31052010 swapped the channel with RE- 2/2,3/12 both are working without any problem. (Reported on June 02). RE-2/2,3/11-Tripping- During TA31052010 swapped the channel with RE- 2/2,3/12 both are working without any problem. (Reported on June 02). During last 3 months total 7 (6 trpping+1spikes) HV problems were reported Saleh Muhammad NCP Islamabad Endcap HV Interventions and Status

16. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Endcap LV Interventions and Status RE+2/2, 3/31- Tripped (LV FEB) Fixed the problem by changing the problematic module with good spare one. ( On March 11). RE+2/2, 3/31- Tripped (LV FEB) Fixed the problem by changing the problematic module with good spare one. ( On March 11). RE-3/2, 3/25 –Tripped (LV FEB) Localized the problem in the module channels. Fixed the problem by changing the module also by repairing the connector which break during changing the module. (Reported on May19) RE-3/2, 3/25 –Tripped (LV FEB) Localized the problem in the module channels. Fixed the problem by changing the module also by repairing the connector which break during changing the module. (Reported on May19) LV modules serial no.47 which was replaced form P5 test it in bld.904 testing setup. The board did not give the problem again so put it back into system at P5 on May31. LV modules serial no.47 which was replaced form P5 test it in bld.904 testing setup. The board did not give the problem again so put it back into system at P5 on May31. On 1 st of April the X2S32 rack (YE-1 far side, X2 level) all electronics become off due to rising the temperature of MAO at 29C and LV boards at 40C. This happen due to power failure of the turbine. But after 4 hours power came back and just switching on the MAO and boards everything start working. On 1 st of April the X2S32 rack (YE-1 far side, X2 level) all electronics become off due to rising the temperature of MAO at 29C and LV boards at 40C. This happen due to power failure of the turbine. But after 4 hours power came back and just switching on the MAO and boards everything start working. Saleh Muhammad NCP Islamabad

17. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Endcap Summary (RPC TCM) Systems HV HVLV Slow control Slow controlGas RPC TCM New pending Problem Spare cables in use DB spare ch. in use Single gap RPCs Full RPC out New pending problem Full RPCs out TH setting Problem pending T and RH sensor problem pending Known big leaks channels Known big leaks channels April26, 2010 30118101702 May 03, 2010 01117101702 May 10, 2010 10117101702 May 17, 2010 10117101702 May 23,2010 10117111702 June 02,2010 00117101702 June 07, 2010 00117101702 June 14, 2010 00117101702 Saleh Muhammad NCP Islamabad

18. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Trigger USC electronics CAEN VME controller failed in one Trigger Crate on night 18/19 April 2010. Replaced by a spare on 19 April. It was checked that only mezzanine board with optics was damaged (most probably optical transmitter). CAEN VME controller failed in one Trigger Crate on night 18/19 April 2010. Replaced by a spare on 19 April. It was checked that only mezzanine board with optics was damaged (most probably optical transmitter). At the moment no other problems - 100% of the system functional!!! At the moment no other problems - 100% of the system functional!!! Karol Buńkowski, CERN, UW

19. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Trigger Link System ~10 March: one Half Box tripped during normal operation (not during power on), first we have replaced the LV module, but it did not helped, so then we have replaced all boards from that Half Box (CB, 2x MLB,4x SLB), and this helped. Test showed that: ~10 March: one Half Box tripped during normal operation (not during power on), first we have replaced the LV module, but it did not helped, so then we have replaced all boards from that Half Box (CB, 2x MLB,4x SLB), and this helped. Test showed that: CB Actel damaged, the CB itself is OK, CB Actel damaged, the CB itself is OK, GOLs on both MLB damaged, GOLs on both MLB damaged, SLBs are OK, SLBs are OK, RB+1 far (24 April 2010) - problems with the CCU ring: RB+1 far (24 April 2010) - problems with the CCU ring: FEC mezzanine was not well attached to the CCU board. FEC mezzanine was not well attached to the CCU board. When it was corrected CCU errors appeared again. So the FEC mezzanine was swapped with other one. After that both FECs were working correctly. Possible reason: FEC and the ribbon fiber connectors “not compatible”??? When it was corrected CCU errors appeared again. So the FEC mezzanine was swapped with other one. After that both FECs were working correctly. Possible reason: FEC and the ribbon fiber connectors “not compatible”??? RB+2_near (31 May 2010) - the CCU errors become very frequent, ~every second (before that ring had a few CCU errors per day), RB+2_near (31 May 2010) - the CCU errors become very frequent, ~every second (before that ring had a few CCU errors per day), The ring B for that tower was disconnected, after that no CCU errors. The ring B for that tower was disconnected, after that no CCU errors. CCU errors appear only on one-two rings, with low rate. In most of the cases are recovered by the reset_fec automatically executed by the online software. CCU errors appear only on one-two rings, with low rate. In most of the cases are recovered by the reset_fec automatically executed by the online software. The system is working stable - 100% of the system functional The system is working stable - 100% of the system functional Karol Buńkowski, CERN, UW

20. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Trigger Firmware 22 April - CB firmware updated - problem with the LB firmware loading from FLASH corrected 22 April - CB firmware updated - problem with the LB firmware loading from FLASH corrected Karol Buńkowski, CERN, UW 14 June 2010

21. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Trigger Online Software Status The Trigger Supervisor control over the full system (Link System, Trigger System, RBC/TTU) is working fine. Latest improvements: Latest improvements: Monitoring panels in TS (the data from the XDAQs to the TS are send with use of the flash lists) Monitoring panels in TS (the data from the XDAQs to the TS are send with use of the flash lists) Panel in the TS LBox Worker: Panel in the TS LBox Worker: Disabling the LBox XDAQs from the TS control Disabling the LBox XDAQs from the TS control Loading the LB firmware Loading the LB firmware Presenting the LBox XDAQs status Presenting the LBox XDAQs status Panel in the TS Cell: Panel in the TS Cell: disabling the TC XDAQs from the TS control disabling the TC XDAQs from the TS control Presenting the TC XDAQs status Presenting the TC XDAQs status Alarms in the HOTSOPT Alarms in the HOTSOPT GUI for disabling the Sorter Crate inputs (in case of the Trigger Crate malfunction) GUI for disabling the Sorter Crate inputs (in case of the Trigger Crate malfunction) To do next: To do next: Better management of the configuration data (different local config keys for different parts of the system), Better management of the configuration data (different local config keys for different parts of the system), FEB configuration from the DB FEB configuration from the DB Storing monitoring data in the condition database Storing monitoring data in the condition database Karol Buńkowski, CERN, UW

22. Ian Crotty 14 June 2010 RPC Cooling Two main changes have occurred this year Regulation Valve change on the “Endcap cooling circuit Week 17 2010 The changes of set Point and regulation Parameters since Week 20 2010 Continuing Concerns Low contingency Temps still high in Endcaps Temp gradients along Z in RE

23. Ian Crotty 14 June 2010 RB Temperature Log since April 2010

24. Ian Crotty 14 June 2010 RE Temperature Log since April 2010

25. Ian Crotty 14 June 2010 Endcap Circuit Set-point Changed to 16.5°C

26. Ian Crotty 14 June 2010 X2 Power Cut & New Set-point Change

27. Ian Crotty 14 June 2010 RPC Temperature Distributions +Z-Z RB0 20.33C19.90C19.79C20.25C 18.67C19.41C20.83C21.33C20.32C 21.92C 0.5C0.8C0.6C0.7C0.8C 0.75C Gains since April 2010

28. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Gas Status New gas leak in WP2_S05_RB2 notified to CERN gas group. New gas leak in WP2_S05_RB2 notified to CERN gas group. W0 readout board replaced (middle of March). No effect on RPC operation W0 readout board replaced (middle of March). No effect on RPC operation Warning due to purifiers fixed. Warning due to purifiers fixed. New Flow cell threshold settings for warnings and alarms uploaded into DB. This did not solve the sporadic warnings due to spikes on cells readout. For this is needed a new approach in the warning treatment to be implemented into RPC-DCS. Discussion on this is ongoing with G.Polese. New Flow cell threshold settings for warnings and alarms uploaded into DB. This did not solve the sporadic warnings due to spikes on cells readout. For this is needed a new approach in the warning treatment to be implemented into RPC-DCS. Discussion on this is ongoing with G.Polese. L. Benussi

29. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Gas Flow cell Reading fluctuations L. Benussi

30. Geneva 15.06.2010 CMS Week, RPC Operation Review Anton Dimitrov Conclusions  Partitions that cannot be threshold controlled are increasing in time, leading to the dilemma “noise vs. masking”!  Apart from that Detector Performance is stable!  For better Detector performance threshold refinement is required => DB is highly needed!  Temperatures in Endcap still higher than 20°C, but decreasing after the change of coolant set-point.  Signals of after pulsing raised again the question of individual chamber HV working point definition.  Still lot of effort to be put in order to update all the instructions!