1. MAGNET TESTS OPERATION in SM18 V. Chohan SM18 Operation

2. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 2 Talk Outline Introduction to SM18 Operation: –Objectives, –historical perspective included in what follows Operation: »what we do »Who »How »Services encircling the Tests Needs Brief overview of Operational Tools Concluding Remarks

3. Brief Introduction to SM18

4. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 4 SM18 Magnet Tests : What it’s all about Demands on SM18 Operation Project Management Magnet Production LHC Beam Running in & Minimum dI, dE provision Accelerator Physicists

5. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 5 SM 18 : Objectives The objectives of SM18 Magnet tests facility is to :  Check the Electrical, Cryogenic, Quench performance of the LHC Cryo- magnets.  For good LHC accelerator performance and running-in, Field quality measurements must also be carried out. 1706 number of cryogenic magnets that constitute the LHC ring must be tested at this facility. To complete the Tests activity in time for LHC start up it was decided to implement 24 x 7 operation at SM18 in the middle of 2002.

6. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 6 SM 18: Round-the-Clock operations Who would run this facility ? ( we go back to 2001) CERN budgetary crisis, hence outsourcing not an option, Due to demands of physics, it was not possible to shut down accelerators and liberate operation staff, However, 7 CERN accelerator operation staff did volunteer to assist. (from different accelerator backgrounds or control rooms PCR, MCR) Therefore, bulk of operation staff had to be obtained from elsewhere. (India, through a DAE-CERN collaboration effort from 4 different labs.) Since then up to 18 to 20 personnel from India participate in the tests on a one-year rotational basis. What’s in it for India?

7. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 7 SM 18: Work Environment in 2002 {Last Safety audit was done in that era} How to run this facility ? The work environment in 2002 had: Only 2 prototype version Test Benches (old Cluster A) Workings of tests (not standardized) : Tests were conducted with an R&D framework, magnet(dipole) delivery was erratic & in very early stages ( 8 pre-series dipoles delivered tested in period 2001 to May2002) The tests systems were limited & only partially automatic:  The tests were conducted by Equipment & Magnet experts.  Need for tampering at floor level with the equipment related to the magnets and electronic racks was required to carry out tests.  Far from Understanding of complexities involved in handing over such systems to personnel with no experience in magnet tests.

8. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 8 SM 18: Work Environment in 2002…. The work environment in 2002 had…………… Equipment was ‘laboratory’ type:  Some of the equipment was lab. type, which is unsuited for 24 hrs operation. (some these limitations still remain although to a restricted extent) To start 24 x 7 facility with this work environment in conjunction with the fact that the operation staff had absolutely NO EXPERIENCE in magnet tests was quite challenging.. Hence operational work practices, methods & techniques had to be correctly formulated & developed to start operation.

9. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 9 SM 18: Work Environment now (2005) New CFB deliveries progressively in 2003 & 2004 We only have all 12 modern test Benches working since ~ 1 July 04 ( 27June04) All Test Benches have had TIS Safety Compliance Installation/Acceptance Procedures applied prior to Usage

10. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 10 Work Environment in more detail.. Fixed 12 Test Benches arranged in pairs so, 6 clusters( A to F) 6 Main Power Converters 6 sets of Electronics for testing 1 per cluster Mobile For Q-location & MM we need to install Special 15 m Shafts Shafts require Anticryostat mounting in magnets Mobile Racks for HV insulation tests & Magnetic Measurements Utilities Water for 4 magnet powering at the same time Cryogenics capacity and limits Manpower [ 24 hr Tests Operation Staff special arrangements] 7 CERN staff (up to end 2004), upto 12-13 FTE ( in 2005 to early 2006 ) on loan from CERN Accelerator Operation 15-20 persons on exact 1 year contracts from India & constantly rotating Cryogenic Support staff ( 3 per shift) from Industrial Support contract ICS “Magnet connectors” working 24 hours since June 2004, but not ROCLA operators Manpower for Warm Tests –(almost) normal hours – 2 persons Support Generalized Equipment support( AT-MTM) as opposed to specialized support as found in 24 Hr. OPERATION in CERN & “physics” is not at stake if specialist ‘fixing’ is delayed However, AB-PO have 24 hr piquet but NOT AB-CO

11. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 11 Environment & the Principals The various “actors” in the magnet tests game….. Magnet Preparation Team

12. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 12 Work Flow: E-traveller Requirement With 12 benches in operation, the tests scenario in SM 18 is very dynamic. Description The E-traveller is a software tool interfaced to mobile phones that alert the various teams that their services are required on a particular magnet. Advantages Time to locate members of other teams is reduced. Reduced verbal communications between teams (Indian personnel have limited knowledge of French while Cryo & Mechanics teams are predominantly French speaking).

13. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 13 Work Flow Interaction : E-traveller

14. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 14 Resources Limits & Operation Fixed 12 Test Benches arranged in pairs so, 6 clusters( A to F) 6 Main Power Converters 6 sets of Electronics for testing 1 per cluster Utilities Water for 4 magnet powering at the same time Cryogenics capacity and limits : 3 to 5 Magnets @ 1.9K and under cold tests, 1 magnet in 300K to 90 K phase, 1 magnet in Last (Warm up) Quench to 300K phase, 1 magnet in 90K to 1.9 K phase with a further capacity constraint of not exceeding 9 recoveries of high field quenches or initial cooling from 4.2 K to 1.9 K per 24 h 3 Magnets may now (2005) share the Warmup or initial Cooling phases

15. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 15 Work Tools : Integrated Resources Display & PRIORITIES Requirement The operation team has to juggle with the resources to carry out the tests. Description This integrated resource display was generated by interacting with software programmes controlling the resources and running under various operating systems. Utility This enables the operation crew to optimise the resources usage by setting the priorities and preventing resource conflicts from arising. Power Cryo Water

16. OPERATIONAL PRACTICES

17. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 17 Operation Practices: To Do List The “To Do” List describes the minimum tests to be performed on a magnet. The tests were sequentially numbered Prefix such as “Prep”, “PT”, “MM” were used to denote the type of test Extra or special tests are conducted but ONLY if requested & organized in advance.

18. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 18 Operation Practices : Magnet Tests Reports The Magnet Tests Report is the paper log of the results of the tests carried out on the magnet. The sequence of tests in the MTR obeys the “To-Do” list Checklist & Operational notes appended This technique helped in conducting the tests systematically and efficiently.

19. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 19 Work Practices : White Board  The ‘White Board” summarizes –Tests done & pending on a magnet –Additional information, instructions as well as tests priorities are also written on this board.  Transfer of Information between shifts is essential for smooth operations  In a dynamic environment like in SM18, information exchange between shifts has to be and has become seamless and transparent

20. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 20 Operation Practices : Miscellaneous Documentation –Routine error-free operations are possible by means of organised methods & procedures. –A brief but complete description of each test and related equipment is documented. –These documents are written “by the operation staff for operation”. Safety –To ingrain safe operating practices, it is mandatory for the operation team to attend courses in electrical & cryogenic safety. Training –It is an important part in human resource development, Talks by Equipment & Magnet experts are arranged, Magnet Factory visits are arranged. »(In Annual Shutdowns)

21. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 21 Operational Tools: SMTMS Description The SM 18 tests management system (SMTMS) uses the CERN backbone in the web based services and features of the ACCESS data base. It provides: Automatic generation of results summaries hence, reducing the ‘bureaucratic’ activity to a minimum after the tests are conducted on a magnet Tracking the tests progress & tests statistics are important spin-offs of this tool.

22. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 22 Foreseen Times Average times: –12 hrs connection –12 hours vacuum pumping –26 hours cooldown –36 hours cold tests –12 hours warm up –12 hours disconnection Total 108 hours per Dipole (4.5 days of 24 hours)

23. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 23 Real Times

24. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 24 SM18 Tests Operation Personnel Safety & certain points 1.General Safety course for new arrivals ( Tues Aft in English) ; Level I II & III 2.Cryo Level I in English ( Goran Perinic) –Inscription on Arrival, HR –Training formalisation etc 3.Electrical Risks Awareness course ( F. Szoncso) –New Benches and hence new reduced risks definitions (GS- SC) in 2004 –‘a la carte’ meaning twice a year ? –Needs formalisation or regularisation like Cryo Level I ? Historically : Original AB-OP staff (& few Indian Associates of that era) had followed the APAVE Elec. course in 2002, 2003

25. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 25 Concluding Remarks UNIQUE ISSUES of: - Cohabitation of Acclr Ops, Contract Cryo Ops & external Collab (from India) all in the same Control Room - Uneven work load (unlike normal Acclr Ops) UNIQUE ASPECTS due to the Ext. Collaboration(India) of: -Shifts & Shifts rotation -On Job Training -local living & transport arrangements -Non-shift free time for other work -Op empowerment & Motivation -Procedures Writing & Documentation -Integration & Team Spirit -Working across 2 CERN Departments( AT & AB) & using most practical ways of going forward..( EDH, MAPS, etc)

26. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 26 SM18 Control Room

27. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 27 Connections/Disconnections: Magnetic Measurements

28. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 28 Connection / Disconnections: HV Tests

29. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 29 SM18 “Fitness” Centre Control Room Tests Floor

30. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 30 Few thoughts for the belly…. SM18 Kitchen

31. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 31 Just for the records…..

32. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 32 All Work and No play ………

33. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 33 Additional Slide

34. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 34 Results till end-May 2005 DipolesDipoles Repeated Arc-SSSArc-SSS Repeated IR-SSS Repeated Year 2003 95 Not Applicable: all pre-series 2 Not Applicable Year 2004 35645 (~ 11%) 49330 Till May 2005 17623641123 Cumulative Total 6951298 Total 832

35. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 35 Cumulative Total: All Magnets (including Repeats & Rejects)

36. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 36 Utilities : Cryogenics Capacity Cryogenics capacity (in 2004 & now ) and limits :  3 to 5 Magnets @ 1.9K and under cold tests,  1 magnet in 300K to 90 K phase(up to 2004),  1 magnet in Last (Warm up) Quench to 300K phase(2004),  1 magnet in 90K to 1.9 K phase with a further capacity constraint of not exceeding 9 recoveries of high field quenches or initial cooling from 4.2 K to 1.9 K per 24 h We need to ensure that the magnets under the tests programme follow the cryogenic phase distribution as above, within the limits of possibilities due to the varying training performance ( a priori not known) and respecting all the other environment constraints mentioned earlier 3 Magnets may now (2005) share the Warmup or initial Cooling phases

37. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 37 Additional Tools To improve operational efficiency and smoothen work flow, several other operational tools were designed and implemented, Many innovative ideas and proposals emerged from discussions held within the operation team, Some tools have been developed by the ingenuity of operational personnel: –SM 18 Test Management System (SMTMS), –E-Traveller, –Integrated Resources Display.

38. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 38 Operational Methods: Basics Two of the guiding principles in setting and running operation successfully was to provide: –Transparency : The role of the operation team MUST be well defined. It is important because it instils confidence in the operator with regards to “what has to be done” & “Rules of the game”. –Sound Knowledge base : It is essential to provide & increase the knowledge of the operation team for superior performance and thoughtful execution of tests (of the expensive LHC magnets).

39. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 39 “ The people in the field are the closest to the problem, closest to the situation, therefore that is where the real wisdom is.” “Headquarters can support innovation culturally and financially, but it is usually not headquarters that actually comes up with the great innovations and forward leaps- it is the units in the trenches.” Extract from the Book ”The Leadership secrets of Colin Powell” by Oren Harari

40. V. Chohan Magnet Tests Operation, SM18 Safety Review, 21June 2005 40 Concluding Remarks 1.There is no universal “setting up & running” methodology to guarantee operational success. Each facility has its own characteristic problems that need to be addressed & solved. 2. Methods implemented for testing the LHC Cryo magnets at SM18 have been described. Operational simplicity has been achieved as well as the synergy gained among the various actors, thanks to some of the techniques implemented. 3. It is hoped that this presentation would give this audience an insight to the complexities of the LHC project and the spirit of “operation before LHC operation”.

41. OPeration TOOLS