1. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 1 CO deliverables for HWC – lessons learnt and the proposed way ahead Markus Zerlauth, AB-CO-MI for the CO group

2. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 2 Outline  Post Mortem analysis and test automation  Circuit Synoptic  Logging  CCC environment for HWC  E-logbook  Conclusion

3. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 3 Post Mortem Analysis

4. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 4 3. PNO.2 auto analysis D2 Q5 Q4 1 s Current PMA packages Courtesy of PM team

5. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 5 Current Architecture of PMA system v5 Individual system analysis Logical system analysis Magnet system SDDS logging additional results Raw data Result data 1 Analysis sequencer Result data 2 General system analysis Cryo circuits PC PIC QPS Cryo Vac BLM Circuit 600 A PIC 1, 2 Circuit Electrical circuits Equip DFB, SCLink Stand. cell GSA Analysis sequencer ready 29-06-2007 Circuit 60, 80, 120A Courtesy of A.Rijllart

6. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 6 Post Mortem Analysis – further automation  No fully automatic analysis feasible yet, but further automated hand-shake between sequencer and PMA possible  Complete Logical system analysis + with further dedicated analysis packages of type PNO.2  Priority must be on circuit types <6kA (largest quantity)  PO, PMA and sequencer teams are investigating possibilities to further automate PO related tests for 60A and 600A circuits (PCC, crow-bar, etc..)  Issues: –In-homogeneous test plan & interest of involved parties requires global coordination of priorities for Final sequences (need to agree on last changes & freeze as much as possible) Related automated analysis –Development priorities need to be urgently agreed upon in between all involved parties in the next weeks to provide operational & tested packages already for sector 45 –Automated test plan versus manual (MTF reporting, parameter update,..)  Series of SACEC meetings to address these issues

7. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 7 Sequences versus ‘automated’ PMA 60-120A x 1036 PIC1.x PCC PIC2.x PNO.1 PNO.2 PNO.3 PNO.4 PAC 600A X 410 PIC1.x PCC PIC2.x PCS.1-4 PLI1.x PLI2.x PLI3.x PNO.1 PNO.6 PNO.10 IPD X 16 PIC1.x PCC PIC2.x PLI1.x PLI2.x PLI3.x PNO.5 PNO.6 PNO.10 PAC IPQ X 78 PIC1.x PCC PIC2.x PLI1.x PLI2.x PLI3.x PNO.5 PNO.6 PNO.7 PNO.10 PSQ PAC 13kA X 32 PIC1.x PCC PIC2.x PCS.1-4 PLI1.x PLI2.x PLI3.x PNO.1 PNO.6 PNO.10 PAC Sequences ready PM analysis automated

8. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 8 Post Mortem Analysis – ongoing improvements  PM Data Browser –Automated loading of correlated PM data from QPS/PIC/EE, etc. based on the FGC event –Event search based on circuit name and/or time window –possibility to manage the list of the signals to display in the HWC view (via a text editor) –user can save the jpeg of the CHART or the excel sheet from the TABLE in a local folder –Link with cryogenic signals link between Electrical circuits & cryogenic equipment foreseen to be established in the Layout DB used for PM Data browser, Cryo SCADA, CIET and Circuit Synoptics scripts generating a set of signals / circuit

9. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 9 Post Mortem System – scalability  Dedicated scalability tests + HWC experience have not shown any bottlenecks in the available PM infrastructure (data collection on PM server and data conversion) –in case it does in the future, further servers will be deployed  Current limitations mostly lie with the gathering of data on the client side  PM scalability @ example of AUG test 09 th July at 9:50:28 –49 ext FGC events, 77 self FGC events, 17 DQAMS events –The PM data was received on the server in two batches, each ~1min long: at 9:51 and 9:56. No errors on the server side, no client disconnections detected during the test. –Conversion time to pmd format ~50ms for 600A converter file, ~500ms for 60A converter file, ~40ms for DQAMS file

10. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 10 Circuit Synoptic

11. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 11 Circuit Synoptic  Circuit Synoptic an important tool during HWC, gathering necessary information related to circuit powering on a single screen  2 major improvements proposed for next sectors –Inclusion of 60A converters (60A SW power permit and FGC status) –Link of circuit synoptics with related cryogenic signals, e.g. CL temperatures, liquid levels in related DFBs, CRYO_START, CRYO_MAINTAIN, etc...  Requires considerable work on configuration DB for mapping of signals (together with ACR) + data publishing from CRYO system (ACR & CO), but could be exploited by circuit synoptic, CIET, CRYO SCADA, etc..  Would allow for ‚real-time‘ trends of e.g. circuit current vs. current lead temperature within PVSS, etc... Courtesy of F.Bernard

12. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 12 Circuit Synoptic v2 Courtesy of F.Bernard

13. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 13 Logging

14. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 14 LHC Logging Service The LHC Logging Services - Architecture Logging DB Measurement DB Technical Services DB HTTP JDBC PL/SQL Generic Usage Measurement Database  Contain raw time series data, at up to 2Hz rate  Store data for 7 days  Derive new values based on measured values  Sends filtered data of interest to Logging DB  Generate statistics on accelerator performance Logging Database  Contain relatively slow time series data  Store data online for at least 20 years 10g AS Oracle RAC Database Host  2 x SUN Fire V240  Dual 1 GHz CPU  4 GB RAM  Dual internal disks  Dual power supply  SUN StorEdge 3510FC disk array  2-Gb fiber channel bus  Dual RAID controllers 1 GB cache  12 x 146 GB 10k rpm FC disks  24 x 300 GB 10k rpm FC disks (2007)  SUN Solaris 9  Veritas Volume Manager 3.5  Oracle9i Enterprise Edition Release 9.2.0.7.0 60,000,000 records / day 129,000,000 records / day 147,000 records / day 6,800,000 records / day Today’s Data Rates Courtesy of C.Roderick

15. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 15 Logging - Infrastructure + Interface  HW infrastructure –current Logging service has to cope with rapidly increasing demands –initially intended to deal purely with LHC machine/beam related data; yet today, data comes from PS Complex, SPS, SPS-EA, CNGS, ATLAS, LHCb, CMS, CIET… –Worry about interdependency with LSA (currently on the same database server) + scalability for 8 parallel sectors –DM has launched a study and will present a proposal for the backend upgrades at a TC on August 16. HW upgrades are tough to be in place for October 2007...  Data extraction interface –technology behind the current data extraction tool allows for an easy web- access from inside and outside CERN, but is for this reason also specific and by now outdated –Already planned to be replaced new implementation, thus addressing the open issues such as compatibility, performance and flexibility –implementation has partially started but the increasing workload for projects such as LSA, hardware commissioning,.. has made it impossible to further progress up to now (estimated to 2man-months, due to specific knowledge no easy 'outsourcing‘ possible) –Currently on hold….

16. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 16 Logging & Data providers – example of CRYO data  The present Logging system is used by most players of HWC, which store their system data at rates and accuracies which are primarily defined by system experts and inherent limitations of their systems (while CO will discuss with every client to minimise data volumes for long-term storage if possible)  For the global event analysis, additional data of in this case the cryogenic system has been retrieved, but first attempts showed limited time resolution and accuracy  First meeting with experts revealed several possibilities of improvement in the controls infrastructure that deserve follow-up, but also inherent limitations of the acquisition electronics -> see also summery paper of Mike (to be released)  Requirements for data storage of system designers (doing process control & system setup) and HWC & MPP experts (doing event analysis) can be very different and should be discussed by the latter two in case the provided accuracy and data rates are not sufficient for HWC activities

17. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 17 The CIET system LT data 0.1 Hz FEC 5 Hz WFIP Signal Processing (median calculation) Delay 30 s SCADA Threshold (0.5%) db LT data 0.1 Hz FEC 5 Hz WFIP PLC The CRYO SCADA system 1Hz Time stamp PLC SCAD A Threshold (0.08%) Threshold (0.1%) SCADA Archive db Dead time (10s) Signal Processing (median calculation) Logging – CRYO data Courtesy of M.Koratzinos

18. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 18 CCC Environment + elogbook

19. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 19 Comments on working environment…  Generally, CCC environment very much appreciated by teams  Passwords, account-timeouts, CMF updates and limited access to GPN perceived as disturbing during HWC, but need to follow new CERN security policy  Unique way to start programs from Linux and Windows consoles not easily possible as many clients only run on Windows (PVSS) or Linux, but to be followed up  Printing from the different consoles perceived difficult  Installation of LabView 8.2 on Windows consoles to run PM application directly  Elogbook –Vital tool for tracking and re-construction of HWC activities (EICs can confirm, especially when multiple tests are ongoing and time is limited) –Operators have the good habit to accurately fill these logbooks –Especially for parallel commissioning, changing shifts and if tests are carried out manually (no LSA tracking) an asset  Do we need further μCCC during HWC?

20. Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 20 Conclusions and Outlook  Many CO services involved during sector 78 commissioning, which all gathered valuable input for further evolvement and improvements that will be digested and implemented during the next weeks (timing, infrastructure, PIC, WIC, CRYO SCADA, CIET, QPS SCADA, PMA, applications, WorldFIP, databases, Logging, etc...)  Further input needed from HWC and MPP to have asap a common view on priorities for further improvements, test automation & global analysis -> SACEC  Issues as databases, timing, etc not treated here but will be adressed with concerned teams -> SACEC  Feedback from equipment groups & this HWC review will be addressed and followed up within the CO group during a dedicated technical commitee next week