Selected Subjects on Controls System - Quality Assurance P.Charrue On behalf of the AB Controls Group LHC Machine Advisory Committee 16 June 2006

LHC Machine Advisory Commitee2 Preamble AB/CO was asked to talk about Quality Assurance - this is a wide subject Today we talk about a selection of subject which are representatives in the QA domains and which deserve attention from the management

16 June 2006LHC Machine Advisory Commitee3 Outline Overview of the Controls Infrastructure Network Security (CNIC project) LHC Application production Post-Mortem project Conclusions

Drawing here

16 June 2006LHC Machine Advisory Commitee5 Outline Overview of the Controls Infrastructure Network Security (CNIC project) LHC Application production Post-Mortem project Conclusions

16 June 2006LHC Machine Advisory Commitee6 The CNIC Working Group The Computing and Network Infrastructure for Controls working group was created by the CERN Executive Board  From the recommendations made by the Technical Network Management Working Group (Jul 2004) Delegated by the CERN Controls Board (Sep 2004)  “…with a mandate to propose and enforce that the computing and network support provided for controls applications is appropriate” to cope with security issues.  Mandate covers only control systems, not office computing Members from all CERN controls domains and activities  Service providers (Network, NICE, Linux, Computer Security)  Service users (AB, AT, LHC Experiments, SC, TS)

16 June 2006LHC Machine Advisory Commitee7 Networking at CERN General Purpose Network (GN)  For office, mail, www, development, …  No formal connection restrictions by CNIC Technical Network (TN) and Experiment Network (EN)  For operational equipment  Formal connection and access restrictions  Limited services available (e.g. no mail server, no external web browsing)  Authorization based on MAC addresses  Network monitored by IT/CS

Office development PC Trusted Application Gateways Home or remote PC CERN Firewall Connection to Internet INTERNET CERN Public Gateways (LXPLUS, CERNTS)

16 June 2006LHC Machine Advisory Commitee9 Possible Threats Malicious access  A hacker accessing our devices from outside  A deliberate attack  ‘Sniffing’ the data that transits on the TN Erroneous access  Un-intentional errors  Errors committed by CERN personnel in ignorance Control/Grant access from outside the CCC (Cern Control Center) ‘Anonymous’ traceability Generic accounts with weak password

16 June 2006LHC Machine Advisory Commitee10 Malicious access Not much protection possible from CO side against intentional and motivated security attack from outside or within CERN However the TN is relatively difficult to get into from outside without a CERN account IT security covers protection against these type of threads. CNIC is currently studying intrusion detection on TN

16 June 2006LHC Machine Advisory Commitee11 What can be done Security enhancement and traceability are possible at four different levels :  Communication Layer  Accounts  CNIC  Applications

16 June 2006LHC Machine Advisory Commitee12 Communication Implement a ‘role-based’ access to the equipment in the communication infrastructure Depending on WHICH action is made, on WHO is making the call, and from WHERE the call is issued, the access will be granted or denied This will allow for filtering, for control and for traceability of the access to the equipment

16 June 2006LHC Machine Advisory Commitee13 Accounts Forbid ‘anonymous’ generic accounts Enforce usage of accelerator-oriented accounts Enforce the password change regularly Limit operational accounts to CCC All these measures cost nothing They may be seen as constraints to the operators working habits

16 June 2006LHC Machine Advisory Commitee14 Main outcomes of CNIC 9 January 2006 : closure of the GPN TN connection  No communication allowed to cross the bridge except from TRUSTED hosts on the GPN to EXPOSED hosts on the TN  This reduced the TRUSTED hosts from 10’000+ to 2’000 NICEFC and LINUXFC deployed operationally on more than 200 hosts Restrict access to the Network Description Database (NETOPS) via identification More than 40 Application Gateways deployed Connection to the TN requires authorization MAC address authentication

Operator in the CCC Specialist access from home Access from the office inside CERN Office development PC Trusted Application Gateways Home or remote PC CERN Firewall Connection to Internet INTERNET CERN Public Gateways (LXPLUS, CERNTS) 3 typical Use Cases

16 June 2006LHC Machine Advisory Commitee16 Pending Studies Areas Critical Settings encryption  Discussions still on-going Authentication means (e.g. card readers in the consoles, bank-like authentication, …) Reduction of the Trusted list

16 June 2006LHC Machine Advisory Commitee17 Outline Overview of the Controls Infrastructure Network Security (CNIC project) LHC Application production Post-Mortem project Conclusions

16 June 2006LHC Machine Advisory Commitee18 Mandate The Controls group  provides core control functionality & applications (HWC sequencer, equip state, equip monitoring, SDDS,…) in collaboration with AB/OP  produces and maintains standard facilities (Logging, FDs, LASER, JAPC, SIS, BIC, OASIS, CCM, PM, …)  develops, maintains and supports UNICOS based applications (Cryo, QPS, PIC, WIC,..) for industrial control system  provides support for modeling of the Controls database (SPS, HWC, LEIR, LHC) and for the logging and measurement services (Timber, Meter) AB/CO is also providing development environment, tools and graphical components to be used by application developers, equipment and MD specialists FESA editor Java dataviewer General purpose graphical beans Java GUI frame LabVIEW development environment UNICOS frame Working sets & Knobs Jython Build and release tools Software support to developers

16 June 2006LHC Machine Advisory Commitee19 Frameworks for LHC Applications Three approaches in place to build applications  Beam based control applications Majority of applications Java infrastructure  Industrial control PLC/SCADA based applications UNICOS frame based on PVSS  Post Mortem data analysis Based on LabVIEW

LHC Java Applications and Core LSA Controls System Core FESA Equipments Controls Middleware Monitoring & Concentration LSA Trim Beam Steering Settings Generation BDI Applic Fixed Displays Controls Settings LSA API FESA Equipments FESA Equipments Standard Equipment Access (JAPC) Core applications Equipment and instrumentation applications Standard Equipment Access High-Level Services LSA Core

16 June 2006LHC Machine Advisory Commitee21 LHC Java Applications - Organization The work is done in a close collaboration with the OP group - we work in a team One single project in place (LSA) providing the common architecture Aim to use for LEIR, SPS, their transfer lines TT40, TI8, TI2, LHC HWC and operation Test/validate using every possible controls or operational milestone and several dry runs

16 June 2006LHC Machine Advisory Commitee22 Issues - Remote Access and Security Experts and on-call teams require access to LHC controls from outside the CCC Who has the right to modify LHC parameters?  Control of certain devices (Schottky) from other institutes is already requested (US-LARP collaboration) We need remote access and role based access policy and manpower to implement it Identification of the originating account and host has to be registered and propagated through all the chain (who and where from) Business logic between the GUI and the equipment has to react differently according to the origin of the request

16 June 2006LHC Machine Advisory Commitee23 Issues -Time allocated for Testing TT40/TI8, HWC, LEIR, CNGS and SPS ring will be used now to validate the LSA core and applications extensively Due to the most probable cancellation of the LHC Sector Tests end of 2006, AB/CO will :  organize scalability tests for the complete controls infrastructure  need well coordinated dry runs We request time allocated during LHC commissioning for the final tests of the deployed software

16 June 2006LHC Machine Advisory Commitee24 Issues - Resources Major core activities are staffed by temporary or departing staff The same application developers are working for HWC, LEIR, CNGS, PS and SPS startup LHC applications list documented but not fully staffed clearly showing lack of resources See Today 4 FTE from AB/CO/AP, 3 from OP and 1 associates are working on the LHC software production We need experienced Java software developers Since Apr’05 we actively seek for 6 more associates (3 for HWC and 3 for LSA) :  Hired 1 for HWC in April’06, 1 for LSA in July’06 and 1 for LSA in September  We still miss 2 for HWC and 1 for LSA

16 June 2006LHC Machine Advisory Commitee25 Outline Overview of the Controls Infrastructure Network Security (CNIC project) LHC Application production Post-Mortem project Conclusions

16 June 2006LHC Machine Advisory Commitee26 Post Mortem project mandate After a failure during the operation of the LHC, leading to a beam abort or a power abort, a coherent set of so called “Post Mortem” information will be collected from the various sub-systems to analyze the causes of failure. To be able to understand the failure before resuming LHC operation, the collected information needs to be analysed within a few minutes and this requires a highly automated data collection and analysis process. The Post Mortem system is aimed at providing the operators and system experts with data visualisation tools which can combine raw data and automatically analysed data.

16 June 2006LHC Machine Advisory Commitee27 Summary of systems with PM requirements

16 June 2006LHC Machine Advisory Commitee28 PMA: Data flow QPSPICPC LHC Raw data files Systems Result data Logging Alarms Other systems … PM viewer PM analyser Data bases PM server

PM used for LHC Hardware Commissioning Example: Automatic analysis of the QPS tests for quality assurance. 1.The quench detection signal gets driven over the 100 mV threshold. 2.View of QPS signals to see that the system triggered and the quench heaters fired. 3.Automatic analysis of the quench heater discharge (log scale) showing the results. 4.Automatic analysis of the event with “passed/not passed” indication

16 June 2006LHC Machine Advisory Commitee30 PM: Milestones 1.June ‘06: Data Viewer for QPS, PIC and PC data 2.Sept. ‘06: Extended PM data storage model for new clients 3.Sept. ’06:Dry run, correlation of QPS, PIC and PC data 4.Oct. ‘06:PM system scaling test, including BI, BT and RF 5.Nov. ‘06:HW commissioning analysis, as defined in LHC-D-HCP During‘07:Analysis for Beam Commissioning

16 June 2006LHC Machine Advisory Commitee31 Issues A successful PM system was developed for SM18 magnet quench analysis served as the base of the LHC PM system Recently a new Project Leader has been assigned due to succession planning and the scope has increased through data collection, storage, browsing and analysis Many technological choices and user interfaces still to be defined and solved We are rather late with the work due to the late arrival of user specifications.

16 June 2006LHC Machine Advisory Commitee32 Outline Overview of the Controls Infrastructure Network Security (CNIC project) LHC Application production Post-Mortem project Conclusions

16 June 2006LHC Machine Advisory Commitee33 Conclusions Network and Security :  Activities are well defined  Reduction of the TRUSTED list is not trivial  encryption, authentication and role based access need global coordination LHC applications  Framework well defined  There are issues on resources and on time for testing  Hiring JAVA experts is very difficult PostMortem  First operational version used in HWC for QPS, PIC and PC  Project changed leadership and mandate has been extended  Work is late