Supervision of production computers DCS security Remote access to DCS Peter Chochula 9 th DCS Workshop, March 15, 2004 Geneva

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 2  Acknowledgments Presented information was collected from several sources. Main information source are discussions in FWWG, SUPC WG, CERN Control System Security Day (December and private communication with colleagues from IT (D. Heagerty, L.Cons, A.Pace, J.M.Jouanigot, R.D.G.aparicio …)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 3 Outline  This talk is an update to the talk given at DCS workshop in Heidelberg (2003)  New information is included on: Supervision on control computers Remote access to DCS systems

DCS Security

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 5 Understanding the risks In general, the DCS should be protected against:  malicious attacks attacks from hackers from outside “attacks” from ambitious users from inside  non-malicious mistakes typing mistakes ambitious system testers

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 6 What should we protect?  Communication between managers inside PVSS is considered to be secure  Open questions: DIM, OPC….  There is very little we can do against malicious attacks inside PVSS environment These topics, together with control computers supervision etc. will be addressed by a new working group (with ALICE participation)  FWWG aims to provide an access control component consisting of program libraries and recommendations (rules)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 7 Present situation  The risk of attacks on DCS network remains very high despite the fact that CERN security is very well organized: If we do nothing, the probability of an incident is 1.0 (Minutes from CERN Security day)  Both Linux and Windows computers are targeted by hackers  PLC become a “very popular” target in hackers community  There is a lack of resources for central support – experiments will need to act actively

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 8  Damage to DCS system can be very severe – ranging from data loss through damage to equipment up to damage to people.  Connection between ALICE DCS network and CERN campus network will be established – there is a risk of contamination and unauthorized activities  There are several requests to provide remote access to DCS systems mainly for monitoring purposes

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 9 Preventive measures  Deployment of minimal core operating system  Preventive scans (IDS, packet sniffing)  Controlled access to the network (advanced authentication methods, packet filtering)  Deployment of patches  Restrictive computer access rules (more viruses are entering CERN through main gate rather than through network). All users are subject to Operational circular No.5

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 10 Experiment specific topics  We will operate a 24/7 365/365 system which imposes some serious limitations Dangerous to deploy patches during data taking (we cannot interrupt some services) All patches must be tested in advance as the DCS is a non-standard environment Difficult to test patches for all possible interference with DCS system Impossible to force updates during OS bootstrap (system may be recovering from power cut and there is no time to install software updates at this point) Risk of interference with antivirus software

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 11 Additional complications  Some DCS applications are using ports targeted by viruses (e.g. OPC opens ports used by Blaster)  Lack of authentication in PLC protocols  Use of some software (e.g. port scanner) is violating the CERN rules (Oper. Circ. No.5)  Institutes need to deploy software updates We cannot allow laptops to connect to DCS network directly  Risk of contamination with removable media (e.g. memory sticks etc.)  …

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 12 Present strategy  Understanding of user requirements is essential – knowing the activities will help to search for anything suspicious  All remote access should be restricted to maximum possible extent, ports will be opened only if this is the last possibility  There is no ‘democracy” in DCS systems: anything not explicitly allowed is forbidden  A set of rules must be defined and agreed in the collaboration  We need to define and implement mechanisms for fast recovery after an incident (this involves backup policy, system installation, configuration of PVSS-II…)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 13 Current prototyping  We will participate in working group which will be established soon  In the mean time we are evaluating some solutions: Intrusion detection system (Snort) Security scan software (NeWt, Nessus) Remote deployment of patches and updates (SUS, Windows SMS, Landesk) The tools are evaluated on dedicated network in order to avoid clashes with CERN rules

DCS Access Control (FW developments – credits to S. Schmelling)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 15  Access control based on Domain User Group  Different levels of privileges: Monitor  the user is allowed to view the object but cannot change any parameters Control  the user is required to have this privilege level to be able to change a restricted range of a particular object’s parameters Debug  the user is required to have this privilege level to be able to change all parameters of a particular object Modify  the user is required to have this privilege level to be able to modify the structure of this particular object

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 16  Each object will be assigned by set of privileges needed to for requested action  Authentication will be guaranteed by Domain Controller Possibility to use NICE login  Inheritance of privileges similar to Windows strategy  Set of framework functions will be available to users Access control will be implemented at the level of HMI (each button will check if the requested operation has been authorized) First release will contain limited functionality (it should help users to start implementing access control to their panels) Full release is pending until new release of PVSS (encrypted libraries)

Remote access to DCS systems

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 18  Remote access should be allowed only for monitoring purposes  Any possibility of remote control presents a very high risk: Interference with running systems Damage to people or equipment (e.g. person setting the HV must be present in control room and make sure the nobody is touching the equipment)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 19  What kind of remote access do we need to implement? Do we need access to the application or to the machine running the application (remote terminal access) Do we need to access the published data remotely directly talking to servers? (OPC, DIM…)  We need to define, which traffic has to be allowed across CERN firewall

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 20 Solutions from remote access  Control screens and applications can be managed remotely via encrypted tunnels Locally installed applications encrypted inside SSH ( VNC (Virtual Network Computing) encrypted inside SSH ( CERN VPN encrypted connections ( allow remote computers to connect as if running on the CERN Campus Networkhttp://cern.ch/vpn

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 21 Remote Terminal Services  Windows offer a powerful method for remote access to computer – Windows terminal Services  We propose to run a Windows server with enabled remote access. Users can login to this server using their NICE account PVSS remote UI running on this server will provide access to application

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 22 Remote terminal services – licensing issues  DCS prototype server is included in CERNs licensing policy and allows for 255 concurrent sessions  We investigate a possibility to clone CERNs terminal service (cernts) and make one server dedicated for ALICE DCS (PVSS remote UI does not operate on standard cernts)  Client to terminal services exist for Windows, Unix/Linux, and Mac.  On client side a license fee might be required: Windows clients license is already included in the operating system license Other systems must pay ~ 6CHF/year Clarifying situation for users connecting via VPN ( no fee should be required)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 23 Remote access example Main PVSS-II applicationFED ServerDIM Server Terminal Server Remote UI project Remote client – outside CERN CERN Firewall RDP PVSS-PVSS DIM Sub-detector hardware

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 24 Additional remote access possibilities  Direct connection to machine running master project should be disabled  WEB interface – each project can run a web server allowing for remote access to datapoints User has in fact rewrite its application in HTML Standard web vulnerabilities  Publishing of data across firewall (e.g. DIM ) should be disabled Lack of authentication in protocols (DIM, OPC, PLC…) creates high risk It is very easy to make a non-malicious mistake

Supervision of production computers

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 26 Definition of the problem  All DCS computers will need remote supervision  This includes Monitoring of resources Monitoring of performance Monitoring of processes (presence)  High number of computers require automation  There are several approaches implemented at CERN

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 27 Operating systems and platforms  Mostly PC based computers Windows Linux  Special computers: PLCs Power supplies VME masters Readout controllers (FPGA executing Linux)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 28  Present efforts in ALICE concentrated on testing of individual system components - no real computer supervision implemented (yet)  Test systems are based on JCOP framework tool PCMON PCMON server executes on every supervised system (Windows or Linux) Gathered data are published via DIM Any DIM client can subscribed to monitored data PVSS is used as a main client platform – published data connected to datapoints

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 29 PCMON-PVSS connection LINUX WXP Dead PCMON connection

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 30 PCMON-PVSS connection

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 31 SUPC working group  Alice DCS relies on common solutions where applicable  Participation in SUPC WG  Aim is to identify solutions which could be implemented for ALICE. This includes all topics included in this talk (which go beyond the mandate of present group)

Peter Chochula 9 th ALICE DCS Workshop, March 15, 2004 Geneva 32 Conclusions  Computer supervision, administration, security and networking open a new field for ALIC DCS  Collaboration with IT has been launched Many problems have not yet been covered at all  ALICE DCS will adopt common solution where applicable  We understand complexity of presented problems, however without your feedback it is almost impossible to find optimal solution