1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Software and Hardware Inventory Initiatives Computer Security Team, Steve Traylen (IT-PES), Matthias Schröder (IT-OIS), Michał Kwiatek (IT-OIS)

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Agenda: – Goals and motivation – Computer Security background – Linux desktops – Quattor-managed Linux Clusters – Mac desktops – Windows computers – Feedback 2 Software and Hardware Inventory Initiatives

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Software and Hardware Inventory Initiatives Goals: Monitor the state and evolution of computers on the CERN site – Software and Hardware – Mac, Linux and Windows – Computer Centre and Personal Computers 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Motivation: Efficient Service Management – Ease software deployment – Precondition for Change Management Ease User Support – Provide tools to Service Desk Protect computers from security risks – Improve (automate) our insight in software vulnerabilities across CERN – Keep computers up to date – Promptly respond to new threats 4 Software and Hardware Inventory Initiatives

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Timely updating and patching is our 1st line of protection! Computer Security Team Software and Hardware Inventory Initiatives

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Background Any unprotected/unpatched/outdated computer connected to the Internet is likely to be infected within minutes! From OC5: “The user shall take the necessary precautions to protect his personal computer or work station against unauthorized access.” Timely updating and patching is the 1 st line of protection! This applies for MS Windows, but also to Linux and Macs. Worse: Attacks are moving away from the O/S and targeting now the application level. A central patch monitoring portal allows every user and service manager (as well as the Security Team ) to understand the security posture of their computer and servers. Areas for improvement and vulnerable computer/servers can be spotted in real-time and the corresponding user/manager can be quickly informed and asked for mitigation. 6 Computer Security Background

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Linux Desktops Matthias Schröder (IT-OIS) OS Patch Deployment Monitoring

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Background About active 4k nodes on site Automatic updates enabled by default – But easy to disable… – Kernel updates require reboot – Conflicts can block updates Basic configuration done via lcm – Ncm-components and local profiles – Relies on SW updates for changes No further central management No central backups 8 Scientific Linux Desktops

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Current situation OCS-inventory – Open source inventory software – Available for Mac, Linux, Windows and more – Data collectors running on clients Little load on client Available for many OS Configured via ncm-component – Reporting to central server Hardware of nodes Installed software Running kernel Keeps only snapshot User activity is not reported – Installed on all updating nodes 9 Scientific Linux Desktops

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 OCS host listing 10 Scientific Linux Desktops

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 OCS Summary Example 11 Scientific Linux Desktops

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 OCS Node Info Example 12 Scientific Linux Desktops

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Future steps Deployment started spring 2011 Next: – Develop queries for data mining – Extend CERN specific info 13 Scientific Linux Desktops

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Quattor-managed Linux Clusters Steve Traylen (IT-PES) Software and Hardware Inventory Initiatives

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Quattor Managed Background CERN CC contains quattor configured hosts: – SLC4 : SLC5 : SLC6 = 301 : 7375 : 32 – RHEL4 : RHEL5 : RHEL6 = 242 : 283 : 3 Managed as 117unique clusters. – Each cluster is pinned to an SLC snapshot date. e.g OSDATE=20110523. – Each cluster has it’s own package update policy. – Today time range of OSes are > 1 year. Quattor configuration only prescriptive. – It does what you ask, no matter what. 15 Quattor-managed Linux Clusters

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Quattor Current situation OSDATE Monitoring of CDB Clusters – Monthly email sent per cluster to each IT-Contact. – e.g lxplus: Cluster: lxplus Minimum OSDATE within lxplus is 201106XX Most frequently occurring OSDATE within lxplus is 201106XX Of a total 117 clusters lxplus is calculated as number 13 in the ordered list of most up to date clusters. This monitors configuration only not reality. – This monitoring is very imprecise, reality may be worse. General details on the OSDATE mechanism: http://twiki.cern.ch/twiki/bin/view/ELFms/OsUpdates http://twiki.cern.ch/twiki/bin/view/ELFms/OsUpdates 16 Quattor-managed Linux Clusters

17. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Quattor Managed Future steps Package Level Inventory – We need to know what is installed. For both security and operational reasons. – Results to be cluster neutral and correlated with RedHat CVE guidelines. Traditionally Pakiti has been solution. Pakiti produces a list of outstanding CVEs per node. OCSagents are being deployed across CC. – OCSagents collect everything Pakiti needs. An OCS collector can be added to report limited CDB data. – e.g cluster name, clustersub name. – Allow joins of OCS to existing DBs: CDB, SDB, …. 17 Quattor-managed Linux Clusters

18. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Quattor Managed Future steps Run Pakiti engine on extracted results of OCS database. – Pakiti client itself dropped, a duplication of collection. Web Interface for Pakiti results: – Views needed for security team and cluster managers. – Evaluate if Pakiti web-interface can be used or adapted. Early attempts were unusable, batch deluge results. – Evaluate if an existing CERN aware web-interface can be adapted to pakiti results. e.g. cluman, desktop DB (see later). – Create a new web-interface which is e-group, cdb cluster aware. Monthly Report – A monthly report of CVEs per cluster can be generated. Quattor and non-managed will be treated equally. – Pakiti results for SLC desktops will also be available. 18 Quattor-managed Linux Clusters

19. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Mac Desktops Matthias Schröder (IT-OIS) OS Patch Deployment Monitoring

20. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Background About 2k active clients on site System and main apps check for updates – But users can de-activate this – Users only reminded that updates available No central management No central configuration No central back-ups 20 Mac Desktops

21. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Current situation K2 to monitor usage of licensed SW – Only on nodes using licensed SW – Rather complete monitoring Hardware Software Can monitor usage of selected SW – Requires license per node 21 Mac Desktops

22. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 K2 Node List 22 Mac Desktops

23. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 K2 Licence Information 23 Mac Desktops

24. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 K2 Software List 24 Mac Desktops

25. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Future steps Plan to install OCS Inventory on all nodes – Gradual process Share OCS Server with Linux Need to keep K2 for licensed SW 25 Mac Desktops

26. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Computers Michal Kwiatek (IT-OIS) Software and Hardware Inventory Initiatives

27. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Background Windows computers at CERN: – 6000 Centrally Managed – 1500 Locally Managed – 1500 not in the CERN domain 27 Windows Computers Not in the CERN Domain In the CERN Domain Managed Centrally Locally

28. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Background Windows computers that belong to the CERN domain are managed with CMF CMF enables: – Deployment of the desired software configuration, incuding patches – When necessary, delegation of software deployment tasks to Local Administrators (ex. Experiments, Controls) – Reporting of the actual configuration of Windows Computers Requires manual configuration for unsupported apps 28 Windows Computers

29. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Background Every day, we actively assess the risk of security exploits of CERN computers 29 Windows Computers History of computers reinstalled because of detected security problems (per week)

30. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Background To manage software lifecycle, we must understand configurations across CERN 30 Windows Computers

31. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Current Situation 6000 Centrally Managed PCs and Servers – Monthly deployment of patches for OS and supported applications – Email alerts for owners of computers running unsupported applications with known security vulnerabilities 1500 Locally Managed computers – Monthly recomendation to Local Admins concerning patch deployment – Email alerts for Local Admins when their computers run a configuration with a known security flaw (ex. unsupported OS, no Antivirus) 1500 computers which are not in the CERN domain – Computers belonging to short-term visitors, managed by their respective owners (IT has no control) 31 Windows Computers

32. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Current Situation Microsoft patch deployment follow-up 32 Windows Computers

33. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Windows Current Situation Follow-up for unsupported applications 33 Windows Computers

34. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Future Steps DesktopDB – Initially designed to keep history of desktop configurations across all OS – Now extended to quattor-managed clusters in the Computer Centre 34 Windows Computers CMFOCS DesktopDB

35. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Future Steps DesktopDB – Evolution of SW and HW configurations – Across all OS: Windows, Mac and Linux Including Quattor-managed Linux Clusters – Prototype for ITIL CMDB data source Service Desk tool 35

36. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it IT Forum, June 2011 Feedback? Software and Hardware Inventory Initiatives