EGI-Engage Recent Experiences in Operational Security: Incident prevention and incident handling in the EGI and WLCG infrastructure Dr Linda Cornwall, STFC. HEPiX Spring th March 2015 Linda Cornwall, STFC1

WLCG and EGI The (Worldwide) LHC Computing Grid and The European EGI Infrastructure share a lot of the same resources Also share Security teams and activities 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 2

Contents Incident Prevention Policy definition Vulnerability handling Security monitoring Incident handling and incidents from the last year Evolving the work 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 3

Security Incident Prevention Far more work goes into preventing incidents than handling them Security Policy definition Software Security, especially Software Vulnerability handling Security monitoring - monitoring for known vulnerabilities and insecure configuration 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 4

Security Policy Definition Security Policy definition is carried out by the EGI Security Policy Group (SPG) Defines the behaviour expected from NGIs, Sites, Users and other participants to maintain a beneficial and effective working environment Output is various policy documents Parties read and sign, so that they know and understand what they should and should not do List of policy docs at: 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 5

Minimizing vulnerabilities in the infrastructure Handling vulnerabilities found/reported Main activity of the EGI Software Vulnerability Group Assessing software for vulnerabilities Formally and informally Preventing new vulnerabilities being introduced Developer education, awareness Considering new software to be used in the infrastructure 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 6

Software Vulnerability Handling Approved procedure (Under Revision) Anyone may report an issue By to This may be because they have found it in software Or it may be that it has been announced If it has not been announced, SVG contacts the software provider and the software provider investigates (with SVG member, reporter as appropriate ) 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 7

Relevance and Risk The relevance to EGI is considered, and what affect it could have Then it is risk assessed, and put in 1 of 4 categories ‘Critical’, ‘High’, ‘Moderate’ or ‘Low’ If it has not been fixed, target date set ‘Critical’ 3 days, ‘High’ 6 weeks, ‘Moderate’ 4 months, ‘Low’ 1 year 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 8

Advisory issued An advisory is issued when vulnerability is fixed if EGI SVG IS the main handler of vulnerabilities for this software, or software is in EGI UMD regardless of the risk. E.g. Grid Middleware, tools developed in EGI and collaborating projects If EGI is NOT the main handler, e.g.linux advisory only issued if ‘High’ or ‘Critical’ 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 9

SVG message – if you find a vulnerability If it is NOT public knowledge DO NOT Discuss on a mailing list – especially one with an open subscription policy or which is archived publically Post information on a web page Publicise in any way without agreement of SVG DO report to SVG via 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 10

High and Critical Vulnerabilities monitored Sites are monitored for ‘High’ and ‘Critical vulnerabilities. EGI CSIRT chases sites which are exposing ‘Critical’ vulnerabilities Sites may get suspended if they expose critical vulnerabilities and don’t respond Respond if asked to by IRTF/CSIRT For ‘High’ risk, up to the local NGIs. 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 11

Vulnerabilities reported during last year 42 new entries in vulnerability tracker (RT) 12 concerned Grid Middleware - 2 critical (1related to heartbleed, 1 related to perfsonar/cacti) 4 high 16 Linux – 3 critical (heartbleed, Shellshock, Kernel) 5 high 4 Cloud enabling – 3 high 6 VO software – 3 high Others include 1 high 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 12

Changing types of Vulnerabilities Until about 1 year ago most vulnerabilities concerned Grid Middleware Now more concerning VO specific software Including Data Protection issues VOs take it into their head to ‘monitor’ activities in a way that is traceable back to user Cloud specific software Less knowledge about this 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 13

Incident Handling Approved Incident handling procedure Incidents are handed by the Incident Response Task Force. Fortunately there are not many Incident prevention is quite successful 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 14

Incidents during last year (8) Primecoin mining (Policy violation) Open Hostkey leaking private information User cert mis-use Fed Cloud incident Due to bad endorsed VM UI compromised (4 user IDs compromized) Shellshock related compromises to Perfsonar nodes (multiple sites) Compromise due to port left open DDoS to some EGI services 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 15

Evolving the Security Work Evolving the security work is necessary due to e.g. The EGI federated Cloud Changing responsibility model Changing technology Long Tail of Science Different trust model Have some H2020 funding for EGI engage to carry out this evolution 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 16

Policy documents under revision Getting rid of ‘Grid’ Policies apply to all technology and services Acceptable use policy External draft – request for feedback and comments Use_Policy_March_2015 Security Policy for the endorsement and operation of Virtual Machine images Especially for Fed Cloud experience 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 17

New policy documents Data Protection Policy Formerly only had “Grid Policy on the handling of User Level Job accounting data Finding Data protection policy needed as User level data is being monitored and exposed inappropriately. Long Tail of Science Policy Related to allowing access other than by large VOs, IGTF certificates User sub-proxy. 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 18

Vulnerability handling evolution Now more software is coming into use where SVG members have no knowledge New members of SVG who know about cloud software, especially tools written within the community ‘Expert’ contact for all software Cloud enabling software deployed in the Fed Cloud VO software – assume VO security contact is responsible and know who to contact No more than 2 steps to the right person. 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 19

Software security checking For some community cloud enabling software have a detailed ‘Technology provider’ questionnaire For other software propose something simpler:-- License details How long will it be under security support? How are security problems reported? Are security problems announced? Check compliance with Data Protection policy Some other simple technical checks – e.g. is user input is validated, bad constructs – not obviously bad 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 20

Incident response evolution Changing responsibility model in the cloud will mean changes to incident response. A lot of work is going on including traceability – See Ian Collier’s talk 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 21

Questions?? 24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 22

24 th March 2015 Linda Cornwall, STFC. HEPiX Spring conference, Oxford 23