1. Recent lessons learned: Operational Security David Kelsey CCLRC/RAL, UK d.p.kelsey@rl.ac.uk GDB Meeting, BNL, 5 Sep 2006

2. 5 Sep 2006Operational Security2 Overview The EGEE Operational Security Coordination Team –OSCT (Ian Neilson – CERN - leading) The EGEE Grid Security Vulnerability Group –GSVG (Linda Cornwall – RAL - leading) Speed of updating CA configuration Recent Operational Security issues –Incident report –Globus vulnerability –What went well? –And what went badly?

3. 5 Sep 2006Operational Security3 OSCT First formal meeting – CERN – 21 Jun 2006 it was agreed that - OSCT Membership and mail lists The operational core of the OSCT is composed of 2 individuals (prime contact and deputy/alternate) from each ROC. Each ROC will provide a generic mailing list in which (at least) the 2 core contacts will be registered. For security matters personal contact is important so the names and contact details of the prime and alternate contacts will be gathered and circulated within the OSCT core members. The ROC generic mailing lists will be used to populate the mailing list project-lcg-security-support@... (alias project-egee- security-support@...) The security-support mail list will be is now registered in GGUS to receive Security Support tickets.

4. 5 Sep 2006Operational Security4 OSCT responsibilities Tickets assigned to security-support will be handled by the affected ROC A new role: OSCT Duty Contact (OSCT-DC) –Follows same schedule as the ROC-on-duty –Backup role also defined –Co-location eases communication –Will monitor all security tickets and ensure they are properly assigned –Attends the weekly Ops meetings Security Incident Handling –Agreed policy “Depending upon the severity, complexity, duration, and scope of an incident", defines a requirement for a Team Leader to be appointed

5. 5 Sep 2006Operational Security5 Incident handling OSCT members monitor the incident reporting list When an incident is reported, the OSCT contact for the ROC responsible for the affected/reporting site will have prime responsibility to evaluate the requirement for an incident handling team and ensure the team leader is appointed. The OSCT-DC and backup should monitor the situation and assume responsibility to coordinate in an appropriate timeframe depending on the severity of the incident.

6. 5 Sep 2006Operational Security6 OSG and EGEE Work is underway to share security contact information and incident information –Cross population of mail lists –EGEE sites in the OSG lists And vice-versa –Technical details still to be agreed Read access to GOC-DB etc

7. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Operational Security 7 GSVG Work started in GridPP (Linda Cornwall) In EGEE-II there is SA1 manpower specified for the “Grid Services Security Vulnerability and Risk Assessment” Task The aim is “to incrementally make the Grid more secure and thus provide better availability and sustainability of the deployed infrastructure” –This is recognition that it cannot be made perfect immediately Handling of Specific Vulnerability issues is the largest activity in this task

8. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Operational Security 8 Vulnerability issues Log issues people become aware of in database Include software vulnerabilities, issues arising from lack of functionality, deployment issues Members of the Risk Assessment Team (RAT) carry out a Risk assessment Divide into 4 categories of risk –Extremely Critical –High –Moderate –Low Set Target Date according to risk (fixed formula) Make public on target date

9. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Operational Security 9 GSVG status RAT has been formed and is active –Have been reviewing a number of current issues Status will be presented at the EGEE’06 conference Then seek EGEE PEB approval for the new mandate –Full responsible public disclosure on target dates

10. 5 Sep 2006Operational Security10 CA updating CA V1.7 announced by IGTF on 24 July (includes new UK CA) –EGEE announcement on 26 July –Sites asked to upgrade in 7 days CA V1.8 released on 7 August –Announced to EGEE on 8 Aug Problems reported on 23 Aug –new UK CA certs still cannot access some resources –some sites - even Tier 1s – still on V1.6 VERY important that all sites upgrade in a timely manor It was noted that the release notes in the IGTF distribution should be kept in the EGEE announcement –This happened for V1.8

11. 5 Sep 2006Operational Security11 Security Incident Many HEP sites affected by the recent incident –Local root compromises (on up to date machines) –Many compromised accounts (password sniffers) –Not a Grid attack as such but involved many LCG sites Timeline –RAL first to announce to Grid csirts list on 11 th July –Within days – MANY sites confirmed attack –Imperial London had been compromised earlier –SNO was attacked back in May

12. 5 Sep 2006Operational Security12 Lessons Don’t intend to review the technical details here What went well? –Many people worked very hard –collaboration was excellent –Sharing of necessary information was good –The Grid csirts list (and HEPIX security list) kept people informed

13. 5 Sep 2006Operational Security13 Lessons (2) What did not go so well? (matters for OSCT) UK site decided (on the basis of following guidance) not to inform the Grid csirts No incident handling team created (was it needed?) –But CERN took the lead (as far as I can tell) Private information leaked out on to several public mail lists and google searchable archives and web sites –Much effort taken in removing these –Future mails – include warning header and unique ID Discussion supposed to happen on “contacts” list not “csirts” list – much activity on csirts list Concern that sites who said they were not involved had not looked carefully enough Need to strive for the correct balance in Open vs Closed communication But must encourage sites to report

14. 5 Sep 2006Operational Security14 Globus Vulnerability voms-proxy-init, grid-proxy-init, myproxy-init are used to create proxy certificates to authenticate against Grid services. The current versions are affected by flaws caused by an insecure handling of temporary files during the generation of the proxy certificates. Consequently, under some circumstances, a local attacker could create carefully crafted files, in order to obtain the newly generated Grid proxy certificates of other users or to cause an arbitrary file writable for the user to be overwritten.

15. 5 Sep 2006Operational Security15 GSVG advisory The Grid Security Vulnerability Group strongly recommends that all sites upgrade the relevant components to the following versions BEFORE 2006/09/18. This advisory is (will be) available at the following URL http://glite.web.cern.ch/glite/packages/R3.0/updates Timeline: 2006-08-15 Vulnerability announced by Globus 2006-08-16 Initial response from the Grid Security Vulnerability Group 2006-08-16 Initial response from the VOMS developers 2006-08-18 Initial response from the VDT developers 2006-08-25 First updated sources received by the integration team 2006-08-29 All updated sources received by the integration team 2006-09-01 Updated LCG and gLite packages available 2006-09-04 Public disclosure

16. 5 Sep 2006Operational Security16 Review Lots of good work! (and done quickly) –Romain Wartel pushed the advisory definition Many others worked on analysis, fixing and testing Template for an advisory is now developed Lack of formal agreement yet on GSVG policy –Hindered choice of publication date and advice on how quickly to upgrade –This issue was published by Globus of course No GSVG web site yet (being worked on) –Where/how to publicise? –How can people subscribe to notification? UI’s affected (not as well controlled as Grid services) –How to enforce updates/check all updated? –Time will tell!