1 Enforcing Compliance: A Patch Management Strategy That Works

2 Copyright Notices ITIL® is a registered trademark of the UK Office of Government Commerce. ITIL® is a registered trademark of the UK Office of Government Commerce. Visible Ops is the trademark of the IT Process Institute ( Visible Ops is the trademark of the IT Process Institute ( All other trademarks and company names are the property of their respective owners. All other trademarks and company names are the property of their respective owners. This webinar is the property of Spafford Global Consulting, Inc. This webinar is the property of Spafford Global Consulting, Inc.

3 Overview Patching Challenges Patching Challenges Policies and Procedures Policies and Procedures Change Management Change Management Release Management Release Management Metrics Metrics Questions and Answers Questions and Answers

4 Question Do you have a Change Management process? Do you have a Change Management process? 1. Yes 1. Yes 2. We are in the process of developing one 2. We are in the process of developing one 3. No 3. No

5 Question How familiar are you with the Information Technology Infrastructure Library (ITIL) as it pertains to Service Support? How familiar are you with the Information Technology Infrastructure Library (ITIL) as it pertains to Service Support? 1. Not familiar 1. Not familiar 2. Have heard of it 2. Have heard of it 3. Somewhat familiar 3. Somewhat familiar 4. Familiar 4. Familiar 5. Very familiar, refer to it routinely. 5. Very familiar, refer to it routinely.

6 Patching Challenges Patches are time consuming to assess and apply. Patches are time consuming to assess and apply. Patches are released constantly. Patches are released constantly. They fail during installation. They fail during installation. They cause services to fail. They cause services to fail. One patch can undo another patch. One patch can undo another patch. They can introduce errors. They can introduce errors. Developing policies and procedures that work. Developing policies and procedures that work.

7 Policies and Procedures We want to use standards to reduce variation We want to use standards to reduce variation Standardization can help improve security and compliance postures while managing costs Standardization can help improve security and compliance postures while managing costs Policies and procedures need to be realistic and add value Policies and procedures need to be realistic and add value Shelfware achieves nothing – the policies and procedures must be feasible and make a positive difference (WIIFM) Shelfware achieves nothing – the policies and procedures must be feasible and make a positive difference (WIIFM) Getting started is the hardest part. Use your best and brightest people to develop policies and procedures. Getting started is the hardest part. Use your best and brightest people to develop policies and procedures. So … where do we start? So … where do we start?

8 Don’t rush to apply patches immediately! Uncontrolled change can do more harm than good! Many high-performing IT organizations do not rush their patches and, in fact, patch less frequently. Moreover, they do not patch production systems directly! They do so in pre-production.

9 Defense in Depth Think of the rings of walls in a castle. More walls equate to an overall better defensive posture. Think of the rings of walls in a castle. More walls equate to an overall better defensive posture. Processes, systems and people always have variation – go for layers. Processes, systems and people always have variation – go for layers. The idea is to layer controls in a cost effective fashion. The idea is to layer controls in a cost effective fashion. If the first control fails, then there is a second, etc. If the first control fails, then there is a second, etc. Compensating Controls Compensating Controls Firewall – perimeter and segments Firewall – perimeter and segments Network Segmentation Network Segmentation Intrusion Detection Systems Intrusion Detection Systems Log Monitoring / Alerting / Security Event Management (SEM) Log Monitoring / Alerting / Security Event Management (SEM) Antivirus/Anti-malware on clients, hosts, gateways Antivirus/Anti-malware on clients, hosts, gateways Integrity Management Systems Integrity Management Systems Control 3 Control 2 Control 1

10 What are we really talking about? Change and Release Management with support from Configuration Management

11 ITIL Definitions Change Management Is the set of standardized processes and tools used to handle change requests in order to support the business while managing risks. Change Management Is the set of standardized processes and tools used to handle change requests in order to support the business while managing risks. Release Management Uses formal controls and processes to safeguard the production environment. Release Management Uses formal controls and processes to safeguard the production environment. Configuration Management Focuses on tracking and documenting configurations and then providing this information to other areas including Change and Release Management. Configuration Management Focuses on tracking and documenting configurations and then providing this information to other areas including Change and Release Management.

12 Human Error is Huge! May 17, 2005 – Third annual CompTIA study shows human error still counts for the majority of security incidents – 79.3%. That number is virtually the same as May 17, 2005 – Third annual CompTIA study shows human error still counts for the majority of security incidents – 79.3%. That number is virtually the same as Comp TIA, Human error accounts for 80% of network availability issues. -- Stephen Elliott, Senior Analyst, Network and Service Management, IDC 2004 Human error accounts for 80% of network availability issues. -- Stephen Elliott, Senior Analyst, Network and Service Management, IDC 2004

13 Change Management is the organization’s last firewall against human error and malicious activity before production.

14 Phase I: Ungoverned Change time Change rate Failed changes and/or Number of unauthorized changes Unplanned work (Unplanned work > 100%) Source: IT Process Institute,

15 Phase I: Stabilized Patient time Change rate Failed changes or Num of unauth chgs Unplanned work Source: IT Process Institute,

16 What does this mean? If we can reduce the errors going into production, then unplanned work can be reduced. If we can reduce the errors going into production, then unplanned work can be reduced. If unplanned work is reduced, then projects can get done. If unplanned work is reduced, then projects can get done. If projects can get done, then, hopefully, IT is enabling the functional areas to move towards their objectives and the organization towards its goal. If projects can get done, then, hopefully, IT is enabling the functional areas to move towards their objectives and the organization towards its goal. By patching, or introducing change, IT should be adding value by enabling the business or assisting in the mitigation of risks. With uncontrolled change, IT adds risks. By patching, or introducing change, IT should be adding value by enabling the business or assisting in the mitigation of risks. With uncontrolled change, IT adds risks.

17 Process References For a definitive reference, see ITIL’s Service Support Volume For a definitive reference, see ITIL’s Service Support Volume Microsoft’s Operations Framework Microsoft’s Operations Framework British Educational Communications and Technology Agency (BECTA) British Educational Communications and Technology Agency (BECTA) IT Process Institute’s Visible Ops Methodology IT Process Institute’s Visible Ops Methodology

18 A Basic Change Management Process Identify a potential change Identify a potential change Create Request For Change (RFC) Create Request For Change (RFC) Seek Approval to Proceed Seek Approval to Proceed Plan the Change Plan the Change Plan & Prepare Plan & Prepare Test Test Develop Rollback Plan Develop Rollback Plan Peer Review Peer Review Seek Approval to Implement Seek Approval to Implement Deploy Deploy Review Review

19 What is Release Management? “The focus of Release Management is the protection of the live environment and its services through the use of formal procedures and checks.” – ITIL Service Support “The focus of Release Management is the protection of the live environment and its services through the use of formal procedures and checks.” – ITIL Service Support Release management is often squeezed between the development environment and production. Release management is often squeezed between the development environment and production. Production Environment Test Environment Development Environment Release Management

20 Release Management Processes To plan and oversee rollouts To plan and oversee rollouts Acceptance Testing Acceptance Testing Design and implement procedures for the distribution and installation of changes. Design and implement procedures for the distribution and installation of changes. Automation can reduce variation and speed deployment in known environments. Automation can reduce variation and speed deployment in known environments. This means that change, release and configuration management must work together. This means that change, release and configuration management must work together. To ensure only authorized and tested “releases” are deployed. To ensure only authorized and tested “releases” are deployed. Ensures that all master copies of software is stored in the Definitive Software Library (DSL) Ensures that all master copies of software is stored in the Definitive Software Library (DSL) Ensures that the Configuration Management Database (CMDB) appropriately reflects new Releases. Ensures that the Configuration Management Database (CMDB) appropriately reflects new Releases.

21 A Sample Process Development/Engineering/Security identifies potential patches. Development/Engineering/Security identifies potential patches. Change Management reviews the RFCs for the patches and, if approved, do the planning, testing, etc. Change Management reviews the RFCs for the patches and, if approved, do the planning, testing, etc. Approved patches/changes are reviewed and consolidated into a given release. Approved patches/changes are reviewed and consolidated into a given release. Integration testing is performed and requires effective Configuration Management. Integration testing is performed and requires effective Configuration Management. Once tested and accepted, approved releases are stored in the Definitive Software Library (DSL). Once tested and accepted, approved releases are stored in the Definitive Software Library (DSL). Releases and schedules are communicated. Releases and schedules are communicated. Operations then reviews the Release, formally accepts and deploys the Release from the DSL. Operations then reviews the Release, formally accepts and deploys the Release from the DSL. The more automated the deployment, the better as it reduces the possibility of human error but necessitates solid Change and Configuration Management. The more automated the deployment, the better as it reduces the possibility of human error but necessitates solid Change and Configuration Management. Patch 1Patch 2Patch 3 Release Planning Integration Testing Authorized Release Change Management

22 Metrics to consider Total Number of Changes Total Number of Changes Total Number of Emergency Changes Total Number of Emergency Changes Total Number of Service Affecting Outages Total Number of Service Affecting Outages % of Successful Changes (Meaning they installed according to plan) % of Successful Changes (Meaning they installed according to plan) Mean Time To Repair Mean Time To Repair Availability Availability Unplanned work Unplanned work

23 In Summary Patches are changes and must follow the organization’s Change and Release Management processes. Patches are changes and must follow the organization’s Change and Release Management processes. The goal is to manage risks and add value – not just to patch for the sake of patching. The goal is to manage risks and add value – not just to patch for the sake of patching.

24 Thank you! George Spafford Daily News Archive