Factors Impacting the Effort Required to Fix Security Vulnerabilities Lotfi ben Othmane, Golriz Chehrazi, Eric Bodden, Petar Tsalovski, Achim Bruker, Philip Miseldine 09 September 2015

Vulnerabilities Fixing Process at SAP Fixing Processes Released software  SAP security response process Under development software  Fixing process for security testing Participants include Central security teams IMS maintenance organization Security experts Developers ….

Introduction - The Problem Goal: Predict the time to spend on analyzing and fixing a given vulnerability? Let t = f (x1, ……xj) What are x1……xj ? SAP collects data about fixing security vulnerabilities. However, the data does not capture important issues such as team setup. This work collects information about how really security fixes are done through interviews of the experts. The results are the result of outside observers about the practices at SAP AG—they are not our own.

Introduction - Motivation Cost of implementing security fixes Vulnerabilities Average fix time (min) Dead Code (unused methods) 2.6 Poor logging: system output stream 2.9 XSS (stored) 9.6 Lack of authorization check 6.9 Unsafe threading 8.5 Null dereference 10.2 SQL injection 97.5 Cornell, RSA 2012 The only factor considered is vulnerability type What about the others?

The Study - Scope Goal: Identify the factors that impact the fixing time Method: Interview participants in the vulnerability fixing process Result: The major factors that impact the fixing time

The Study - Conduct of the Study Interviews were conducted from 8 to 12 Dec. 2014 Number of participants 12 (12 hours) 9 from Germany and 3 from India Security experts, developers, coordinators, project leaders NetWeaver experts, custom application experts, application experts Selection of participants Preparation of the questions Interviews Transcribe the interviews Code the interviews Consolidate the data Analyze the results

The Study - Conduct of the Study – Cont. Selection of participants Preparation of the questions Each interview is transcribed into about 16 pages Identified 21 code classes from 3 sample interviews Coded each transcript in a report of 4 pages Each interviewee is asked to review the report of his interview Interviews Transcribe the interviews Code the interviews Consolidate the data Analyze the results

The Study - Conduct of the Study - cont. Coding examples “Code injections are difficult to fix” vulnerability type Vulnerability characteristics “If the function module is the same in all these 12 or 20 releases […] , then I just have to do one correction” Similarity of code in the different releases Software structure

Factors that Impact the Vulnerability Fix time Factor categories # of factors Freq. Vulnerabilities characteristics 6 9 Software structure 19 10 Technology diversification 3 5 Communication and collaboration 7 8 Availability and quality of information and documentation Experience and knowledge 12 11 Code analysis tool 4 Other

Observed Fixing Process Case 1: Analysis and design of global solution Implemen-tation Test Release Pre-analysis Case 2: Analysis and design area solution This is the actual process. It maps the simplified process from the interviews, from the reality and not from the theoretical process in documents This is how the process is implemented in practice and observed by external analyst We found three cases for fixing issues Case 1: Require design of a global solution Impacts several products May better be addressed using API change Addressed by several teams and involves the central security team Case 2: Require design of a solution specific to a group of products Apply to a specific development area—e.g., mobile applications area The security expert of the area designs the solution with the developers Case 3: Local solution Apply to a specific vulnerability instance The developer fixes the issue with potential help from colleagues Case 3: Analysis and design local solution Implemen-tation Test Release Iterations among successive steps are performed implicitly / not marked

Take-Away Vulnerability type is one among many factors (65) that impact the vulnerability fix time The 8 factor categories reflect the main areas for improving the vulnerability fixing processes E.g., software structure, training, etc.

Threats to Validity Control of the threats to the validity of the results The interviewees are diversified 2 researchers coded each interview and the results are consolidated The participants validated the reports of their interviews Weaknesses Used one method to identify the factors—interviewing experts Interviewed only 2 developers External use Diversity of product areas Distribution of development teams

Lessons learned The main interview questions shall help the interviewees to tell their own stories “What” questions are inefficient to enumerate elements The participants sometimes have their own messages to deliver Vulnerability fixing processes are as many as the process participants Do not try to base the fix effort estimate on “a process”

Thank you lotfi.ben.othmane@sit.fraunhofer.de