1. 1 An Empirical Analysis of Vendor Response to Vulnerability Disclosure Ashish Arora, Ramayya Krishnan, Rahul Telang, Yubao Yang Carnegie Mellon University

2. 2 Motivation Information security breaches: A significant and increasing threat Lack of systematic policy for how vulnerability information should be disclosed Self reported security incidences

3. 3 Motivation While theoretical models are useful to understand the issues surrounding vulnerability disclosure, we need empirical estimates for policy making. One of the key factors is to understand how vendors respond to disclosure and disclosure policies? An empirical estimate on vendor response to disclosure window will be very useful in calibrating the current policies. However, data collection is non- trivial.

4. 4 Research goals Whether (and by how much) early disclosure induces vendors to patch faster. What are other key factors that condition patching time?

5. 5 Literature Arora, Telang, and Xu (2003) outline a model for the optimal policy for software vulnerability disclosure. Telang and Wattal (2004) show that disclosure is costly to vendors and hence provides incentives to vendors to improve the quality of their software Market based mechanism –Camp and Wolfram (2004) describe a means for creating market for vulnerabilities in order to increase the security of systems –Kannan and Telang (2004) show that markets always perform worse that CERT because of poor disclosure rules –Schechter (2002) argues that vendors should create and exploit a market for testers –Ozment (2004), an auction based market based mechanism

6. 6 Vendors face cost of patching. More time they have for patching less it costs them. Vendors’ customer incur loss when they are breached. Depending on the market structure, vendors “internalize” some of the customer loss. The more loss they internalize, more costs they incur and earlier is the patch. Disclosure of vulnerability is potentially hurtful to customers because disclosure makes it easier for hackers to find the information too. Thus disclosure threat supposedly forces vendors to patch faster because disclosure increases their costs. However, there is little (if any) empirical evidence that vendors indeed patch faster and by how much. Predictions of Analytical Model (Arora, Telang and Xu [2003])

7. 7 Model Prediction Besides understanding the role of disclosure, we also investigate other factors that have bearing on vendor response. Some of the factors are –Severity of the vulnerability –Vendor characteristics –Open source / closed source –Disclosure source –Publicly traded firm –Effect of September 11.

8. 8 Data Vulnerabilities published by SecurityFocus or CERT/CC. Information on the key time variables (Patching time = Date of patch – Date of notification). CERT provided us with information on when they notified the vendors. The date on which vendors delivered a patch to them etc. Vendor information from Hoover’s online business information database and vendor’s website Vulnerability information from the NIST ICAT database Time period from 9/26/2000 to 8/11/2003 1280 observations, related to 255 unique vendors and 303 unique ICAT database documented vulnerabilities

9. 9 CERT/CC Vs SecurityFocus Two major vulnerability disclosure sources CERT/CC (A Federal supported R&D center) –Typically 45 days of secret period after notifying vendors –No exploit code disclosed SecurityFocus (An online open forum) –Policy of instant disclose (many time individuals may provide vendors some time before disclosure) –Disclose full information We discard all vulnerabilities which are reported first by vendors

10. 10 Early disclosure Anytime vulnerability is disclosed within the disclosure window (mostly 45 days) and vendor has not patched, early disclosure happens. However, in our sample most of the time disclosure happens quite early. –Instant disclosure is a case when disclosure happens before or at the same time when vendor is notified of the vulnerability. “Not early” case on SecurityFocus –Identifiers tend to be careful in using this powerful instant disclosure tool. They inform the vendor first and wait for the vendor patch before posting on SecurityFocus website –30% in our sample “Early” case on CERT/CC –Disclosure by others in CERT/CC secret time period –Already known public when CERT/CC picked it up –A vendor was missed when CERT/CC notify other vendors –Disclosure before 45 days if 80% of the vendors are ready

11. 11 Impact of instant disclosure Without instant disclosureWith instant disclosure Patching Time (days)58.08(78.30)44.37(80.01) Severity Metric29.97(22.68)23.44(21.34) Obs / vuls48993791245 Published by CERT/CCPublished by SecurityFocus Patching48.41 (78.13)63.91 (94.79) Severity Metric27.38 (22.37)8.76 (4.48) Obs / vuls1181 / 25899 / 43 Impact of publication source

12. 12 Impact of disclosure source (for Instantly disclosed vuls) CERT/CCSecurityFocusOthers Patching Time24.46(36.96)42.95(78.59)59.42(97.71) Metric38.41(23.79)21.60(21.63)16.97(13.33) Obs / vuls1532439814724074 Disclosure by CERT has a significant impact on patching speed of the vendor than disclosure by Securityfocus or by other sources

13. 13 Vendor Characteristics MeanStd Dev No. of Employee2264075997 Open Source0.210.41 Public Firms0.420.50 There are total 255 unique vendors. Above statistics is based on the 121 vendors that we have reliable information. There are total 301 unique vulnerabilities. Average Severity Score was 16.25. Each vulnerability affected on an average 11 vendors. Vulnerability Characteristics

14. 14 Analysis Two sets of analysis –Impact of disclosure on patching time. Conditional on not having patched until time t-1, how will disclosure at time t will affect vendor’s patching speed. We choose different values of t. –Impact of expected “disclosure window” on patching time. How will change in disclosure window affect vendors’ patching behavior?

15. 15 T e = 0 daysT e = 4 - 7 days (1.1) Vendor fixed effect (1.2) Vendor characteristics (2.1) Vendor fixed effect (2.2) Vendor characteristics CERT -0.55 (0.18) -0.47 (0.17) 0.27 (0.21) 0.30 (0.18) Disclosure -0.78 (0.10) -0.86 (0.10) -0.50 (0.20) -0.43 (0.20) Firm Size 0.00 (0.01) 0.00 (0.02) Public firm -0.06 (0.13) 0.08 (0.14) Open source -0.59 (0.35) -0.52 (0.12) -1.05 (0.40) -0.20 (0.15) Severity metric -0.08 (0.04) -0.07 (0.04) -0.04 (0.06) -0.06 (0.05) Post September/11 -0.44 (0.11) -0.41 (0.10) 0.08 (0.15) 0.08 (0.14) Constant 4.44 (0.23) 4.37 (0.22) 3.81 (0.27) 3. 69 (0.25) R2R2 0.08830.08490.0254 0.0205 N 1280 388 Notes: * indicates significant at 10% level, ** indicates significant at 5% level and *** indicates significant at 1% level.

16. 16 Results Disclosure accelerates the patch delivery significantly. For vulnerabilities that are disclosed instantly, patch comes 55% faster than otherwise. When disclosure happens later the patch still comes significantly faster but the difference between with and without disclosure patching speed seems to reduce. Open source vendors tend to patch faster; almost 44% faster. Significant impact of 9/11. Patches come faster post 9/11.

17. 17 T e =0 days (1.1) Vendor fixed effect (1.2) Vendor characteristics C_C -1.02** (0.23)-0.95** (0.21) C_S -1.01** (0.20)-1.06** (0.18) C_O -0.63** (0.21)-0.60** (0.19) C_None -0.04 (0.20)0.04 (0.18) Firm Size -0.55* (0.35)0.00 (0.01) Public firm -0.06 (0.13) Open source -0.52** (0.12) Severity metric (log) -0.07* (0.04) Post September/11 -0.41** (0.11)-0.38** (0.11) Constant 3.92** (0.22)3.88** (0.21) R2R2 0.10120.0966 N 1280 Notes: * indicates significant at 10% level, ** indicates significant at 5% level Impact of Disclosure Source

18. 18 T e =0 daysT e =4 - 7 days (1.1) Vendor fixed effect (1.2) Vendor characteristics (2.1) Vendor fixed effect (2.2) Vendor characteristics C_C -1.02***(0.23)-0.95***(0.21)-0.69*(0.36)-0.65*(0.37) C_S -1.01***(0.20)-1.06***(0.18)-0.54(0.46)-0.38(0.46) C_O -0.63***(0.21)-0.60***(0.19)2.04***(0.74)1.64**(0.70) C_None -0.04(0.20)0.04(0.18)0.52***(0.19)0.47***(0.17) Firm Size 0.00(0.01)-0.01(0.02) Public firm -0.06(0.13)0.07(0.13) Open source -0.55*(0.35)-0.52***(0.12)-1.05***(0.39)-0.17(0.15) Severity metric (log) -0.07*(0.04)-0.07*(0.04)-0.05(0.05)-0.06(0.05) Post September/11 -0.41***(0.11)-0.38***(0.11)0.12(0.15)0.09(0.14) Constant 3.92***(0.22)3.88***(0.21)3.56***(0.25)3.56***(0.24) R2R2 0.10120.09660.0486 0.0482 N 1280 388 Notes: * indicates significant at 10% level, ** indicates significant at 5% level and *** indicates significant at 1% level.

19. 19 Impact of Disclosure Window “T” We now want to understand what is the impact of disclosure window on patching time. This is the information a policy maker like CERT needs. Before they decide how much time should be given vendors, they need to know what is impact of giving one additional day. CERT provides approximately 45 days. However, it is clear the most of the time disclosure happens much earlier. This means that expected disclosure window “T” is much smaller and is unobservable to econometrician. But we know that for all vulnerabilities that are disclosed instantly, T = 0. For all other, T>0. Thus these two samples should provide us with the directional effect of “T” on patching time.

20. 20 Impact of disclosure window “T” We use only CERT data to analyze this because CERT has a more well defined policy. We test whether there is significant difference between patching times for vulns instantly disclosed and otherwise in the CERT sample.

21. 21 With disclosure sourceWithout disclosure source (1.1) Vendor fixed effect (1.2) Vendor characteristics (2.1)Vendor fixed effect (2.2)Vendor characteristics Disclosure -0.83** (0.11) -0.93** (0.10) Disclosed_by_C -0.97** (0.17) -0.99** (0.15) Disclosed_by_S -0.94** (0.13) -1.09** (0.12) Disclosed_by_O -0.56** (0.14) -0.63** (0.14) Firm Size 0.00 (0.02) 0.00 (0.02) Public firm -0.07 (0.14) -0.08 (0.14) Open source -0.55* (0.36) -0.56** (0.13) -0.60* (0.36) -0.55** (0.13) Severity metric -0.06 (0.04) -0.05 (0.04) -0.07* (0.04) -0.06* (0.04) Post 9/11 -0.44** (0.12) -0.40** (0.11) -0.48** (0.12) -0.43** (0.11) Constant 3.86** (0.19) 3.92** (0.19) 3.94** (0.18) 3.97** (0.19) R2R2 0.09910.09530.0903 0.0878 N 1181 Notes: * indicates significant at 10% level, ** indicates significant at 5% level and

22. 22 Results Vendors are 56% faster when T = 0 compared to when T > 0. On an average the disclosure happens in our sample in 20 days. If we believe that the effect is linear then on an average, one day decrease in the disclosure window increases the patching speed by 2.8%.

23. 23 Conclusions We find that disclosure has significant and expected result on vendor’s patching behavior. There is a significant CERT effect. Involvement of CERT leads to faster patching time irrespective of disclosure. Open source vendors patch faster; more severe vulnerabilities are patched faster and there is a significant post 9/11 effect.