Presentation is loading. Please wait.

Presentation is loading. Please wait.

Intrusion Detection Evaluation

Published byKerry Burke Modified over 6 years ago

Similar presentations

Presentation on theme: "Intrusion Detection Evaluation"— Presentation transcript:

1 Intrusion Detection Evaluation
CS 5323 Intrusion Detection Evaluation Prof. Ravi Sandhu Executive Director and Endowed Chair Lecture 15 Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact!

2 IDS Categorization World-Leading Research with Real-World Impact!
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Table 1 © Ravi Sandhu World-Leading Research with Real-World Impact! 2

3 IDS Categorization World-Leading Research with Real-World Impact!
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Table 1 © Ravi Sandhu World-Leading Research with Real-World Impact! 3

4 Design Space Workloads Metrics Measurement methodology
© Ravi Sandhu World-Leading Research with Real-World Impact! 4

5 Workloads World-Leading Research with Real-World Impact! © Ravi Sandhu
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 1 © Ravi Sandhu World-Leading Research with Real-World Impact! 5

6 Workloads World-Leading Research with Real-World Impact!
Table III and IV Table II Figure 2 Publicly available traces DARPA 98, 99, 00 KDD 99 (derivative) Symantec onsite testing Table V Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 1 © Ravi Sandhu World-Leading Research with Real-World Impact! 6

7 Vulnerability and Attack Injection
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 2 © Ravi Sandhu World-Leading Research with Real-World Impact! 7

8 Honeypots World-Leading Research with Real-World Impact! © Ravi Sandhu
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 3 © Ravi Sandhu World-Leading Research with Real-World Impact! 8

9 Metrics World-Leading Research with Real-World Impact! Not discussed
in lecture Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 4 © Ravi Sandhu World-Leading Research with Real-World Impact! 9

10 Metrics: Basic World-Leading Research with Real-World Impact!
Dependent on base rate Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 4 © Ravi Sandhu World-Leading Research with Real-World Impact! 10

11 Receiver Operating Curve (ROC)
© Ravi Sandhu World-Leading Research with Real-World Impact! 11

12 Zero Reference Curve (ZRC)
© Ravi Sandhu World-Leading Research with Real-World Impact! 12

13 ROC Limitations Intrusion detection is not a binary yes/no problem
Unit of measurement is ambiguous Flow versus packet Does not account for base rate P(I) © Ravi Sandhu World-Leading Research with Real-World Impact! 13

14 Receiver Operating Curve (ROC)
Assumed ROC: fixes 1 point, 0.7 Detection rate, False alarm rate Others are reported results from literature All anomaly detectors are in Fig 2 Axelsson, Stefan. The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security (TISSEC) 3, no. 3 (2000): Figures 2 and 3 © Ravi Sandhu World-Leading Research with Real-World Impact! 14

15 Intrusion Detection Effectiveness
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 15

16 Intrusion Detection Effectiveness
ZRC Compare area under ROC curve Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 16

17 Intrusion Detection Effectiveness
Assumes Base rate, P(I) = 0.1 ZRC Compare area under ROC curve TFP: max acceptable false positive rate Compare area difference between PPVZRC and PPVIDS up to TFP Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 17

18 Metrics: Cost-Based These p1 p2 p3 are different, apply to false alert filter Cα: cost of false positive Cβ: cost of false negative C = Cβ/Cα p1 = P(A) p2 = P(I|A) p3 = P(I|¬A) Cexp = Min(CβB,(1-α)(1-B)) + Min(C(1-β)B,α(1-B)) Crec = CβB + α(1-B) Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. Figure 6 © Ravi Sandhu World-Leading Research with Real-World Impact! 18

19 Metrics: Cost-Based Assumptions: B = 0.1 C = 10 α, β same for
base IDS and its false alarm filter Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 19

20 Measurement Methodology
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 20

21 Case Study: Snort True positive rate = 2/8 = 0.25
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 21

22 Case Study: Snort World-Leading Research with Real-World Impact!
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 22

23 Case Study: Snort World-Leading Research with Real-World Impact!
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 23

24 Measurement Methodology
Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A. and Payne, B.D., Evaluating computer intrusion detection systems: A survey of common practices. ACM Computing Surveys (CSUR), 48(1), p.12. © Ravi Sandhu World-Leading Research with Real-World Impact! 24

25 Future Directions High speed IDSs
IDSs for virtualized environments (e.g., cloud) IDSs for detecting APTs (advanced persistent threats) IDSs for detecting zero day attacks © Ravi Sandhu World-Leading Research with Real-World Impact! 25

Download ppt "Intrusion Detection Evaluation"

Similar presentations

Intrusion Detection Systems (I) CS 6262 Fall 02. Definitions Intrusion Intrusion A set of actions aimed to compromise the security goals, namely A set.

Intrusion Detection Systems (I) CS 6262 Fall 02. Definitions Intrusion Intrusion A set of actions aimed to compromise the security goals, namely A set.
1 New Trends and Challenges in Computer Network Security Ravi Sandhu Executive Director and Endowed Professor September 2010

1 New Trends and Challenges in Computer Network Security Ravi Sandhu Executive Director and Endowed Professor September 2010
1 Authentication with Passwords Prof. Ravi Sandhu Executive Director and Endowed Chair February 1, 2013 © Ravi.

1 Authentication with Passwords Prof. Ravi Sandhu Executive Director and Endowed Chair February 1, © Ravi.
1 Plenary Panel on Cloud Security and Privacy: What is new and What needs to be done? Ravi Sandhu Executive Director and Endowed Professor December 2010.

1 Plenary Panel on Cloud Security and Privacy: What is new and What needs to be done? Ravi Sandhu Executive Director and Endowed Professor December 2010.
Defense Questions # of correlated attacks: under-estimated or over-estimated? Conservative estimation –Average across all the three dataset? Dataset w/

Defense Questions # of correlated attacks: under-estimated or over-estimated? Conservative estimation –Average across all the three dataset? Dataset w/
I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=

I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=
I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=

I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=
1 Privacy and Access Control: How are These Two Concepts Related? Prof. Ravi Sandhu Executive Director and Endowed Chair SACMAT Panel June 3, 2015

1 Privacy and Access Control: How are These Two Concepts Related? Prof. Ravi Sandhu Executive Director and Endowed Chair SACMAT Panel June 3, 2015
Testing Intrusion Detection Systems: A Critic for the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory By.

Testing Intrusion Detection Systems: A Critic for the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory By.
Intrusion Detection Systems. Definitions Intrusion –A set of actions aimed to compromise the security goals, namely Integrity, confidentiality, or availability,

Intrusion Detection Systems. Definitions Intrusion –A set of actions aimed to compromise the security goals, namely Integrity, confidentiality, or availability,
1 Cyber Security Grand Challenges and Prognosis Prof. Ravi Sandhu Executive Director and Endowed Chair

1 Cyber Security Grand Challenges and Prognosis Prof. Ravi Sandhu Executive Director and Endowed Chair
1. Introduction Generally Intrusion Detection Systems (IDSs), as special-purpose devices to detect network anomalies and attacks, are using two approaches.

1. Introduction Generally Intrusion Detection Systems (IDSs), as special-purpose devices to detect network anomalies and attacks, are using two approaches.
1 Big Data Applications in Cloud and Cyber Security Prof. Ravi Sandhu Executive Director and Endowed Professor UTSA COB Symposium on Big Data, Big Challenges.

1 Big Data Applications in Cloud and Cyber Security Prof. Ravi Sandhu Executive Director and Endowed Professor UTSA COB Symposium on Big Data, Big Challenges.
1 Virtualization Prof. Ravi Sandhu Executive Director and Endowed Chair February 7, 2014 © Ravi Sandhu World-Leading.

1 Virtualization Prof. Ravi Sandhu Executive Director and Endowed Chair February 7, © Ravi Sandhu World-Leading.
1 The Quest for Single-Sign On Prof. Ravi Sandhu Executive Director and Endowed Chair February 8, 2013 © Ravi Sandhu.

1 The Quest for Single-Sign On Prof. Ravi Sandhu Executive Director and Endowed Chair February 8, © Ravi Sandhu.
1 Cloud Computing and Security Prof. Ravi Sandhu Executive Director and Endowed Chair April 19, 2013 © Ravi Sandhu.

1 Cloud Computing and Security Prof. Ravi Sandhu Executive Director and Endowed Chair April 19, © Ravi Sandhu.
1 Figure 10-4: Intrusion Detection Systems (IDSs) IDSs  Event logging in log files  Analysis of log file data  Alarms Too many false positives (false.

1 Figure 10-4: Intrusion Detection Systems (IDSs) IDSs  Event logging in log files  Analysis of log file data  Alarms Too many false positives (false.
Ensemble Learning for Low-level Hardware-supported Malware Detection

Ensemble Learning for Low-level Hardware-supported Malware Detection
1 Panel on Data Usage Management: Technology or Regulation? Prof. Ravi Sandhu Executive Director and Endowed Chair DUMA 2013 May 23, 2013

1 Panel on Data Usage Management: Technology or Regulation? Prof. Ravi Sandhu Executive Director and Endowed Chair DUMA 2013 May 23, 2013
1 Security and Privacy in Human-Centric Computing and Big Data Management Prof. Ravi Sandhu Executive Director and Endowed Chair CODASPY 2013 February.

1 Security and Privacy in Human-Centric Computing and Big Data Management Prof. Ravi Sandhu Executive Director and Endowed Chair CODASPY 2013 February.

Similar presentations

Ads by Google