1. Evolving Insider Threat Detection
Pallabi Parveen
Dr. Bhavani Thuraisingham (Advisor)
Dept of Computer Science
University of Texas at Dallas
Funded by AFOSR

2. Outline Evolving Insider threat Detection Unsupervised Learning

3. Evolving Insider Threat Detection
System log j
System traces
System Traces
weeki+1
weeki
Anomaly?
Feature Extraction
& Selection
Testing on Data from
weeki+1
Online learning
Gather Data from
Weeki
Feature Extraction
& Selection
Learning algorithm
Supervised - One class SVM, OCSVM
Unsupervised - Graph based Anomaly detection, GBAD
Ensemble based Stream Mining
Ensemble of Models
Update models

4. Insider Threat Detection using unsupervised Learning based on Graph

5. Outlines: Unsupervised Learning
Insider Threat
Related Work
Proposed Method
Experiments & Results

6. Definition of an Insider
An Insider is someone who exploits, or has the intention to exploit, their legitimate access to assets for unauthorised purposes

7. Insider Threat is a real threat
Computer Crime and Security Survey 2001
$377 million financial losses due to attacks
49% reported incidents of unauthorized network access by insiders

8. Insider Threat : Continue
Detection
Prevention
Detection based approach:
Unsupervised learning, Graph Based Anomaly Detection
Ensembles based Stream Mining

9. Evolving Insider Threat Detection
System log
Feature Extraction
& Selection
Anomaly? j
System traces
System Traces
weeki+1
weeki
Testing on Data from
weeki+1
Online learning
Gather Data from
Weeki
Feature Extraction
& Selection
Learning algorithm
Supervised - One class SVM, OCSVM
Unsupervised - Graph based Anomaly detection, GBAD
Ensemble based Stream Mining
Ensemble of Models
Update models

10. Related work
"Intrusion Detection Using Sequences of System Calls," Supervised learning by Hofmeyr
"Mining for Structural Anomalies in Graph-Based Data Representations (GBAD) for Insider Threat Detection." Unsupervised learning by Staniford-Chen and Lawrence Holder
All are static in nature. Cannot learn from evolving Data stream

11. Related Approaches and comparison with proposed solutions
Techniques Proposed By
Challenges
Supervised/Unsuper vised
Concept-drift
Insider Threat
Graph-based
Forrest, Hofmeyr
Supervised X √
Masud , Fan (Stream Mining)
N/A
Liu
Unsupervised
Holder (GBAD)
Our Approach (EIT)

12. Why Unsupervised Learning?
One approach to detecting insider threat is supervised learning where models are built from training data.
Approximately .03% of the training data is associated with insider threats (minority class)
While 99.97% of the training data is associated with non insider threat (majority class).
Unsupervised learning is an alternative for this.

13. Why Stream Mining
All are static in nature. Cannot learn from evolving Data stream
Current decision boundary
Data Stream
Data Chunk
Previous decision boundary
Normal Data
Anomaly Data
Instances victim of concept drift

14. Proposed Method
Graph based anomaly detection (GBAD, Unsupervised learning) [2] +
Ensemble based Stream Mining

15. GBAD Approach
Determine normative pattern S using SUBDUE minimum description length (MDL) heuristic that minimizes:
M(S,G) = DL(G|S) + DL(S)

16. Unsupervised Pattern Discovery
Graph compression and the
minimum description length (MDL)
principle
The best graphical pattern S minimizes the description length of S and the description length of the graph G compressed with pattern S
where description length DL(S) is the minimum number of bits needed to represent S (SUBDUE)
Compression can be based on inexact matches to pattern S1 S1 S1 S1 S1 S2 S2 S2

17. Three types of anomalies
Three algorithms for handling each of the different anomaly categories using Graph compression and the minimum description length (MDL) principle:
GBAD-MDL finds anomalous modifications
GBAD-P (Probability) finds anomalous insertions
GBAD-MPS (Maximum Partial Substructure) finds anomalous deletions

18. Example of graph with normative pattern and different types of anomalies
GBAD-P (insertion) G C G G G A B C D A B C D A B E D A B C D A B C D
GBAD-MPS
(Deletion)
GBAD-MDL (modification)
Normative
Structure

19. Proposed Method
Graph based anomaly detection (GBAD, Unsupervised learning) +
Ensemble based Stream Mining

20. Characteristics of Data Stream
Continuous flow of data
Examples:
Network traffic
Sensor data
Call center records

21. DataStream Classification
Single Model Incremental classification
Ensemble Model based classification
Ensemble based is more effective than incremental approach.

22. Ensemble of Classifiers+ C2
x,? + + C3
input -
Individual outputs
voting
Ensemble output
Classifier

23. Proposed Ensemble based Insider Threat Detection (EIT)
Maintain K GBAD models
q normative patterns
Majority Voting
Updated Ensembles
Always maintain K models
Drop least accurate model

24. Ensemble based Classification of Data Streams (unsupervised Learning--GBAD)
Build a model (with q normative patterns) from each data chunk
Keep the best K such model-ensemble
Example: for K = 3
Data chunks D1 C1 D2 C2 D4 C4 D5 C5 D3 C3 D6 D5 D4
Update Ensemble
Testing chunk
Model with Normative Patterns
Prediction C4 C5 C1 C2 C4 C3 C5
Ensemble

25. EIT –U pseudocode Ensemble (Ensemble A, test Graph t, Chunk S)
LABEL/TEST THE NEW MODEL
1: Compute new model with q normative
Substructure using GBAD from S
2: Add new model to A
3: For each model M in A
4: For each Class/ normative substructure, q in M
5: Results1  Run GBAD-P with test Graph t & q
6: Results2 Run GBAD-MDL with test Graph t & q
7: Result3 Run GBAD-MPS with test Graph t & q
8: Anomalies Parse Results (Results1, Results2, Results3)
End For
9: For each anomaly N in Anomalies
10: If greater than half of the models agree
11: Agreed Anomalies  N
12: Add 1 to incorrect values of the disagreeing models
13: Add 1 to correct values of the agreeing models
UPDATE THE ENSEMBLE:
14: Remove model with
lowest (correct/(correct + incorrect)) ratio
End Ensemble

26. Experiments 1998 MIT Lincoln Laboratory 500,000+ vertices
K =1,3,5,7,9 Models
q= 5 Normative substructures per model/ Chunk
9 weeks
Each chunk covers 1 week

27. A Sample system call record from MIT Lincoln Dataset
header,150,2, execve(2),,Fri Jul 31 07:46: , +
msec
path,/usr/lib/fs/ufs/quota
attribute,104555,root,bin, ,187986,0
exec_args,1,
/usr/sbin/quota
subject,2110,root,rjm,2110,rjm,280,272,
return,success,0
trailer,150

28. Token Sub-graph

29. Total False Positives/Negative
Performance
Total Ensemble Accuracy
# of
Models
Total False Positives/Negative
True Positives
False Positives
False Negatives
Normal
GBAD 9
920
K=3
188
K=5
180
K=7
179
K=9
150

30. Performance Contd.. 0 false negatives
Significant decrease in false positives
Number of Model increases
False positive decreases slowly after k=3

31. Performance Contd..
Distribution of False Positives

32. Performance Contd.. Summary of Dataset A & B Entry
Description—Dataset A
Description—Dataset B
User
Donaldh
William
# of vertices
269
1283
# of Edges
556
469
Week
2-8
4-7
Day
Friday
Thursday

33. Performance Contd..
The effect of q on TP rates for fixed K = 6 on dataset A
The effect of q on FP rates for fixed
K = 6 on dataset A
The effect of q on runtime For fixed K = 6 on Dataset A

34. Performance Contd.. The effect of K on runtime for
True Positive vs # normative substructure for fixed K=6 on dataset A
True Positive vs # normative substructure for fixed K=6 on dataset A
Performance Contd..
The effect of K on runtime for
fixed q = 4 on Dataset A
The effect of K on TP rates for fixed
q = 4 on dataset A

35. Evolving Insider Threat Detection using Supervised Learning

36. Evolving Insider Threat Detection
System log
Feature Extraction
& Selection
Anomaly? j
System traces
System Traces
weeki+1
weeki
Testing on Data from
weeki+1
Online learning
Gather Data from
Weeki
Feature Extraction
& Selection
Learning algorithm
Supervised - One class SVM, OCSVM
Unsupervised - Graph based Anomaly detection, GBAD
Ensemble based Stream Mining
Ensemble of Models
Update models

37. Outlines: Supervised Learning
Related Work
Proposed Method
Experiments & Results

38. Related Approaches and comparison with proposed solutions
Techniques Proposed By
Challenges
Supervised/Unsupervised
Concept- drift
Insider Threat
Graph-based
Liu
Unsupervised X √
Holder (GBAD)
Masud , Fan (Stream Mining)
Supervised
N/A
Forrest, Hofmeyr
Our Approach (EIT-U)
Our Approach (EIT-S)

39. Why one class SVM Insider threat data is minority class
Traditional support vector machines (SVM) trained from such an imbalanced dataset are likely to perform poorly on test datasets specially on minority class
One-class SVMs (OCSVM) addresses the rare-class issue by building a model that considers only normal data (i.e., non-threat data).
During the testing phase, test data is classified as normal or anomalous based on geometric deviations from the model.

40. Proposed Method One class SVM (OCSVM) , Supervised learning +
Ensemble based Stream Mining

41. One class SVM (OCSVM)
Maps training data into a high dimensional feature space (via a kernel).
Then iteratively finds the maximal margin hyper plane which best separates the training data from the origin corresponds to the classification rule:
For testing, f(x) < 0. we label x as an anomaly, otherwise as normal data
f(X) = <w,x> + bwhere w is the normal vector and b is a bias term

42. Proposed Ensemble based Insider Threat Detection (EIT)
Maintain K number of OCSVM (One class SVM) models
Majority Voting
Updated Ensemble
Always maintain K models
Drop least accurate model

43. Ensemble based Classification of Data Streams (supervised Learning)
Divide the data stream into equal sized chunks
Train a classifier from each data chunk
Keep the best K OCSVM classifier-ensemble
Example: for K= 3 D1 C1 D2 C2 D4 C4 D3 C3 D5 C5 D5 D6 D4
Labeled chunk
Data chunks
Unlabeled chunk
Prediction C5 C4
Addresses infinite length
and concept-drift
Classifiers C1 C4 C2 C3 C5
Ensemble

44. EIT –S pseudo code (Testing)
Algorithm 1 Testing
Input: A← Build-initial-ensemble()
Du← latest chunk of unlabeled instances
Output: Prediction/Label of Du
1: Fu Extract&Select-Features(Du)
//Feature set for Du
2: for each xj∈ Fu do
3. ResultsNULL
4. for each model M in A
Results Results U Prediction (xj, M)
end for
6. Anomalies Majority Voting (Results)

45. EIT –S pseudocode Algorithm 2 Updating the classifier ensemble
Input: Dn: the most recently labeled data chunks,
A: the current ensemble of best K classifiers
Output: an updated ensemble A
1: for each model M ∈ A do
2: Test M on Dn and compute its expected error
3: end for
4: Mn  Newly trained 1-class SVM classifier (OCSVM) from data Dn
5: Test Mn on Dn and compute its expected error
6: A  best K classifiers from Mn ∪ A based on expected error

46. Time, userID, machine IP, command, argument, path, return
Feature Set extracted
Time, userID, machine IP, command, argument, path, return
1 1: :1 8:1 21:1 32:1 36:0

47. PERFORMANCE…..

48. Performance Contd.. Updating vs Non-updating stream approach
False Positives
13774
24426
True Negatives
44362
33710
False Negatives 1
True Positives 9
Accuracy
0.76
0.58
False Positive Rate
0.24
0.42
False Negative Rate
0.1

49. Supervised (EIT-S) vs. Unsupervised(EIT-U) Learning
Performance Contd..
Supervised (EIT-S) vs. Unsupervised(EIT-U) Learning
Summary of Dataset A
Supervised Learning
Unsupervised Learning
False Positives 55 95
True Negatives
122 82
False Negatives 5
True Positives 12 7
Accuracy
0.71
0.56
False Positive Rate
0.31
0.54
False Negative Rate
0.42
Entry
Description—Dataset A
User
Donaldh
# of records
189
Week
2-7 (Friday only)

50. Conclusion & Future Work
Evolving Insider threat detection using
Stream Mining
Unsupervised learning and supervised learning
Future Work:
Misuse detection in mobile device
Cloud computing for improving processing time.

51. Publication Conference Papers:
“Insider Threat Detection Using Stream Mining and Graph Mining,” in Proc. of the Third IEEE international Conference on Information Privacy, Security, Risk and Trust (PASSAT 2011), October 2011, MIT, Boston, USA (full paper acceptance rate: 8%).
”Supervised Learning for Insider Threat Detection Using Stream Mining”, in 23rd IEEE International Conference on Tools with Artificial Intelligence (ICTAI2011), Nov. 2011,
Boca Raton, Florida, USA (full paper acceptance rate is 30%)

52. References
W. Eberle and L. Holder, Anomaly detection in Data Represented as Graphs, Intelligent Data Analysis, Volume 11, Number 6,
W. Ling Chen, Shan Zhang, Li Tu: An Algorithm for Mining Frequent Items on Data Stream Using Fading Factor. COMPSAC(2) 2009:
S. A. Hofmeyr, S. Forrest, and A. Somayaji, “Intrusion Detection Using Sequences of System Calls,” Journal of Computer Security, vol. 6, pp , 1998.
M. Masud, J. Gao, L. Khan, J. Han, B. Thuraisingham, “A Practical Approach to Classify Evolving Data Streams: Training with Limited Amount of Labeled Data,” Int.Conf. on Data Mining, Pisa, Italy, December 2010.

53. Thank You