1. PANACEA Automating Attack Classification for Anomaly-based Network Intrusion Detection Systems
Damiano Bolzoni, Sandro Etalle, Pieter H. Hartel
Presented by - Vijendra Rana

2. Agenda What is Intrusion Detection System
Panacea its basic idea and its Architecture
Machine learning primer
Architecture revisited
N-grams in Detail
Algorithms SVM and ripper
Evaluation and testing
Conclusion

3. What is Intrusion Detection System
An intrusion detection system (IDS) is a device or software application that monitors a network or systems for malicious activity or policy violations.- Wikipedia
Signature based – searching network traffic for a series of bytes or packet sequences known to be malicious.
Anomaly based – Network behavior normal or Abnormal – we will look this in detail.

4. Signature based Works similar to a virus scanner.
This style of detection relies on rules and tries to associate possible patterns to intrusion attempts.
More accurate so less False positives.
Can only detect if the signature is available in the Database.
Whenever a new attack is identified a vendor can take anywhere from few hours to days to update their Signature database.

5. Anomaly based When this system raises an alert
The system just know that there is an abnormal behavior very little information.
No automatic or semi automatic approach currently available (This is the problem Panacea solve)
Hence for all the anomalies manually we have to check if the anomaly is correct (Which is not true in Most cases FALSE POSITIVE)
Then determine the class of the anomaly.

6. Panacea – automating the classification
Uses Machine learning system to automatically and systematically classify attacks detected by an Anomaly based detection System.

7. Basic Idea
Compare payload Sequences from an alert to previously collected data using some Algorithm.
Classify the anomaly detected automatically without using some predetermined method (Method is determined by the Data)
No manual intervention.

8. Its Architecture – 2 sub-components
Alert Information extractor (AIE)- It receives alerts from IDS preprocess the payload to extract the useful information (We will discuss this useful Information in detail later)
Attack classification engine – The information got from AIE is being used to automatically classify the attack.

9. But How this Automatic Classification Works
For this we need to understand a little bit of Machine Learning before discussing further.

10. Machine Learning Primer
Machine Learning is the study of computer algorithms that improve automatically through experience – By Tom Mitchell
Can you draw a line to separate these 2 classes
Through experience means with Data

11. What do you think which line classify better

12. Various steps in machine learning
Getting the Data (Most important)
Finding the relevant features
Define a scoring/evaluating function
Iterate to improve the score (Training)
Deploy (testing)
Improving the score is also called optimization.

13. Finding the cost of a house importance of features
# of rooms
# of people live in the house
# of bathrooms
Paint color on the walls
Neighboring community
And so on
Which one do you think are relevant
Feature Engineering

14. Scoring Function - number of errors

15. Back to Panacea- AIE (Alert information Extractor)
Extract relevant information from payload (Payload is the part of transmitted data that is actual intended message. Also Payload refers to the part of the load that pays for transportation.- Wikipedia)
2 requirements
The extraction function should be able to capture information which is relevant for Classification engine from Payload
The information should be enough to distinguish between the classes.

16. Extracting and storing relevant information
N-gram analysis helps in capturing features of data in an efficient way.
But N-gram feature space is huge (640 GB for 5-gram) here the data is binarized.
Bitmap is an option (640/5 = 128 GB for 5 gram)
Bloom filter (A probabilistic way to tell if a element is “there in the set” or “not there in the set”) along with Bitmap to overcome space dimension problem. (5 KB for 5 gram)
Bloom filter employs k different hash functions at the same time to decrease the probability of FALSE POSITIVE.

17. What is n-gram Say I want to classify the data based on the text
And my text has “New-York times”
New ,York, Times (1-gram)
New_York …..(2-gram)
New_York_times (3-gram)

18. 2 ways of getting the labeled data (extracted info., class) pair
Automatic mode – The payload and attack class of any alert is generated by the SBS (Signature based Intrusion detection System) is used.
Semi Automatic mode- A human operator can classify the alert raised by Anomaly based Intrusion detection system manually.

20. Classification engine – It is an algorithm 2 types
Support Vector Machine
Ripper (A Form of rule learning)

21. Understanding SVM – both lines are correct but which one is better

23. Ripper – learning rules (Play tennis or not)

24. Implementation Prototype is written in Java.
Weka platform is used in training phase as it has inbuilt support for SVM and Ripper algorithms.

25. Benchmark
Difficult to find suitable dataset to have a baseline as no previous research has addressed this.
3 different datasets to evaluate the accuracy
When working in automatic mode Snort (DSA- dataset A)
When using adhoc taxonomies and the manual mode (DSB)
Classifying the unknown attacks (The attacks on which the system was not trained DSC)

26. Evaluation Cross Validation Samples are partitioned into subsets
The analysis is performed alternatively on different subsets
In K fold cross validation the samples are partitioned into k subsets. The process is repeated k times using each of the k subset exactly once to validate the model
Usually the result is calculated by taking a mean

27. Data
3200 different attacks 14 classes

28. Data
1400 attacks

29. DATA
100 alerts over a period of 2 weeks

30. Testing Criteria Number of alerts processed during training
The length of n-grams selected (this is an important parameter as if it is very small less accurate, if large generalization is difficult)
The classification algorithm selected SVM or Ripper

31. Evaluation

32. Test Evaluation

33. Test Evaluation

34. Test Evaluation

35. Results
SVM worked better when few attacks are available and diversity is more.
Both SVM and Ripper can give confidence score. Lower confidence score means the classification is likely to be wrong.
Users can set minimum threshold on confidence and detect possible misclassification.
Lower confidence score data can be given to a expensive human operator to identify (And after classification again fed to the system as training point)

36. Conclusion
Designed a system that can systematically and automatically classify based on payload data.
Panacea works in an automatic way but can be manually tuned for precise classification.
Panacea will keep on improving with more and more data.