URLDoc: Learning to Detect Malicious URLs using Online Logistic Regression Presented by : Mohammed Nazim Feroz 11/26/2013

Motivation  Web services drive new opportunities for people to interact, they also create new opportunities for criminals  Google detects about 300,000 malicious websites per month, this is a clear indication that these opportunities are being used by criminals  Almost all online threats have something in common, they all require the user to click on a hyperlink or type in a website address

Motivation  The user needs to perform sanity checks and assessing the risk of visiting a URL  Performing such an evaluation might be impossible for a novice user  As a result, users often end up clicking links without paying close attention to the URLs – this further makes them vulnerable to malicious websites on the web which in turn exploit them

Introduction  Openness of the web exposes opportunities for criminals to upload malicious content  Do techniques exist to prevent malicious content from entering the web?

Current Techniques  Security practitioners have developed techniques such as blacklisting in order to protect users from malicious websites  Although this approach has minimal overhead, it does not provide complete protection as about only 55% of the malicious URLs are present in blacklists  Another drawback of this approach is that malicious websites are not a part of the blacklist during the period before their detection

Current Techniques  Security researchers have done extensive research in order to detect accounts on social networks that are used for spreading messages that are malicious  The approach still does not provide thorough protection for users in areas such as social networks where the interaction is in real-time because there is a need to build a profile of malicious activity and the process can take a considerable amount of time

Current Techniques  Researchers from TokDoc have used a method that decides on a per-token basis whether a token requires automatic healing  Their work uses n-grams and length as features for detecting malicious URLs  This research builds on their idea by supplementing a set of their features with host-based features as the latter has exhibited a wealth of information that can be used

Approach  URLDoc classifies URLs automatically based on the lexical (textual) and host-based features  Scalable machine learning algorithms from Mahout are used to develop and test the classifier  Online learning is considered over batch learning  The classifier achieves 93-97% accuracy by detecting a large number of malicious hosts, with a modest false positive rate

Approach  If these predictor variables are correctly identified and the URLs metadata is carefully derived then the machine learning algorithms used can sift through tens of thousands of features  Online algorithms are preferred over batch-learning algorithms  Batch learning algorithms look at every example in the training dataset on every step and then update the weights of the classifier – a costly operation if the number of training examples is large

Approach  Online algorithms update the weights according to the gradient of the error with respect to a single training example  Online algorithms are able to process datasets far more efficiently than batch algorithms

Problem Formulation  URL classification lends itself naturally as a binary classification problem  The target variable y (i) can take one of two possible values-malicious or benign  For k predictor variables over all categories then there will be x1 (i),…, xk (i) ; this will result in a k-dimension feature vector characterizing the URL  The goal is to learn a function h(x)=y that maps the space of input values to the space of output values so that h(x) is a good predictor for the corresponding value of y

Problem Formulation  The two main phases involved in building a classification system  The first phase creates the model (i.e. the function h(x)) produced by the learning algorithm  The second phase makes use of that model to assign new data from the test dataset to its predicted target class  Selection of the training dataset and it’s predictor variables, the target classes, and the learning algorithm through which the classification system will learn are vital in the first phase of building the classification system  Predicted labels are compared with known answers to evaluate the classifer

Overview of Features  Lexical features  These features have values of both types-binary and continuous  These features include  Length of the URL  Number of dots in the URL  Tokens present in the hostname, primary domain, and path parts of a URL  Features in the hostname are further characterized as bigrams  Bigrams are able to capture a certain pattern on character strings permuted randomly and occurring in certain combinations  Example:  Bigrams: depts ttu, ttu edu

Overview of Features  Host-Based features  IP address of the URL – A Record  IP address of the Mail Exchanger – MX Record  IP address of the Name Server – NS Record  PTR Record  AS number  IP Prefix

Overview of Features  Malicious websites have exhibited a pattern of being hosted in a particular “bad” portion of the Internet  Example: McColo provided hosting for major botnets, which in turn were responsible for sending 41% of the world’s spam just before McColo’s takedown in November McColo’s AS number was  These portions of the internet can be characterized on a regular basis by retraining on the predictor variables  This allows keeping track of concept drift

Online Logistic Regression with SGD  Logistic regression is a very flexible algorithm as it allows the predictor variables to be of both types- continuous and binary  Mahout greatly helps in the learning process by choosing an optimum learning rate and thus allowing the classification system to converge to the global minimum

Online Logistic Regression with SGD  Online learning when compared to batch learning is usually much faster, adapts to changes in a continuous manner and is much better when the size of the training and test datasets are large  Support Vector Machines were considered but not chosen since they take a longer period of time to train when compared to Online Logistic Regression  Online Logistic Regression converges more quickly if malicious and benign URLs from the training dataset are presented in a random order

Feature Vector  Feature hashing is used in order to encode the raw feature data into feature vectors  In this approach, a reasonable size (i.e. dimension) is picked for the feature vector and the data is put into feature vectors of the chosen size  After carefully considering the datasets, the size of the feature vectors in the research is in the 100,000 dimension space

Feature Vector Example  The data is encoded into the feature vector as continuous, categorical, word-like, and text-like features using the Mahout API

Results 90/10 dataset split 80/20 dataset split Training/Test dataset split

Results Training/Test dataset split 50/50 dataset split Benign:Malicious

Other Approaches Attempted  Term Frequency – Inverse Document Frequency  A bag of words approach was used and term (lexical features) – document (URL) matrix was created  Online Logistic Regression is not affected by good word weighting  Clustering  The URLs are viewed as a set of vectors in vector space  Cosine similarity was used as the similarity measure between URLs  This research focused on classification over clustering since the target classes of the URLs was known – Clustering has known to be useful when the target classes are unknown

Future Work  Study the various features extensively and only use those with the highest contributions – Also add new features that would help in better classification  Try to use algorithms that can benefit from parallelization

Summary  A reliable framework for the classification of URLs is built  A supervised learning method is used in order to learn the characteristics of both malicious and benign URLs and classify them in real time  The applicability and usefulness of Mahout for the URL classification task is demonstrated, and the benefits of using an online setting over a batch setting are illustrated-the online setting enabled learning new trends in the characteristics of URLs over time

Questions ?