1. Issues with Data Mining

2. Data Mining involves Generalization
Data mining (Machine Learning) learns generalizations of the instances in training data
E.g. a decision tree learnt from weather data captures generalizations about the prediction of values for Play attribute
This means, generalizations predict (or describe) the behaviour of instances beyond the training data
This in turn means, knowledge is extracted from raw data using data mining
This knowledge drives end-user’s decision making process

3. Generalization as Search
The process of generalization can be viewed as searching a space of all possible patterns or models
For a pattern that fits the data
This view provides a standard framework for understanding all data mining techniques
E.g decision tree learning involves searching through all possible decision trees
Lecture 4 shows two example decision trees that fit the weather data
One of them is a better generalization than the other (Example 2)

4. Bias Important choices made in a data mining system are
representation language– the language chosen to represent the patterns or models,
search method – the order in which the space is searched
model pruning method– the way overfitting to the training data is avoided
This means, each data mining scheme involves
Language bias
Search bias
Overfitting-avoidance bias

5. Language Bias
Different languages used for representing patterns and models
E.g. rules and decision trees
A concept fits a subset of training data
That subset can be described as a disjunction of rules
E.g classifier for the weather data can be represented as a disjunction of rules
Languages differ in their ability to represent patterns and models
This means, when a language with lower representation ability is used, the data mining system may not achieve good performance
Domain knowledge (external to training data) helps to cut down search space

6. Search Bias
An exhaustive search over the search space is computationally expensive
Search is speeded up by using heuristics
Pure children nodes indicate good tree stumps in decision tree learning
By definition heuristics cannot guarantee optimum patterns or models
Using information gain may mislead us to select a suboptimal attribute at the root
Complex search strategies possible
Those that pursue several alternatives parallelly
Those that allow backtracking
A high-level search bias
General-to-specific: start with a root node and grow the decision tree to fit the specific data
Specific-to-general: choose specific examples in each class and then generalize the class by including k-nearest neighbour examples

7. Overfitting-avoidance bias
We want to search for ‘best’ patterns and models
Simple models are the best
Two strategies
Start with the simplest model and stop building model when it starts to become complex
Start with a complex model and prune it to make it simpler
Each strategy biases search in a different way
Biases are unavoidable in practice
Each data mining scheme might involve a configuration of biases
These biases may serve some problems well
There is no universal best learning scheme!
We saw this in our practicals with Weka

8. Combining Multiple Models
Because there is no ideal data mining scheme, it is useful to combine multiple models
Idea of democracy – decisions made based on collective wisdom
Each data mining scheme acts like an expert using its knowledge to make decisions
Three general approaches
Bagging
Boosting
Stacking
Bagging and boosting both follow the same approach
Take a vote on the class prediction from all the different schemes
Bagging uses a simple average of votes while boosting uses a weighted average
Boosting gives more weight to more knowledgeable experts
Boosting is generally considered the most effective

9. Bias-Variance Decomposition
Assume
Infinite training data sets of the same size, n
Infinite number of classifiers trained on the above data sets
For any learning scheme
Bias = expected error of the classifier even after increasing training data infinitely
Variance = expected error due to the particular training set used
Total expected error = bias + variance
Combining multiple classifiers decreases the expected error by reducing the variance component

10. Bagging Bagging stands for bootstrap aggregating
Combines equally weighted predictions from multiple models
Bagging exploits instability in learning schemes
Instability – small change in training data results in big change in model
Idealized version for classifier
Collect several independent training sets
Build a classifier from each training set
E.g learn a decision tree from each training set
The class of a test instance is the prediction that received most votes from all the classifiers
Practically it is not feasible to obtain several independent training sets

11. Bagging Algorithm Involves two stages Model Generation Classification
Let n be the number of instances in the training data
For each of t iterations
Sample n instances with replacement from training data
Apply the learning algorithm to the sample
Store the resulting model
Classification
For each of the t models:
Predict class of instance using model
Return class that has been predicted most often

12. Boosting Multiple data mining methods might complement each other
Each method performing well on a subset of data
Boosting combines complementing models
Using weighted voting
Boosting is iterative
Each new model is built to overcome the deficiencies in the earlier models
Several variants of boosting
AdaBoost.M1 – based on the idea of giving weights to instances
Boosting involves two stages
Model generation
Classification

13. Boosting Model generation Classification
Assign equal weight to each training instance
For each of t iterations:
Apply learning algorithm to weighted dataset and store resulting model
Compute error e of model on weighted dataset and store error
If e=0 or e>=0.5
Terminate model generation
For each instance in dataset:
If instance classified correctly by model:
Multiply weight of instance by e/(1-e)
Normalize weight of all instances
Classification
Assign weight of zero to all classes
For each of the t (or less) models:
Add –log(e/(1-e)) to weight of class predicted by model
Return class with highest weight

14. Stacking Bagging and boosting combine models of the same type
E.g. a set of decision trees
Stacking is applied to models of different types
Because voting may not work when different models do not perform comparably well,
Voting is problematic when two out of three classifiers perform poorly
Stacking uses a metalearner to combine different base learners
Base learners: level-0 models
Meta learner: level-1 model
Predictions of base learners fed as inputs to the meta learner
Base learner predictions on training data cannot be input to meta learner
Instead use cross-validation results on base learner
Because classification is done by base learners, meta learners use simple learning schemes

15. Combining models using Weka
Weka offers methods to perform bagging, boosting and stacking over classifiers
In the Explorer, under the classify tab, expand the ‘meta’ section of the hierarchical menu
AdaboostM1 (one of the boosting methods) on Iris data classifies only 7 out of 150 incorrectly
You are encouraged to try these methods on your own