1. Ensemble Methods:
Bagging

2. 4 base classifiers 8 training instances Original order 1 2 3 4 5 6 7 8
Training set 1
Training set 2
Training set 3
Training set 4

3. Why ensembles?
Sometimes a learning algorithm is unstable, i.e., a little change in the training set causes a big change in the learned classifier.
Sometimes there is substantial noise in the training set.
By using an ensemble of classifiers, we don’t just depend on the decision of just one classifier.
Disadvantages
Time consuming
Over-fitting sometimes

4. 3 base classifiers

5. Combining Decision Tress and the NN Algorithm
Classify the instance using the NN algorithm
applied on the training instances associated with the classification nodes (leaves).
Outlook
sunny
overcast
rainy
Humidity
Windy
high
normal
false
true
yes

6.         Ensemble paradigm
Use m different learning styles to learn from one training data set.
Combine decisions of multiple classifiers using, e.g, weighted voting.
Training Data
Data1
Data m
Data2
       
Learner1
Learner2
Learner m
Model1
Model2
Model m
Model Combiner
Final Model

7. Learner1 = Learner2 = … = Learner m
Homogenous ensembles
Use a single, learning style but manipulate training data to make it learn multiple models.
Data1  Data2  …  Data m
Learner1 = Learner2 = … = Learner m
Different methods for changing training data:
Bagging: Resample training data with replacement
Boosting: Weigh individual training vectors
In WEKA, Classify=>Choose=>classifiers=>meta They take a learning algorithm as an argument (base classifier) and create a meta-classifier.

8. Bag size
Original training set size: n
No. of independent base classifiers: m
For each base classifier, randomly drawing n’ examples from the original data, with replacement
n’ usually < n
If n=n’, on average it will contain 63.2% unique training examples. The rest are duplicates.
Combine the m resulting models using simple majority vote.
Decreases overall error by decreasing the variance in the results due to unstable learners, algorithms (like decision trees) whose output can change dramatically when the training data is slightly changed.

9. Bagging example: 2 classes

10. Class boundary by 1 decision tree

11. Boundary by 100 trees

12. Satellite Images Data Generated by NASA
Generated by NASA
Own by the Australian Centre for Remote Sensing
One frame of Landsat imagery consists of 4 digital images of the same scene in 4 different spectral (wavelength) bands.
Two of these are in the visible region: green and red
Two are in the near infra-red
A pixel in the image corresponds to 80m by 80m of real land
Pixel value = spectral band intensity
Pixel value = 0 means darkest
Pixel value = 255 means brightest

13. Record format
Example:
Each line of data corresponds to a 3x3 square neighborhood of pixels
Example:
Each line contains the pixel values in the 4 spectral bands
(3x3)x4 = 36 numbers
The last number indicates type of land
The records are given in random order so that you cannot reconstruct the original landscape

14. Class labels Class no. Type of land 1 red soil 2 cotton crop 3
grey soil 4
damp grey soil 5
soil with vegetation stubble 6
mixture type 7
very damp grey soil
There are no examples with class 6 in this particular dataset. The classification for each pixel was performed on the basis of an actual site visit by Ms. Karen Hall, when working for Professor John A. Richards, at the Centre for emote Sensing at the University of New South Wales, Australia.

15. Weka’s bagging Single classifier
Use satellite image training and test data
Classify test data using NaiveBayesSimple
Observe the outputs
Bagging
Classify=>Choose=>meta=>Bagging
Set bagSizePercent to 80
Try numIterations = 80
Observe error rate
Try numIterations = 90

16. Misclassification rates
CART: Classification And Regression Tree