1. Asymmetric Gradient Boosting with Application to Spam Filtering
Jingrui He Bo Thiesson
Carnegie Mellon University Microsoft Research

2. Roadmap Background MarginBoost Framework
Boosting with Different Costs (BDC)
Cost Functions
BDC in the Low False Positive Region
Parameter Study
Experimental Results
Conclusion

3. Background Classification Boosting Symmetric Loss Function
Neural networks, Support Vector Machines
Ensemble classifier
Boosting
Symmetric Loss Function
The same cost for misclassified instances from different classes
Weak learner
Training data
Reweight

4. Boosting with Different Costs to the rescue!
Spam Filtering
Classification task
Logistic regression, AdaBoost, SVMs, Naïve Bayes, Decision Trees, Neural Networks, etc
Misclassification of good s: false positives
False positives more expensive than false negatives
Stratification
Spam s de-emphasized in the same way
Unable to differentiate between the noisy and characteristic spam s
Boosting with Different Costs to the rescue!

5. MarginBoost Framework Mason et al., 1999
Training set:
Strong classifier: voted combination of weak learners
Loss functional
Weak learner:
Classification result
Weight:
Margin
Correct prediction: +
Incorrect prediction: -
Sample average
of the cost function
Cost function:

6. MarginBoost Framework cont. Mason et al., 1999
To minimize the loss functional
NO traditional parameter optimization
Gradient descent in the function space
In iteration t with classifier , find the direction s.t decreases most rapidly.
Negative functional
derivative of S at
Indicator function at
Derivative of C with respect to the margin

7. MarginBoost Framework cont. Mason et al., 1999
If comes from some fixed parameterized class, it should maximize
maximizes the weighted margins for all the data points, where weight
Coefficient for
Line search; more sophisticated method
Stopping criterion
Maximum number of iteration reached

8. MarginBoost Specialization
Cost function +
Cost Function
Differentiable
Monotonically decreasing
AdaBoost
LogitBoost
Logistic
regression

9. MarginBoost Specialization cont.
Cost Function
Weak Learner:
Decision stumps
is the most discriminating feature in that iteration
Strong Classifier:
Output:
Upon convergence: logistic regression
Stop earlier: feature selection

10. Boosting with Different Costs
Advantages
Weights of mislabeled spam
Regular boosting
BDC
Weights of mislabeled ham
Regular Boosting
Larger and larger weights as more weak learners are combined
Large weights for moderately misclassified spam
Small weights for extremely misclassified spam
Always High

11. Boosting with Different Costs cont.
Cost Function
Ham:
Spam:
Weight for Training Instances

12. BDC at Low False Positive Region
Linear threshold
Noisy spam message
After one iteration in
regular boosting
After one iteration in BDC 4 3
High false positive region
Low false positive region

13. Parameter Study in BDC Noisy Data Sets
: the maximum cost for spam (stratification)
: the slope of the cost around
Noisy Data Sets
Noise probability 0.03
Noise probability 0.05
Noise probability 0.1

14. Parameter Study in BDC cont.
The effect of with
FN at FP 0.03
FN at FP 0.05
FN at FP 0.1
FN at FP 0.03
FN at FP 0.05
FN at FP 0.1

15. Experimental Results Data Methods for Comparison
Hotmail Feedback Loop data
Training set: 200,000 messages received between July 1st, 2005 and August 9th, 2005
Test set: 60,000 messages received between December 1st, 2005 and December 6th, 2005
Methods for Comparison
Logistic regression, regularized logistic regression
LogitBoost
LogitBoost and logistic regression with stratification

16. Experimental Results cont.
Weak learner: decision stumps
Weak learner: decision trees of depth 2

17. Conclusion MarginBoost in Email Spam Filtering
Logistic regression as a special instance
Smart feature selection in logistic regression
BDC: Asymmetric Boosting Method
Different cost functions for ham and spam
Misclassified ham always have large weight
Moderately misclassified spam have large weight
Extremely misclassified spam have small weight
Able to improve the false negative rates at the low false positive region

18. Thank you! www.cs.cmu.edu/~jingruih
Q&A
Thank you!