1. Learning with AdaBoost
Fall 2007

2. Learning with Adaboost
4/24/2017
Outline
Introduction and background of Boosting and Adaboost
Adaboost Algorithm example
Adaboost Algorithm in current project
Experiment results
Discussion and conclusion
4/24/2017
Learning with Adaboost
Xin Li Adaboost

3. Learning with Adaboost
4/24/2017
Outline
Introduction and background of Boosting and Adaboost
Adaboost Algorithm example
Adaboost Algorithm in current project
Experiment results
Discussion and conclusion
4/24/2017
Learning with Adaboost
Xin Li Adaboost

4. Learning with Adaboost
4/24/2017
Boosting Algorithm
Definition of Boosting[1]:
Boosting refers to a general method of producing a very accurate prediction rule by combining rough and moderately inaccurate rules-of-thumb.
Boosting procedures[2]
Given a set of labeled training examples ,where is the label associated with instance
On each round ,
The booster devises a distribution (importance) over the example set
The booster requests a weak hypothesis (rule-of-thumb) with low error
After T rounds, the booster combine the weak hypothesis into a single prediction rule.
4/24/2017
Learning with Adaboost
Xin Li Adaboost

5. Boosting Algorithm(cont’d)
The intuitive idea
Altering the distribution over the domain in a way that increases the probability of the “harder” parts of the space, thus forcing the weak learner to generate new hypotheses that make less mistakes on these parts.
Disadvantages
Needs to know the prior knowledge of accuracies of the weak hypotheses
The performance bounds depends only on the accuracy of the least accurate weak hypothesis
4/24/2017
Learning with Adaboost

6. background of Adaboost[2]
4/24/2017
background of Adaboost[2]
4/24/2017
Learning with Adaboost
Xin Li Adaboost

7. Learning with Adaboost
4/24/2017
Adaboost Algorithm[2]
4/24/2017
Learning with Adaboost
Xin Li Adaboost

8. Advantages of Adaboost
Adaboost adjusts adaptively the errors of the weak hypotheses by WeakLearn.
Unlike the conventional boosting algorithm, the prior error need not be known ahead of time.
The update rule reduces the probability assigned to those examples on which the hypothesis makes a good predictions and increases the probability of the examples on which the prediction is poor.
4/24/2017
Learning with Adaboost

9. Learning with Adaboost
The error bound[3]
Suppose the weak learning algorithm WeakLearn, when called by Adaboost, generates hypotheses with errors Then the error of the final hypothesis output by Adaboost is bounded above by
Note that the errors generated by WeakLearn are not uniform, and the final error depends on the error of all of the weak hypotheses. Recall that the errors of the previous boosting algorithms depend only on the maximal error of the weakest hypothesis and ignored the advantages that can be gained from the hypotheses whose errors are smaller.
4/24/2017
Learning with Adaboost

10. Learning with Adaboost
4/24/2017
Outline
Introduction and background of Boosting and Adaboost
Adaboost Algorithm example
Adaboost Algorithm in current project
Experiment results
Discussion and conclusion
4/24/2017
Learning with Adaboost
Xin Li Adaboost

11. Learning with Adaboost
4/24/2017
A toy example[2]
Training set: 10 points (represented by plus or minus) Original Status: Equal Weights for all training samples
4/24/2017
Learning with Adaboost
Xin Li Adaboost

12. Learning with Adaboost
4/24/2017
A toy example(cont’d)
Round 1: Three “plus” points are not correctly classified; They are given higher weights.
4/24/2017
Learning with Adaboost
Xin Li Adaboost

13. Learning with Adaboost
4/24/2017
A toy example(cont’d)
Round 2: Three “minuse” points are not correctly classified; They are given higher weights.
4/24/2017
Learning with Adaboost
Xin Li Adaboost

14. Learning with Adaboost
4/24/2017
A toy example(cont’d)
Round 3: One “minuse” and two “plus” points are not correctly classified; They are given higher weights.
4/24/2017
Learning with Adaboost
Xin Li Adaboost

15. Learning with Adaboost
4/24/2017
A toy example(cont’d)
Final Classifier: integrate the three “weak” classifiers and obtain a final strong classifier.
4/24/2017
Learning with Adaboost
Xin Li Adaboost

16. Learning with Adaboost
4/24/2017
Outline
Introduction and background of Boosting and Adaboost
Adaboost Algorithm example
Adaboost Algorithm in current project
Experiment results
Discussion and conclusion
4/24/2017
Learning with Adaboost
Xin Li Adaboost

17. Look at Adaboost[3] Again
4/24/2017
Learning with Adaboost

18. Adaboost(Con’d): Multi-class Extensions
The previous discussion is restricted to binary classification problems. The set Y could have any number of labels, which is a multi-class problems.
The multi-class case (AdaBoost.M1) requires the accuracy of the weak hypothesis greater than ½. This condition in the multi-class is stronger than that in the binary classification cases
4/24/2017
Learning with Adaboost

19. Learning with Adaboost
AdaBoost.M1
4/24/2017
Learning with Adaboost

20. Error Upper Bound of Adaboost.M1[3]
Like the binary classification case, the error of the final hypothesis is also bounded.
4/24/2017
Learning with Adaboost

21. How does Adaboost.M1 work[4]?
4/24/2017
Learning with Adaboost

22. Adaboost in our project
4/24/2017
Learning with Adaboost

23. Adaboost in our project
1) The initialization has set the total weights of
target class the same as all other staff.
bird[1,…,10] = ½ * 1/10;
otherstaff[1,…,690] = ½ * 1/690;
2) The history record is preserved to strengthen the updating process of the weights.
3) the unified model obtained from CPM alignment are used for training process.
4/24/2017
Learning with Adaboost

24. Adaboost in our project
2) The history record
weight_histogram(withHistory Record)
weight_histogram( without History Record)
4/24/2017
Learning with Adaboost

25. Adaboost in our project
3) the unified model obtained from CPM alignment are used for training process. This has decreased the overfitting problem.
3.1) Overfitting Problem.
3.2) CPM model.
4/24/2017
Learning with Adaboost

26. Adaboost in our project
3.1) Overfitting Problem.
Why the trained Adaboost does not work for bird 11~20?
I have compared:
I ) the rank of alpha value for each 60 classifiers
II) how each classifier has actually detected birds in train process
III) how each classifier has actually detected birds in test process.
The covariance is also computed for comparison:
cov(c(:,1),c(:,2))
ans =
K>> cov(c(:,1),c(:,3))
ans =
K>> cov(c(:,2),c(:,3))
ans =
Overfitted!
Train data is different from test data. This is very common.
4/24/2017
Learning with Adaboost

27. Adaboost in our project
Train Result (Covariance:6.4746)
4/24/2017
Learning with Adaboost

28. Adaboost in our project
Comparison:Train&Test Result (Covariance: )
4/24/2017
Learning with Adaboost

29. Adaboost in our project
3.2) CPM: continuous profile model; put forward by Jennifer Listgarten. This is very useful for data alignment.
4/24/2017
Learning with Adaboost

30. Adaboost in our project
The alignment results from CPM model:
4/24/2017
Learning with Adaboost

31. Adaboost in our project
The unified model from CPM alignment:
without resampled
after upsample and downsample
4/24/2017
Learning with Adaboost

32. Adaboost in our project
The influence of CPM for history record
4/24/2017
Learning with Adaboost

33. Learning with Adaboost
4/24/2017
Outline
Introduction and background of Boosting and Adaboost
Adaboost Algorithm example
Adaboost Algorithm in current project
Experiment results
Discussion and conclusion
4/24/2017
Learning with Adaboost
Xin Li Adaboost

34. Learning with Adaboost
Browse all birds
4/24/2017
Learning with Adaboost

35. Learning with Adaboost
Curvature Descriptor
4/24/2017
Learning with Adaboost

36. Learning with Adaboost
Distance Descriptor
4/24/2017
Learning with Adaboost

37. Learning with Adaboost
Adaboost without CPM
4/24/2017
Learning with Adaboost

38. Adaboost without CPM(con’d)
4/24/2017
Learning with Adaboost

39. Good_Part_Selected (Adaboost without CPM con’d)
4/24/2017
Learning with Adaboost

40. Adaboost without CPM(con’d)
The Alpha Values
Other Statistical Data: zero rate: ;
covariance: ; median:
4/24/2017
Learning with Adaboost

41. Learning with Adaboost
Adaboost with CPM
4/24/2017
Learning with Adaboost

42. Adaboost with CPM(con’d)
4/24/2017
Learning with Adaboost

43. Adaboost with CPM(con’d)
4/24/2017
Learning with Adaboost

44. Good_Part_Selected (Adaboost without CPM con’d)
4/24/2017
Learning with Adaboost

45. Adaboost without CPM(con’d)
The Alpha Values
Other Statistical Data: zero rate: ;
covariance: ; median:
0.0635
4/24/2017
Learning with Adaboost

46. Learning with Adaboost
4/24/2017
Outline
Introduction and background of Boosting and Adaboost
Adaboost Algorithm example
Adaboost Algorithm in current project
Experiment results
Discussion and conclusion
4/24/2017
Learning with Adaboost
Xin Li Adaboost

47. Conclusion and discussion
1) Adaboost works with CPM unified model;
This model has smoothed the trained data set and decreased the influence of overfitting.
2) The influence of history record is very interesting. It will suppress the noise and strengthen the WeakLearn boosting direction.
3) The step length of KNN selected by Adaboost is not discussed here. This is also useful for suppress noise.
4/24/2017
Learning with Adaboost

48. Conclusion and discussion(con’d)
4) The Adaboost does not rely on the trained order.
The obtained Alpha value has very similar distribution for all the classifiers.
There are two examples:
Example 1: four different train orders have obtained the Alpha as follow:
1) 6 birds Alpha_All1=
                                    ) 6 birds Alpha_All2=
                                    ) 6 birds Alpha_All3 =                                   ) 6 birds Alpha_All4=
                                   0.6212
4/24/2017
Learning with Adaboost

49. Conclusion and discussion(con’d)
4/24/2017
Learning with Adaboost

50. Conclusion and discussion(con’d)
Example 2: 60 parts from Curvature Descriptor, 60 from Distance Descriptor;
1) They are trained independently at first;
2) Then they are combined to be trained together.
The results are as follow:
4/24/2017
Learning with Adaboost

51. Conclusion and discussion(con’d)
4/24/2017
Learning with Adaboost

52. Conclusion and discussion(con’d)
5) how to combine the curvature and distance descriptor will be another important problem. Currently I can obtain nice results by combining them. 10 birds are all found.
Are they stable for all other class? How to integrate the improved Adaboost to combine the two descriptors? Maybe Adaboost will improve even further (for general stuff, for example, elephant or camel).
4/24/2017
Learning with Adaboost

53. Conclusion and discussion(con’d)
Current results without Adaboost:
4/24/2017
Learning with Adaboost

54. Conclusion and discussion(con’d)
6) How about the influence from the search order?
Could we try to reverse the search order?
My current result has improved by one more bird, but not too much.
7) How many models could we obtain from the CPM model?
Currently I am using only one unified model.
8) Why does the rescaled model not work?
(I do not think curvature is so sensitive to the rescale).
9) Could we try to boosting the Neural Network? ? ?
4/24/2017
Learning with Adaboost

55. Conclusion and discussion(con’d)?
10) Could we try to change the boosting function?
Currently I am using the Logistical Regression projection function to transmit the error information to Alpha value; anyway, there are many methods to do this work. For example:c45, decision stump, decision table, naïve bayes, voted perceptron and zeroR. etc.
11) How to use decision tree to replace Adaboost? I think this will impede the search speed; but I am not sure the quality. ?
4/24/2017
Learning with Adaboost

56. Conclusion and discussion(con’d)?
12) How about the fuzzy SVM or SVM to address this good parts selection problem?
13) How to understand the difference among good parts selected by computer and by human?
(Do the parts from computer program have the similar semantic meaning?)
14) How about the stability of Curvature and Distance Descriptors? ? ?
4/24/2017
Learning with Adaboost

57. Thanks!

58. Learning with Adaboost
4/24/2017
Reference
[1] Yoav Freund, Robert Schapire, a short Introduction to Boosting
[2] Robert Schapire, the boosting approach to machine learning; Princeton University
[3] Yoav Freund, Robert Schapire, A decision-theoretic generalization of on-line learning and application to boosting
[4] R. Polikar, Ensemble Based Systems in Decision Making, IEEE Circuits and Systems Magazine, vol.6, no.3, pp , 2006.
4/24/2017
Learning with Adaboost
Xin Li Adaboost