1. The 2005 UK Workshop on Computational Intelligence 5-7 September 2005, London L2-SVM Based Fuzzy Classifier with Automatic Model Selection and Fuzzy Rule Ranking Shang-Ming Zhou and John Q. Gan Department of Computer Science, University of Essex, UK

2. Background and Objectives(1/4) n The challenges : To apply SVM techniques to parsimonious fuzzy system modelling for regression and classification. Difficult to link the kernel functions in SVM to basis functions in fuzzy system. n Advantage of SVM: Parsimonious solutions based on quadratic programming

3. Background and Objectives(2/4) n Chen and Wangs work [Chen and Wang 2003]: Established this sort of relation for fuzzy classification based on L1-SVM techniques. Parameters: kernel parameters and regularization parameter not updated optimally from data for fuzzy rule induction. n One objective : To apply L2-SVM techniques to fuzzy system modelling to optimally learn the parameters from data in terms of radius- margin bound J; Radius-margin bound: not hold in L1-SVM.

4. n Rule ranking, rule selection: Rule base structure [Setnes and Babuska 2001] SVD-QR with column pivoting algorithm and pivoted QR decomposition method [Yen and Wang 1998,1999, Setnes and Babuska 2001]; Contribution of fuzzy rule consequents: More effective [Setnes and Babuska 2001] OLS [Chen et al 1991] Both rule base structure and contribution of fuzzy rule consequents: Highly desired [Setnes and Babuska 2001] Not reported yet in literature. Background and Objectives(3/4)

5. n Another objective: -values of fuzzy rules: Contribution of rule consequents; -values of fuzzy rules: Rule base structure and contribution of rule consequents. Background and Objectives(4/4)

6. L2-SVM based Fuzzy Classifier Construction (1/10) Fuzzy Classifier:

7. L2-SVM based Fuzzy Classifier Construction (2/10) n Conditions of Applying SVM to Fuzzy Classifier Construction: are Mercer kernel; If are generated from a reference function through location shift, then are Mercer kernel [Chen and Wang 2003]; leading to Gaussian MFs; Kernel parameters manually selected in [Cheng and Wang 2003].

8. L2-SVM based Fuzzy Classifier Construction (3/10) n L2-SVM based Fuzzy Classifier: Parameters optimally updated in terms of radius-margin bound: The number of rules L, prototypes, weights, bias, and scaling parameters.

9. L2-SVM based Fuzzy Classifier Construction (4/10) n Two quadratic programming problems: 1) st where are Lagrangian multipliers,

10. L2-SVM based Fuzzy Classifier Construction (5/10) 2) st n Radius-margin bound:

11. L2-SVM based Fuzzy Classifier Construction (6/10) n Automatic Model Selection Algorithm

12. L2-SVM based Fuzzy Classifier Construction (7/10) n Extraction Fuzzy Rules from L2-SVM Learning Results The number of fuzzy rules L is the number of support vectors; The premise parts of fuzzy rules: where is the jth element of the ith support vector. The consequent parts of fuzzy rules: where are the non-zero Lagrangian multipliers.

13. L2-SVM based Fuzzy Classifier Construction (8/10) n Fuzzy rule ranking based on L2-SVM learning R-values of fuzzy rules: [Setnes and Babuska 2001] Absolute values of the diagonal elements of matrix R in the QR decomposition of firing strength matrix; -values of fuzzy rules: Determining the depth of the effect of the rule consequent. -values of fuzzy rules: Considering both rule base structure and effect of the rule consequent.

14. L2-SVM based Fuzzy Classifier Construction (9/10) n Fuzzy rule selection procedure Evaluate the misclassification rates (MRs) of on the validation data set V and the test data set T separately: and ; Select the most influential fuzzy rules where is the threshold. Construct a fuzzy classifier by using the influential fuzzy rules selected.

15. L2-SVM based Fuzzy Classifier Construction (10/10) n Fuzzy rule selection procedure (cont.) Apply to the validation data set V and the test data set T to obtain new MRs and ; If >, stop selection; otherwise, assign a higher threshold value and go to step 2.

16. Experimental Results(1/6) n Benchmark problem-ringnorm 2 classes; 7400 samples; 20 attributes; Theoretically expected MR: 1.3% [Breiman 1998] 400 training samples; 5000 testing samples; 2000 validation samples. n Initial conditions: C=1; ; Learning rates for updating C and : 0.0001 and 0.01 separately Threshold for updating the radius-margin bound:

17. Experimental Results(2/6) n L2-SVM Induced Fuzzy Classifier: 249 fuzzy rules generated; MR: 1.32% on test data set; n Comparison with the well-known methods on generalization performance: AlgorithmsLDAQDAOLS-RBF with Gausian BFs OLS-RBF with Cauchy BFs MLPThe proposed MRs 24.54%2.6%2.52%3.12%13.0%1.32%

18. Experimental Results(3/6) n Fuzzy rule ranking results:

19. Experimental Results(4/6)

20. Experimental Results(5/6)

21. Experimental Results(6/6) Using R-value indexUsing -value index No. of rules selected 02491.45%1.32%02491.45%1.32%02491.45%1.32% 0.0012421.45%1.32%0.001901.45%1.32%0.0001901.45%1.32% 0.0022141.45%1.32%0.002891.50%1.32%0.0006891.45%1.32% 0.0031931.80%1.5%0.005881.55%1.38%0.0008881.50%1.34% Fuzzy rule selection results:

22. n To have applied L2-SVM to fuzzy rule induction for classification: Fuzzy rules optimally generated in term of radius-margin bound. Efficient way of avoiding the curse of dimensionality in high dimensional space. n Two novel indices for fuzzy rule ranking: Experimentally proved to be very effective in producing parsimonious fuzzy classifiers. Conclusions and Discussions(1/1)