1. Rotem Golan Department of Computer Science Ben-Gurion University of the Negev, Israel

2.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

3. Competition overview A database of 60 music performers has been prepared for the competition. The material is divided into six categories: classical music, jazz, blues, pop, rock and heavy metal. For each of the performers 15-20 music pieces have been collected. All music pieces are partitioned into 20 segments and parameterized. The feature vector consists of 191 parameters.

4. Competition overview (Cont.) Our goal is to estimate the music genre of newly given fragments of music tracks. Input: A training set of 12,495 vectors and their genre A test set of 10,269 vectors without their genre Output: 10,269 labels (Classical, Jazz, Rock, Blues, Metal or Pop). One for each vector in the test set. The metric used for evaluating the solutions is standard accuracy, i.e. the ratio of the correctly classified samples to the total number of samples.

5.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

6. A long story You have a new burglar alarm installed at home. It is fairly reliable at detecting a burglary, but also responds on occasion to minor earthquakes. You also have two neighbors, John and Mary, who have promised to call you at work when they hear the alarm. John always calls when he hears the alarm, but sometimes confuses the telephone ringing with the alarm and calls then, too. Mary, on the other hand, likes rather loud music and sometimes misses the alarm altogether. Given the evidence of who has or has not called, we would like to estimate the probability of a burglary.

7. A short representation

8. Observations In our algorithm, all the values of the network are known except the genre value, which we would like to estimate. The variables in our algorithm are continuous and not Discrete (except the genre variable). We divide the possible values of each variables into fixed size intervals. The number of intervals is changed throughout the evolution. We refer to this process as the discretization of the variable. We refer to the Conditional Probability Table of each variable (node) as CPT

9. Naïve Bayesian Network

10. Bayesian Network construction Once we determined the chosen variables (amount and choice), their fixed discretization and the structure of the graph, we can easily compute the CPT values for each of the nodes in the graph (according to the training set). For each vector in the training set, we will update all the network’s CPTs by increasing the appropriate entry by one. After this process, we will divide each value with the sum of its row (Normalization).

11. Exact Inference in Bayesian Networks For each vector in the test set, we compute six different probabilities (Multiplying the appropriate entries of all the network’s CPTs) and chose the highest one as the genre of this vector. Each probability is for a different assumption on the genre variable value (Rock, Pop, Blues, Jazz, Classical and Metal).

12.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

13. Preprocessing I divided the training set into two sets. A training set – used for constructing each Bayesian Network in the population. A validation set – used for computing the fitness of each network in the population. These sets has the same amount of vectors for each category (Rock vectors, Pop vectors, etc.)

14. The three dimensions of the evolutionary algorithm The three dimensions are: Variables amount. Variables choice. Fixed discretization of the variables. Every network in the population is a Naïve Bayesian Network, which means that its structure is already determined.

15. Fitness function In order to compute the fitness of a network, we estimate the genre of each vector in the validation set, and compare it to it’s known genre. The metric used for computing the fitness is standard accuracy, i.e. the ratio of the correctly classified vectors to the total number of vectors in the validation set.

16. Selection In each generation, we choose population_size/2 different networks at most. We prefer networks that have the highest fitness and are distinct from each other. After choosing these networks we use them to build a fully sized population by mutating each one of them. We use bitwise mutation to do so. Notice that we may use a mutated network to generate a new mutated network.

17. Mutation Bitwise mutation. Parent: BitSet Dis Child: BitSet Dis 110001 101001 100150018 49000

18. Crossover Single point crossover. Parent 1: Parent 2: Child 1: Child2: 110001 4900018 111101 5510 015 111101 4910 015 110001 5500018

19. Results (Cont.) Model - Naive Bayesian Population size - 40 Generations - 400 Variables - [1,191] discretization - [5,15] First population score - 0.7878 Best score - 0.8415 Test Set score - 0.7323 Website’s score: Preliminary result - 0.7317 Final result - 0.73024 “Zeroes” = cpt_min/10

20. Observation Notice that there’s approximately 10% difference between my score and the website’s score. We will discuss this issue (over fitting) later on.

21. Adding the forth dimension The forth dimension is the structure of the Bayesian Network Now, the population includes different Bayesian Networks. Meaning, networks with different structures, variables choice, variables amount and Discretization array.

22. Evolution operations The selection process is the same as in the previous algorithm. The crossover and mutation are similar. First, we start like the previous algorithm (Handling the BitSet and the discretization array) Then, we add all the edges we can from the parent (mutation) or parents (crossover) to the child’s graph. Finally, we make sure that the child’s graph is a connected acyclic graph.

23. Results Model - Bayesian Network Population size – 20 Generations – Crashed on generation 104 Variables - [1,191] discretization - [2,6] First population score - 0.4920 Best score - ~0.8559 Website’s score : It Crashed

24. Memory problems The program was executed on amdsrv3, with a 4.5 GB memory limit. Even though the discretization interval is [2-6], the program has crashed due to java heap space error. As a result I decided to decrease the population size to 10 instead of 20.

25. Results (Cont.) Model - Bayesian Network Population size – 10 Generations – 800 Variables - [1,191] discretization - [2,10] First population score - 0.5463 Best score - 0.8686 Website’s score: Preliminary score - 0.7085

26. Results (Cont.) Model - Bayesian Network Population size – 10 Generations – 800 Variables - [1,191] discretization - [2,20] First population score - 0.5978 Best score - 0.8708 Website’s score: Preliminary score - 0.6972

28. Overfitting As we increase the discretization interval, my score increases and the website’s score decreases. One explanation can be that increasing the search space may cause the algorithm to find patterns with strong correlation to the specific input data I received. While these patterns has no correlation at all to the real life data. One possible solution is using k-fold cross validation

29. Final competition scores

30. My previous score

31.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

32. ECOC

33. ECOC properties

34. ECOC (Cont.) 12345678910111213141516171819202122232425262728293031 Rock1111111111111111111111111111111 Pop0000000000000000111111111111111 Blues0000000011111111000000001111111 Jazz0000111100001111000011110000111 Classical0011001100110011001100110011001 Metal0101010101010101010101010101010

35. Entropy

36. Recursive minimal entropy partitioning (Fayyad & Irani - 1993) The goal of this algorithm is to discretizes all numeric attributes in the dataset into nominal attributes. The discretization is performed by selecting a bin boundary minimizing the entropy in the induced partitions. The method is then applied recursively for both new partitions until a stopping criterion is reached. Fayyad and Irani make use of the Minimal-Description length principle to determined a stopping criteria.

37. RMEP (Cont.)

39. Results

40.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

41. Example of a decision tree

42. C4.5 algorithm

43. Splitting criteria

44. Results of c4.5 alone

45.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

46. A new prediction model

47. Results

48. Results (Cont.)

49. My new score

50.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

51. Boosting (AdaBoost) I’ve tried to use boosting as a tool for building an ensemble of Naïve Bayesian networks. Each of these networks were trained on different training set weights according to the AdaBoost algorithm. Intuitively, AdaBoost updated the training set weights in correlation with the performance of previous trained networks. The algorithm reduces the weights of instances which were correctly predicated by previous networks and increases the weights for instances which hadn’t been predicted correctly.

52. AdaBoost - training

53. AdaBoost - testing

54. AdaBoost - parameters

55. AdaBoost

56. K-fold Cross validation Cross-validation is a technique for assessing how the results of a statistical analysis will generalize to an independent data set In k-fold cross-validation, the original sample is randomly partitioned into k subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data.

57. K-fold Cross validation (Cont.)

58.  Competition overview  What is a Bayesian Network?  Learning Bayesian Networks through evolution  ECOC and Recursive entropy-based discretizaion  Decision trees and C4.5  A new prediction model  Boosting and K-fold cross validation  References

59. References