Aleysha Becker Ece 539, Fall 2018 Using a Multi-Layer Perceptron Model to Predict MLB Post-Season Outcomes Aleysha Becker Ece 539, Fall 2018

PROBLEM Goal : predict outcome of MLB Postseason based on regular season statistics Tasks: PCA on feature vectors Train MLP (2008 – 2016) Test MLP (2017 - 2018) Success: Greater than 50%[1] Vegas[2], other Machine Learning Applications[3] can get just under 60% accuracy The goal was to predict the outcome of Major League Baseball Posteason matchups as a binary classifier based on regular season team statistics. This first task involved running PCA on the feature vectors The MLP was trained on 8 years worth of data Then the MLP was tested on 2 years worth of data My goal is to confirm others’ findings that machine learning applications for baseball game prediction can do better than random chance, and I’ll take that as a success. However industry best right now is just under 60%, Vegas prediction experts, as well as other machine learning applications (for example the one linked is a support vector machine) can get 58-59% when predicting baseball games

METHODS: DATA Statistics from BaseballReference.com[4] Manual Excel manipulation: Team statistics after PCA for each matchup were subtracted (Team1 – Team2) to get a differential input vector 1: Team1 won; -1: Team1 lost Order was chosen so that ~1/2 the input vectors had output 1 and ~1/2 had output -1 Statistics can be downloaded from BaseballReference.com in an Excel-friendly format, and then were manually manipulated in Excel both before and after PCA. Baseball is a very statistically rich sport with hundreds of different statistics and I used 82 different statistics for batting, pitching and fielding. Rather than choosing which statistics I thought were most applicable, I ran PCA to reduce the feature vector size from the initial 82 statistics down to 20. Then to get one input feature vector per matchup, the two teams’ statistics were subtracted to get a differential statistic feature vector. An output of 1 indicated that the first team won, and an output of -1 indicated that the second team won. Since all outcomes were known, order was chosen in an alternating fashion so that half of the input vectors had each class label. [5]

METHODS: PROGRAM PCA[6] and Back Propagation[7] programs from Professor Hu were used with slight modifications Heuristic experimentation: Number of hidden layers : [1, 3, 5] Neurons per hidden layer : [5, 10, 20] Learning Rate : [0.05, 0.1, 0.2] Momentum Constant : [0.7, 0.8, 0.9] Epoch Size : [24, 41, 64] I used professor Hu’s programs for PCA and Back Propogation learning with slight modifications. Professor Hu’s programs were chosen as they allowed for easy manipulation of the heuristic variables and perceptron layout, while the programs that I found in researching on Github were designed for a set neural net structure. His program allowed for experimentation with a wide variety of variables to see how those affected the outcome of the classifier. The actual heuristics I experiemented with were the number of hidden layers, the number of neurons per hidden layer, the learning rate, alpha, the momentum constant, and the epoch size between weight updates [8]

RESULTS Maximum Testing Classification Rate = 76% Best Classification was with: 1 or 3 Hidden Layers 5 Neurons per Layer Alpha = 0.1 Momentum = 0.8 Epoch size of 64 For each of the experimental heuristic values on the previous slide, I ran the program three times and shown here is the averaged classification rate for each condition over the three trials. I was able to achieve a maximum testing classification rate in a single run of 76% with the confusion matrix shown. The best classification results were with 1 or 3 hidden layers, 5 neurons per layer, a learning rate of 0.1, a momentum constant of 0.8, and an epoch size of 64. One thing in particular that I found interesting was the average classification rate didn’t change between 1 and 3 hidden layers.

DISCUSSION 76% classification rate was much higher than expected Not very repeatable – may be due to random chance and small testing set 10 trials with “ideal” classifier averaged 58.23% classification rate More testing data ! This model doesn’t take into account game-by-game variance The 76% classification rate was enormous and probably due to a random chance and the small testing set. I originally proposed using data from 2008-2017 to train and only 2018 to test, but modified that and used both 2017 and 2018 to test. However, that’s still only 16 test vectors, which means small differences in weight calculations can have a huge impact on your classification rate. After experimenting with the heuristics, I used the “ideal” set of conditions and did ten trials, and those averaged to a 58% classification rate, which is on par with most experts and other machine learning applications to baseball prediction. Future work would obviously be using a large testing set, and before turning in the final paper I plan to change the partition again and add another 2 years of data from the training set into the testing set, to get about the 60/40 training to testing ratio that is more on par with the industry standard. Additionally testing on regular season matchups as well could be a good next step. Finally, there are plenty of factors that this model doesn’t take into account, such as whether it’s a day or night game, significant player injuries, who’s pitching, team momentum, etc.

REFERENCES [1] Tim Elfrink, “Predicting the outcomes of MLB games with a machine learning approach,” 18-Jun-2018. [Online]. Available: https://beta.vu.nl/nl/Images/werkstuk-elfrink_tcm235-888205.pdf. [Accessed: 01-Oct-2018]. [2] Jia, R. Wong, C, et al. “Predicting the Major League Baseball Season,” 2013. [Online]. Available: http://cs229.stanford.edu/proj2013/JiaWongZeng-PredictingTheMajorLeagueBaseballSeason.pdf. [Accessed: 05-Dec-2018]. [3] Soto-Valero, C. “Predicint Win-Loss Outcomes in MLB Regular Season Games – a Comparative Study Using Data Mining Methods,” Dec-2016. [Online]. Available: https://www.researchgate.net/publication/311862823_Predicting_Win- Loss_outcomes_in_MLB_regular_season_games_-_A_comparative_study_using_data_mining_methods. [Accessed 05-Dec- 2018]. [4] “2018 Major League Baseball Season Summary,” Baseball Reference, 08-Oct-2018. [Online]. Available: https://www.baseball-reference.com/leagues/MLB/2018.shtml. [Accessed: 08-Oct-2018]. [5] image from : https://thehillnews.org/sports/marin-murphy/new-york-houston-la-chicago-analysis-mlb-postseason [6] Yu Hen Hu, myPCA, 2-February-2016. [Online]. Available: http://homepages.cae.wisc.edu/~ece539/matlab/mypca.m. [Accessed: 10-Nov-2018]. [6] Yu Hen Hu, bpconfig, 15-October-2003. [Online]. Available: http://homepages.cae.wisc.edu/~ece539/matlab/bpconfig.m. [Accessed: 10-Nov-2018]. [7] Image from : http://kindsonthegenius.blogspot.com/2018/01/basics-of-multilayer-perceptron-simple.html