EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid-enabled parameter initialization for high performance machine learning tasks Kyriakos Chatzidimitriou, Fotis Psomopoulos and Pericles Mitkas Thessaloniki, Greece

Enabling Grids for E-sciencE EGEE-III INFSO-RI Introduction to the scope of work The algorithm –NeuroEvolution of Augmented Reservoir (NEAR) = NeuroEvolution of Augmented Topologies + Echo State Networks (NEAT + ESN) Questions we want to answer Testbeds –Supervised learning –Reinforcement learning Experimental setup Results obtained Conclusions Future work Presentation Overview Grid-enabled parameter initialization for NEAR 2

Enabling Grids for E-sciencE EGEE-III INFSO-RI Approach – Problem – Solution Reinforcement Learning paradigm  appropriate for agents Real world/complex tasks  Function Approximator Echo State Networks  Non-linear/Non-Markovian tasks Evolution and learning  adapt the reservoir to the problem at hand How?  NeuroEvolution (NEAT) and Temporal Difference learning Hierarchy of involved areas Reinforcement Learning Function Approximator Evolutionary Computation Neuro Evolution of Augmented Topologies Temporal Difference Neuro Evolution Neural Networks Recurrent NN Reservoir Computing Echo State Networks Introduction 3 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI Echo State Networks An Echo State Network (ESN) [Jaeger, 2001 & 2002] Input Linear Features Reservoir Non-linear Features - Memory Output Linear/Non-linear Combination Trainable Static 4 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI Start minimally & complexify Weight & structural mutation Speciation through clustering to protect innovation Crossover networks through historical markings on connections [Stanley, PhD, 2004] NeuroEvolution of Augmented Topologies 5 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI Use NEAT as a meta-search method Start from minimal reservoirs (1 neuron) Perform weight and structural mutation –Add neurons, add connections Maintain ESN constraints Apply speciation through clustering –Similarity metric ~ Reservoir’s Macroscopic Features (Spectral Radius, Reservoir Neurons & Sparseness) Apply crossover using historical markings on neurons Identical performance and in some cases better against “rival” algorithms Work under review NeuroEvolution of Augmented Reservoirs (NEAR) 6 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI We ask the following questions: Questions are formulated as sets of parameters Experimentation to answer them Questions Degree of elitism Selection With crossover Mutation only Reproduction Speciation Individualism Fitness Continuous complexification Survival of the fittest Crossover Sparse Dense Reservoir 7 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI problems 5 parameters 64 parameter sets 30 runs per experiment A total of evolutionary procedures Population of 100 individuals Evolutionary process of 100 generations networks were evaluated Set up Jobs AlgorithmParametersProblem ResultsAggregationProcessing 8 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI Time Series –Mackey-Glass –Multiple Superimposed Oscillator –Lorentz Attractor Predict the next value of sequence Train on sequence T Calculate fitness, 1/NRMSE, by feeding output to input on F chunks of sequence T Validate on sequence V Testbeds – Time Series 9 Grid-enabled parameter initialization for NEAR Mackey Glass MSO Lorentz Attractor

Enabling Grids for E-sciencE EGEE-III INFSO-RI Reinforcement Learning –Single and Double pole balancing Balance one pole, or two poles of different lengths and masses for more than time steps –2D and 3D mountain car Escape from a valley by moving the car in two or three dimensions, starting from random states  10 runs for 3D due to extremely large execution time Testbeds – Reinforcement Learning 10 Grid-enabled parameter initialization for NEAR

Enabling Grids for E-sciencE EGEE-III INFSO-RI Mackey-Glass Performance measure: Validation NRMSE Inconclusive Grid-enabled parameter initialization for NEAR 11 RankPerf.Elitis m CrossoverSpeciationComplexifySparse %YesNoYesNo %YesNoYesNo %No YesNo %Yes No %No Yes ………………… %YesNo Yes %YesNo %No Yes %No YesNo %NoYes

Enabling Grids for E-sciencE EGEE-III INFSO-RI MSO Performance measure: Validation NRMSE The most difficult task Know to be an especially difficult task for ESNs Even the best results exhibit poor error behavior The poor performance does not allow us to derive concrete conclusions Grid-enabled parameter initialization for NEAR 12 RankPerf.Elitis m CrossoverSpeciationComplexifySparse %No %No

Enabling Grids for E-sciencE EGEE-III INFSO-RI Lorentz Attractor Performance measure: Validation NRMSE Grid-enabled parameter initialization for NEAR 13 RankPerf.Elitis m CrossoverSpeciationComplexifySparse %YesNoYesNo %YesNoYes %No Yes %YesNo Yes %No Yes ………………… %YesNo %Yes No %YesNo Yes %YesNoYesNo %Yes No

Enabling Grids for E-sciencE EGEE-III INFSO-RI D Mountain Car Performance measure: avg # steps escaping the valley from 1000 random starting states Inconclusive Similar results versus classic CMAC SARSA and the recent NEAT+Q Grid-enabled parameter initialization for NEAR 14 RankPerf.Elitis m CrossoverSpeciationComplexifySparse %No YesNo %YesNoYes %NoYesNo %YesNo %YesNo ………………… %Yes NoYes %Yes No %Yes %Yes No %Yes No

Enabling Grids for E-sciencE EGEE-III INFSO-RI D Mountain Car Performance measure: avg # steps escaping the valley from 1000 random starting states Inconclusive Grid-enabled parameter initialization for NEAR 15 RankPerf.ElitismCrossoverSpeciationComplexifySparse %YesNoYes %No Yes %No %NoYes No %No Yes ………………… %NoYesNo %YesNo %YesNoYes %No Yes %YesNo Yes

Enabling Grids for E-sciencE EGEE-III INFSO-RI Single Pole Balancing Performance measure: # Nets evaluated Grid-enabled parameter initialization for NEAR 16 RankPerf.ElitismCrossoverSpeciationComplexifySparse %No %No Yes %NoYes %YesNo Yes %NoYesNo ………………… %Yes %Yes No %Yes NoYes %Yes NoYes %Yes NoYes

Enabling Grids for E-sciencE EGEE-III INFSO-RI Double Pole Balancing Performance measure: # Nets evaluated Important parameters: Elitism, Crossover Grid-enabled parameter initialization for NEAR 17 RankPerf.ElitismCrossoverSpeciationComplexifySparse %YesNoYesNo %Yes No %YesNoYes %YesNo Yes %YesNo ………………… %No YesNo %No YesNo %NoYes No %No %No Yes

Enabling Grids for E-sciencE EGEE-III INFSO-RI Summarize Testbeds very different Free lunch theorem holds for parameters Results inconclusive besides speciation –Actually good –Multiple ways of finding a good solution without in many environments without worrying much about specific parameter settings –The case the algorithm does not work well (MSO) is mainly due to the restriction of the model itself Grid-enabled parameter initialization for NEAR 18

Enabling Grids for E-sciencE EGEE-III INFSO-RI Execution period on Grid Execution for 1 run on Grid –MG ~ sec –MSO ~ sec –Lorentz ~ sec –2DMC ~ sec –3DMC ~ sec –SPB ~ sec –DPB ~ sec Total time ~ ~ 316 days of sequential execution time Experimentation period on Grid ~ 60 days –allowing for testing, errors, outage, inactivity periods etc. Grid-enabled parameter initialization for NEAR 19

Enabling Grids for E-sciencE EGEE-III INFSO-RI Add more testbeds to the grid search –Non-markov cases that make pole balancing and mountain car problems more difficult (Implememted) –Server Job Scheduling (Implemented) More research on speciation and clustering similarity metric Increase generations when searching for a suitable network for the MSO testbed Future Work 20 Grid-enabled parameter initialization for NEAR

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Thank you for your attention! Fotis Psomopoulos Intelligent Systems and Software Engineering Labgroup Informatics and Telematics Institute Electrical and Computer Eng. Dept. Centre for Research and Technology-Hellas Aristotle University of Thessaloniki Thessaloniki, Greece