Neural Net Update Dave Bailey

What’s New You can now save and load datasets from a file e.g. saving the dataset: You can now save and load datasets from a file e.g. saving the dataset: NeuralNetDataSet myDataSet; … // Fill the DataSet std::ofstream ofs; ofs.open(“myfile.dat”,std::ios_base::out|std::ios_base::trunc); if (ofs.is_open()) { ofs << myDataSet; // Writes the dataset to the stream ofs.close(); }

What’s New (2) Loading a previously saved dataset: std::string filename(“myfile.dat”); NeuralNetDataSet myDataSet(filename); DataSets are saved to a plain text file So be warned, the files can be quite large if you have a lot of data in them You can use as many different datasets as you like to train the network DataSets are saved to a plain text file So be warned, the files can be quite large if you have a lot of data in them You can use as many different datasets as you like to train the network

Feature Fixes There was a problem with part of the conjugate gradients training algorithm Showed up as big jumps in the error function whilst training Due to an oversight (and a small bug) in the linear search that optimises the training step length There was a problem with part of the conjugate gradients training algorithm Showed up as big jumps in the error function whilst training Due to an oversight (and a small bug) in the linear search that optimises the training step length

Search Technique Remember that the network “learns” by The conjugate gradient method attempts to optimise λ and μ for each epoch Remember that the network “learns” by The conjugate gradient method attempts to optimise λ and μ for each epoch

Linear Search Optimisation is performed using a linear search along a direction in “weight space”, looking for the minimum of the error function Slope -σψ’(0)Slope σψ’(0) ψ(0)+step*μψ’(0) Slope ψ’(0) μ,σ are search parameters Try to end up here

Problem and Solution Basic assumption was that the minimum would be found for positive step lengths Not true in real life as the starting point could be anywhere in weight space Line search is now bi-directional i.e. can search backwards up the direction vector Convergence much more stable and the problem of jumping error function is gone Basic assumption was that the minimum would be found for positive step lengths Not true in real life as the starting point could be anywhere in weight space Line search is now bi-directional i.e. can search backwards up the direction vector Convergence much more stable and the problem of jumping error function is gone