Training and Testing Neural Networks 서울대학교 산업공학과 생산정보시스템연구실 이상진

Contents Introduction When Is the Neural Network Trained? Controlling the Training Process with Learning Parameters Iterative Development Process Avoiding Over-training Automating the Process

Introduction (1) Training a neural network –perform a specific processing function 1) 어떤 parameter? 2) how used to control the training process 3) management of the training data - training process 에 미치는 영향 ? –Development Process 1) Data preparation 2) neural network model & architecture 선택 3) train the neural network –neural network 의 구조와 그 function 에 의해 결정 –Application –“trained”

Introduction (2) Learning Parameters for Neural Network Disciplined approach to iterative neural network development

Introduction (3)

When Is the Neural Network Trained? When the network is trained? –the type of neural network –the function performing classification clustering data build a model or time-series forecast –the acceptance criteria meets the specified accuracy –the connection weights are “locked” –cannot be adjusted

When Is the Neural Network Trained? Classification (1) Measure of success : percentage of correct classification –incorrect classification –no classification : unknown, undecided threshold limit

When Is the Neural Network Trained? Classification (2) confusion matrix : possible output categories and the corresponding percentage of correct and incorrect classifications

When Is the Neural Network Trained? Clustering (1) Output a of clustering network –open to analysis by the user Training regimen is determined: –the number of times the data is presented to the neural network –how fast the learning rate and the neighborhood decay Adaptive resonance network training (ART) –vigilance training parameter –learn rate

When Is the Neural Network Trained? Clustering (2) Lock the ART network weights –disadvantage : online learning ART network are sensitive to the order of the training data

When Is the Neural Network Trained? Modeling (1) Modeling or regression problems Usual Error measure –RMS(Root Square Error) Measure of Prediction accuracy –average –MSE(Mean Square Error) –RMS(Root Square Error) The Expected behavior – 초기의 RMS error 는 매우 높으나, 점차 stable minimum 으로 안정화된다

When Is the Neural Network Trained? Modeling (2)

When Is the Neural Network Trained? Modeling (3) 안정화되지 않는 경우 –network fall into a local minima the prediction error doesn’t fall oscillating up and down – 해결 방법 reset(randomize) weight and start again training parameter data representation model architecture

When Is the Neural Network Trained? Forecasting (1) Forecasting –prediction problem –RMS(Root Square Error) –visualize : time plot of the actual and desired network output Time-series forecasting –long-term trend influenced by cyclical factor etc. –random component variability and uncertainty –neural network are excellent tools for modeling complex time-series problems recurrent neural network : nonlinear dynamic systems –no self-feedback loop & no hidden neurons

When Is the Neural Network Trained? Forecasting (2)

Controlling the Training Process with Learning Parameters (1) Learning Parameters depends on –Type of learning algorithm –Type of neural network

Controlling the Training Process with Learning Parameters (2) - Supervised training Neural Network Pattern Prediction Desired Output Desired Output 1) How the error is computed 2) How big a step we take when adjusting the connection weights

Controlling the Training Process with Learning Parameters (3) - Supervised training Learning rate –magnitude of the change when adjusting the connection weights –the current training pattern and desired output large rate –giant oscillations small rate –to learn the major features of the problem generalize to patterns

Controlling the Training Process with Learning Parameters (4) - Supervised training Momentum –filter out high-frequency changes in the weight values –oscillating around a set values 방지 –Error 가 오랫동안 영향을 미친다 Error tolerance –how close is close enough – 많은 경우 0.1 – 필요성 net input must be quite large?

Controlling the Training Process with Learning Parameters (5) -Unsupervised learning Parameter –selection for the number of outputs granularity of the segmentation (clustering, segmentation) –learning parameters (architecture is set) neighborhood parameter : Kohonen maps vigilance parameter : ART

Controlling the Training Process with Learning Parameters (6) -Unsupervised learning Neighborhood –the area around the winning unit, where the non-wining units will also be modified –roughly half the size of maximum dimension of the output layer –2 methods for controlling square neighborhood function, linear decrease in the learning rate Gaussian shaped neighborhood, exponential decay of the learning rate –the number of epochs parameter –important in keeping the locality of the topographic amps

Controlling the Training Process with Learning Parameters (7) -Unsupervised learning Vigilance –control how picky the neural network is going to be when clustering data –discriminating when evaluating the differences between two patterns –close-enough –Too-high Vigilance use up all of the output units

Iterative Development Process (1) Network convergence issues –fall quickly and then stays flat / reach the global minima –oscillates up and down / trapped in a local minima – 문제의 해결 방법 some random noise reset the network weights and start all again design decision

Iterative Development Process (2)

Iterative Development Process (3) Model selection –inappropriate neural network model for the function to perform –add hidden units or another layer of hidden units –strong temporal or time element embedded recurrent back propagation radial basis function network Data representation –key parameter is not scaled or coded –key parameter is missing from the training data –experience

Iterative Development Process (4) Model architecture –not converge : too complex for the architecture –some additional hidden units, good –adding many more? Just, Memorize the training patterns –Keeping the hidden layers as this as possible, get the best results

Avoiding Over-training Over-training – 같은 pattern 을 계속적으로 학습 –cannot generalize – 새로운 pattern 에 대한 처리 – switch between training and testing data

Automating the Process Automate the selection of the appropriate number of hidden layers and hidden units –pruning out nodes and connections –genetic algorithms –opposite approach to pruning –the use of intelligent agents