GATE Reactive Behavior Modeling Neural Networks (GATE-561) Dr.Çağatay ÜNDEĞER Instructor Middle East Technical University, GameTechnologies Bilkent University, Computer Engineering & General Manager SimBT Inc. Game Technologies Program – Middle East Technical University – Fall 2009

GATE Outline Neural Networks

GATE Artificial Neural Networks (NNets) Inspired by human brain. Fundamental functional units of brain are: –Neurons / soma / nerve cells and –Axons / dendrites connecting nerve cells.

GATE Artificial Neural Networks (NNets) Signals can be thought as simple electrical impulses from axons to dendrites. Dendrites collect the signals. Soma performs a kind of summation. Axon fire and transmit the signal.

GATE Artificial Neural Networks (NNets) Neural networks use a similar approach. Consist of: –Input arcs collect the signals, –Neuron sums the signals and –Output arcs transmit the summed signal based on a transmit function. Neural networks can be learned.

GATE Neural Network Architecture i i i h h h h h h o o hh input layerhidden layersoutput layer input 1 input 2 input 3 output 1 output 2 Arcs (storing weights) Neurons

GATE Usage of NNets Environmental Scanning and Classification: –Interpret vision and auditory information Memory: –Can learn through experience Behavior Control: –Outputs can be used as actions to be performed. Response Mapping: –Any input output mapping and association.

GATE Usage of NNets The usage of NNets is a little bit fuzzy. No exact cases and rules. Assume it as a tool and use your creativity to get whatever you like. What NNets do is –To seperate a solution space into partitions

GATE A Neuron Architecture Y X1X1 X2X2 XiXi XnXn w1w1 w2w2 wiwi wnwn X i = input i of neuron w i = weight of input i Y = summing junction y = output of neuron y to input of other neurons y is based on a transfer or activation function B=1 b B = a constant bias node, which may represent past history

GATE Neuron Architecture Y X1X1 X2X2 XiXi XnXn w1w1 w2w2 wiwi wnwn y B=1 b Y a = B*b + Σ X i *w i i=1 n Summing is referred as the input activation Y a Input activation is fed into the activation function f a (x) and output y is acquired y = f a (x)Inputs and weights can be in range ( - ∞, +∞)

GATE Some Activation Functions 1 0 x y θ Step function f s (x) = 1, if x≥θ 0, if x<θ 1 0 x y Linear function f s (x) = x

GATE Common Activation Function 1 0 x y Exponential / Sigmoid function f s (x) = 1 / ( 1 + e -θx ) θ = 1 You can also scale and shift the output if required

GATE A Group Of Neurons A single neuron will not do a lot for us We need a group of them A single layer NNet A two layer NNet

GATE Simple Logic Functions Created by McCulloch and Pitts in AND, OR, XOR functions could be done with simple neural network structures.

GATE Simple Logic Function AND OR XOR

GATE Learning NNets Weights of NNets can be learned by various algorithms. The most common algorşthm is the Backpropagation.

GATE Backpropagation Can learn weights of a multi-layer network –With fixed set of units and interconnections Employs a gradient descent search –To minimize the squared errors between target output and network output It is sub-optimal and may fail to reach true optimal point.

GATE Backpropagation Can learn weights of a multi-layer network –With fixed set of units and interconnections Employs a gradient descent search –To minimize the squared errors between target output and network output

GATE Backpropagation Error surface of the network may have multiple local minima. Therefore backprogation guarantees only to converge a local-optimum and may fail to reach the true-optimum.

GATE Backpropagation input 1 input 2 input 3 output 1 output 2 Initialize the network weights with small random numbers (e.g ) Train the network with m training examples repeatedly for multiple iterations. Stop training when a convergence criteria is met.

GATE Backpropagation o1o1 o2o2 For one training example, assume that: Inputs are x 01, x 02 and x 03 and Expected outputs are t 1 and t 2 We propogate the inputs through the network and Compute the actual outputs o 1, o 2 and o 3 t1t1 t2t2 expectedactual x 01 x 02 x 03

GATE Backpropagation o1o1 o2o2 Errors will occur between actual and expected outputs. We propagate errors backward through the network. Compute error on each output unit k ( δ k ) using: δ k = o k (1 - o k ) (t k - o k ) t1t1 t2t2 expectedactual δ 21 δ 22 output errors e.g. δ 21 = o 21 (1 - o 21 ) (t 21 - o 21 ) x 01 x 02 x

GATE Backpropagation o1o1 o2o2 Compute error on each hidden unit h (δ h ) using: δ h = o h (1 - o h ) Σ w h-k δ k t1t1 t2t2 expectedactual δ 21 δ 22 output errors w w e.g. δ 11 = o 11 (1 - o 11 ) (w δ 21 + w δ 22 ) δ 11 δ 12 δ 13 δ 14 hidden unit errors x 01 x 02 x

GATE Backpropagation o1o1 o2o2 Update each weight w (w f-t ) using: w f-t = w f-t + η δ t x f-t t1t1 t2t2 expectedactual δ 21 δ 22 output errors w w δ 11 δ 12 δ 13 δ 14 hidden unit errors w w x 01 x 02 x 03 η = learning rate (e.g. 0.05) e.g. w = w η δ 11 x x01x

GATE Backpropagation The main loop of the algorithm should repeatedly iterate over the training examples May iterate thousands of times using the same training examples. Until the network performs acceptably well: –Test with the entire training set or –A better way: test with a sparate test set.

GATE An Action Selector Neural Network The inputs are state variables that equal to either 0 (no) or 1 (yes). Outputs are action variables that are between 0 and 1. The action with the greatest output is selected. Enemy Sound Dead Low Health Attack Retreat Wander Search Spawn

GATE A Number Recognizer Neural Network Problem : To recognize normalized bitmaps of handwritten digits from “0” to “9”. Data Set : Collected from 43 people –30 people to training set –13 people to test set –32x32 bitmaps are divided into 4x4 blocks and black pixels are counted –So we have 8x8 matrices and each member stores a number between 0 and 16

GATE A Number Recognizer Neural Network 8x8 pixels = 64 inputs No Missing Input Value Target Value : class number from 0 to 9 Training Set : 3823 Instances Test Set : 1797 Instances

GATE A Number Recognizer Neural Network x 011 = 1 x x 0641 = 1 x x 012 = 1 x n x 212 x 6412 x 013 = 1 x x 213 x n13 “0” “9” Max Bias Control Unit 64 Pixel values (0..16) of 8x8 image Result

GATE A Number Recognizer Neural Network For each optic character (0..9), an output unit is used The class of the output unit having the highest value is chosen as result. Sample target values for training: ‘0’ ‘1’ ‘2’

GATE A Number Recognizer Neural Network Random weights are given (– ) A fixed number of iterations are performed on Training Set –Meanwhile weights of iterations are saved A test phase is performed using saved weights on Test Set to avoid overfitting The iteration weights that maximizes the test set accuracy is chosen

GATE Test Platform General Information New Instance Real Value Result of NNet Output layer of NNet Loaded Instance Set Current instance

GATE First Attempt 4 hidden units 0.1 as learning rate 5000 iterations are performed (>12 hours) Best on Test Set = 86.9% on 450 th iteration –Meanwhile 92.2% on Training Set –82.5% accuracy on 5000 th iteration Best on Training Set = 93.0% on 1110 th iteration –Meanwhile 86.1% on Test Set

GATE Graph of First Attempt

GATE Second Attempt 4 hidden units to as learning rate (same) 5000 iterations to 2500 Best on Test Set = 94.0% on 100 th iteration –Meanwhile 97.7% on Training Set –88.9% accuracy on 2500 th iteration Best on Training Set = 98.0% on 140 th iteration –Meanwhile 93.6% on Test Set

GATE Graph of Second Attempt

GATE Third Attempt 8 hidden units to as learning rate (same) 2500 iterations to 200 Best on Test Set = 95.2% on 176 th iteration –Meanwhile 98.7% on Training Set –95.2% accuracy on 200 th iteration Best on Training Set = 98.7% on 176 th iteration –Meanwhile 95.2% on Test Set

GATE Graph of Third Attempt

GATE Fourth Attempt 16 hidden units (same) 0.1 as learning rate to iterations to 400 Best on Test Set = 95.0% on 122 th iteration –Meanwhile 98.4% on Training Set –94.4% accuracy on 400 th iteration Best on Training Set = 98.7% on 394 th iteration –Meanwhile 94.4% on Test Set

GATE Graph of Fourth Attempt

GATE Last Attempt 16 hidden units to as learning rate (same) 400 iterations to 1000 (4 hours and 38 minutes) Best on Test Set = 96.4% on 315 th iteration –Meanwhile 99.5% on Training Set –96.0% accuracy on 1000 th iteration Best on Training Set = 99.9% on 945 th iteration –Meanwhile 96.1% on Test Set

GATE Graph of Last Attempt

GATE Advantages Handle errors well. Graceful degradation. Can learn novel solutions.

GATE Disadvantages Can’t understand how the learned network works, therefore they are the second best way to do something. Need as many examples as possible. Learning takes too much time / processing. Sometimes the network may not converge.