Slides from: Doug Gray, David Poole Neural Networks Slides from: Doug Gray, David Poole
What is a Neural Network? Information processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information A method of computing, based on the interaction of multiple connected processing elements
What can a Neural Net do? Compute a known function Approximate an unknown function Pattern Recognition Signal Processing Learn to do any of the above
Basic Concepts A Neural Network generally maps a set of inputs to a set of outputs Number of inputs/outputs is variable The Network itself is composed of an arbitrary number of nodes with an arbitrary topology
Basic Concepts Definition of a node: A node is an element which performs the function y = fH(∑(wixi) + Wb) Connection Node
Properties Inputs are flexible any real values Highly correlated or independent Target function may be discrete-valued, real-valued, or vectors of discrete or real values Outputs are real numbers between 0 and 1 Resistant to errors in the training data Long training time Fast evaluation The function produced can be difficult for humans to interpret
Perceptrons Basic unit in a neural network Linear separator Parts N inputs, x1 ... xn Weights for each input, w1 ... wn A bias input x0 (constant) and associated weight w0 Weighted sum of inputs, y = w0x0 + w1x1 + ... + wnxn A threshold function (activation function), i.e 1 if y > 0, -1 if y <= 0
Σ Diagram w1 x1 x0 w0 x2 w2 Threshold . 1 if y >0 -1 otherwise y = Σ wixi xn wn
Typical Activation Functions F(x) = 1 / (1 + e –x) Using a nonlinear function which approximates a linear threshold allows a network to approximate nonlinear functions
Simple Perceptron Binary logic application fH(x) = u(x) [linear threshold] Wi = random(-1,1) Y = u(W0X0 + W1X1 + Wb) Now how do we train it?
Basic Training Perception learning rule ΔWi = η * (D – Y) * Xi η = Learning Rate D = Desired Output Adjust weights based on how well the current weights match an objective
Logic Training Expose the network to the logical OR operation Update the weights after each epoch As the output approaches the desired output for all cases, ΔWi will approach 0 X0 X1 D 1
Results W0 W1 Wb
Details Network converges on a hyper-plane decision surface X1 = (W0/W1)X0 + (Wb/W1) X1 X0
Feed-forward neural networks Feed-forward neural networks are the most common models. These are directed acyclic graphs:
Neural Network for the news example
Axiomatizing the Network The values of the attributes are real numbers. Thirteen parameters w0; … ;w12 are real numbers. The attributes h1 and h2 correspond to the values of hidden units. There are 13 real numbers to be learned. The hypothesis space is thus a 13-dimensional real space. Each point in this 13-dimensional space corresponds to a particular logic program that predicts a value for reads given known, new, short, and home
Prediction Error
Neural Network Learning Aim of neural network learning: given a set of examples, find parameter settings that minimize the error. Back-propagation learning is gradient descent search through the parameter space to minimize the sum-of-squares error.
Backpropagation Learning Inputs: A network, including all units and their connections Stopping Criteria Learning Rate (constant of proportionality of gradient descent search) Initial values for the parameters A set of classified training data Output: Updated values for the parameters
Backpropagation Learning Algorithm Repeat evaluate the network on each example given the current parameter settings determine the derivative of the error for each parameter change each parameter in proportion to its derivative until the stopping criteria is met
Gradient Descent for Neural Net Learning
Bias in neural networks and decision trees It’s easy for a neural network to represent “at least two of I1, …, Ik are true”: w0 w1 wk -15 10 10 This concept forms a large decision tree. Consider representing a conditional: “If c then a else b”: Simple in a decision tree. Needs a complicated neural network to represent (c ^ a) V (~c ^ b).
Neural Networks and Logic Meaning is attached to the input and output units. There is no a priori meaning associated with the hidden units. What the hidden units actually represent is something that’s learned.