Introduction To Artificial Neural System

Artificial Neuron Y =f( Σw i X i ) Σ X 1 X 2 X 3 X n

Basic elements are- Input nodes Weights Activation function Total signal reaching at output are given by-> Output(y)= f Output may be more than one it depends on number of neuron i.e. multiple neuron will cause multiple output.

McCulloch Pits Model McCulloch Pits Model is shown in figure The inputs x i, for i=1,2,3……….n are 1 or 0 depending on the absence or presence of the input impulse at instant k. The neuron output signal is denoted by o and the firing rule for this model is defined as-

Where subscripts k=0,1,2………….denote the discrete time instant. W i -> is the multiplicative weight connecting the i th Input with neuron membrane. W i =+1 for excitatory synapse for this model W i =-1 for inhabitatory synapse for this model

T-> neuron threshold value which need to be exceeded by the weighted sum of signals for It can perform the basic logic operations NOT, OR and AND provided its weights and threshold are appropriately selected. Three input NOR implementation-

Three input NAND implementation:

Memory Cell:

Neuron modeling for artificial neural network: McCullach Pitts model allows a binary 0, 1 states only operates under a discrete time assumption, and assumes synchronization of operation of all neuron in a larger network. Weight and neurons threshold are fifed in the model and no interaction among the network neurons takes place except for signal flow. Every neuron model consist of a processing element with synaptic input connections and a single output

Neuron output signal is given by: Where w is the weight vector x is the input vector Note: All vector in this text are column vectors subscript t denotes a transposition.

The function f(w t x) is often referred to as activation function. Its domain is the set of activation values i.e. net of the neuron model. Therefore we use f(net). And net= w t x is the scalar product of weight and input vector or we can say that the argument of the activation function, the variable net is a analog of the biological neurons membrane potential. We also assume that the modeled neuron has n- 1 actual synaptic connection that come from actual variable inputs x 1, x 2………………………… x n-1

Also x n = -1 and w n =T. Threshold play an important role for some models so we use threshold (T) as a separate neuron model parameter with in the same class of networks the neuron are sometimes considered to perform differently during the different phases of network operations. Therefore its better to replace whenever needed the general neuron model with a specific f(net) and specific neuron model. For other artificial neural networks we say- (1) neural networks are meant to be artificial neural networks consisting of neuron models. and (2) neurons are meant to be artificial neuron model like McCulloch pitts.

Neuron as a processing node perform the operation of summation of its weighted inputs. It perform the nonlinear operation f(net) through its activation function Activation functions

Are known as bipolar continuous and bipolar binary function respectively. The word bipolar is used to point out that both positive and negative response of neurons are produced for the above definition of the activation function By shifting and scaling the bipolar activation function defined by the above equations A &B

Unipolar continuous and unipolar binary activation functions can be obtained respectively as A & C are called sigmoidal characteristics, B&D are called hard limiting activation function. Hard limiting functions describe the discrete neuron model.

Unipolar continuous:

Models for Artificial Neural Network : Over basic definition of ANN as physical cellular network that are able to acquire, store and utilize experimental knowledge has been related to the network capabilities and performance. The neural network can also be defined as an interconnection of neurons, such that neuron outputs are connected, through weights to all other neurons including themselves.

Feed Forward Network: Let us consider an elementary feed forward architecture of m-neurons receiving n-inputs as shown in figure. Its inputs and output vector are respectively- Weight w ij connects the i th neuron with j th input. Note: the double subscript connection used for weights is such that the first and second subscript denote the index of destination and source nodes respectively.

We thus can write the activation value for the i th neuron as- The activation function f(net i ) for i=1,2,……………,m complete the processing of x.

The transformation performed by each of the m- neurons in the network, is a strongly nonlinear mapping expressed as now introducing the nonlinear diagonal matrix to output space o is implemented by the network can be expressed as follows-

Where W is the weight matrix also known as connection matrix: f(.)-> Neuron’s Activation operator

Note:- nonlinear activation functions f(.) on the diagonal of the matrix operator Γ operate component wise on the activation value net of each neuron. Each activation value form a scalar product of an input with respect to the weight vector. Input and output vectors x and o are often known input and output pattern respectively. The mapping of input pattern into an output pattern is instantaneous type since it involves the time delay between x and 0 this can be re-written as

Feedback Network A feedback network can be obtained from the feed forward network by connecting the neurons outputs to their inputs. The result is shown in figure:

Single Layer Discrete time Feedback Network: The essence of closing the feedback loop is to enable useful of output o i through outputs o j, for j=1,2………m. Such control is specially meaningful if the parent output, say o(t) controls the output at the instant o(t+Δ). The time Δ elapsed between t and t+ Δ is introduced by the delay elements in the feedback loop. As we know for feed forward networks-

Mapping of input pattern to output pattern. Now the mapping of o(t) into o(t+ Δ )can now be written as- this formula represented by block diagram shown in figure. the input x(t) is only needed to initialize the instantaneous (feedback) network so that 0(o)=X(o) the input is then removed and the system remains autonomous for t>0. No we consider a special case of feedback configuration such that x(t)=x(0) and no input is provided to the network thereafter or for t>0.

If we consider time as a discrete variable and decide to observe. The network performance at discrete time instants Δ, 2 Δ,3 Δ……….. The system is called discrete time. For a discrete time artificial neural system, we have

Neural Processing: The processing of computation of 0 for x performed by the network is called recall. Recall is the proper processing phase for a neural network, and its objective is to retrieve the information Recall corresponds to the decoding of the stored content which may have been encoded in a network previously. A set of pattern can be stored in the network. If the network is presented with a pattern similar to a member of a stored set, it may associate the input with the closest stored pattern, the process is called the auto associate.

Typically a degraded input pattern serves as a due for retrieval o its original form. Association of input pattern can also be stored is a hetroassociation variant. The hetroassociation processing, the association between pairs of pattern are stored.

Classification is the another form of processing. Let us assume that a set of input pattern is divided into a number of classes or categories. In response to an input pattern from set, the classifier recall the information regarding class membership of the input pattern. Classes are expressed by discrete valued output vectors, and thus output neurons of classifier employ binary activation function

Learning and Adaptation: Learning is a fundamental subject for psychologist, but it also underlies many of the areas. learning is a relatively permanent character in behavior brought about by experience learning in human being and animals is a inferred process, we can see it directly and we can assume that it has occurred by observing hopes in performance. learning in neural networks is a more different process. Also the designing of an associator or a classifier can be based on learning a relationship that transforms inputs into outputs.

Learning as Approximation: Approximation theory focuses on approximating in a continuous, multivariable function h(x) by another function H(w,x), where Are the inputs and weight vector respectively. Now the learning task is to find w that provide the best possible approximation of h(x) based on the training examples {x}. The choice of the function H(w,x) in order to represent h(x) is called a representation problem.

Once H(w,x) has been chosen, the network learning algorithm is applied for finding optimal parameters (w) or weight. More precise formation of the following problem can be stated as calculation involving w k : Where or distance function is a measure of approximation quality between H(w,x) and h(x).

Supervised and Unsupervised Learning: Learning is necessary when the information about inputs/outputs is unknown or incomplete a priori, so that no design of a network can be performed n advance. The majority of neural networks requires training in a supervised or unsupervised learning mode. Some of the networks, however can be designed without incremental training. They are designed by batch learning rather than stepwise training. Note: Batch training takes palace when the network weight are adjusted in a single training step. In this mode of learning the complete set of inputs and outputs training data is needed to determine weights and feedback information produced by the network itself is not involved in developing the network. This learning techniques are called recording.

Neural Network Learning Rules: A neuron is considered to be an adaptive element. Its weight are modifiable depending on the input signal it receives, its output value and the associated teacher response. In some cases the teacher signal is not available and no error information can be used, thus the neuron will modify its weights based only on the input/output as per unsupervised learning. A general rule is adapted in neural in neural network studies: the weight vector i.e.

Increase in proportion to the of input x and learning signal r. where the input learning signal r is in general a function of w i,x and summations of the teacher’s signal d i. the increment of the weight vector w i produced by the learning step at time t according to the general rule is- Where C is a positive number called the learning constant that determines rate of learner.

The weight vector adapted at time t becomes at the next instant or learning step. For the k th step we thus have now continuous time learning can be expressed as

Hebbain Learning Rule: for the Hebbain Learning rule the learning signal is equal simply to neuron’ output (Hebb ) we have The increment Δw i of the weight vector becomes Single weight is adapted using the following increment This can be written briefly as

[Note: This learning rule requires the weight initialization at small random values around w i =0 to learning ] The Hebbian Learning Rule represents a purely feed forward unsupervised learning. The rules states that if the crossproduct of output and input or oscillation term o i x j is positive this result in an increase of weight w ij otherwise the weight decreases

Example Hebbian learning with binary and continuous activation functions of a very simple network is to be trained using an initial weight vector and three inputs

Revisiting the Hebbian learning example with continuous bipolar activation function f(net), using input x 1 and initial weight w 1 we obtain neuron output values and updated weights for ʎ =1. The only difference compared with the previous case is that instead of f(net)=sgn(net). Now the neurons response is computed-

Preceptron Learning Rule: For the percetron learning rule the learning signal is the difference between the desired and the actual neuron’s response(ROSENBLATT-1958). Hence we can say that learning is supervised and the learning signals is equal to:-

Note 1: this rule is applicable only for binary neuron response and the relationship express the rule for the binary case. 2. under this rule weight is only adjusted if and only if o i is incorrect. 3. weight can be initialized at any value since the desired response is either +1 or -1, the weight adjustment reduce to where +ive sign is applicable when -ive sign is applicable when

Delta Learning Rule: The delta learning rule is only valid for the continuous activation functions Part of neuron modeling But this is here is supervised learning mode The learning signal for this rule is called delta and is defined as :

Where is the derivative of activation function f(net) for This learning rule can be readily derived from the condition of least-square error between o i and d i. calculating the gradient vector w.r.t. w i of square error defined as:-

Since the minimization of error requires the weight chaqes to be in the gradient direction, therefore Where ɳ is a +ve constant Now we get from the above equations Or for the single weight the adjustment will be Note: weight adjustment is computed based on minimization of the squared error

Now considering the general learning rule and plugging in the learning signed as defined in since C and ɳ have been assumed to be arbitrary constant. The weight are initialized at any value for this method of training.

Winner-Take-All Learning Rule: Winner-Take-All learning rule differs from the other results. It is used for unsupervised learning rather its used for learning statistical properties of inputs. Learning is based on the concept that mth neuron has the maximum response due to input x. that neuron is declared as winner. W m = [w m1, w m2,………….. w mn ] t In the following figure 1) neurons are arranged in a layer of permits. 2) adjusted weights are highlighted.

Outstar Learning Rule: Outstrar Learning rule is explained better when neurons are arranged in a layer. This rule is concerned with supervised learning. This allow the network to extract statistical properties of the input and output signal. The adjustment weight is computed as follows :