Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 WK1 - Introduction CS 476: Networks of.

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 WK1 - Introduction CS 476: Networks of Neural Computation WK2 – Perceptron Dr. Stathis Kasderidis Dept. of Computer Science University of Crete Spring Semester, 2009

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Contents Elements of Optimisation Theory Definitions Properties of Quadratic Functions Model Algorithm for Smooth Functions Classical Optimisation Methods 1 st Derivative Methods 2 nd Derivative Methods Methods for Quadratic functions Other Methods

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Contents Contents II Perceptron Model Convergence Theorem of Perceptron Conclusions

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Definitions Types of Optimisation Problems: Unconstrained Linear Constraints Non-linear Constraints General nonlinear constrained optimisation problem definition: NCP: minimise F(x) x   m subject to c i (x)=0 i=1..m’ c i (x)  0 i=m’..m

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Definitions II Strong local minimum: A point x* is a SLM of NCP if there exists  >0 such that: A1: F(x) is defined in N(x*,  ); and A2: F(x*)<F(y) for all y  N(x*,  ), y  x* Weak local minimum: A point x* is a WLM of NCP if there exists  >0 such that: B1: F(x) is defined in N(x*,  ); B2: F(x*)  F(y) for all y  N(x*,  ); and B3: x* is not a strong local minimum

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Definitions III UCP: minimise F(x) x   m Necessary conditions for a minimum of UCP: C1: ||g(x*)|| =0, i.e. x* is a stationary point; C2: G(x*) is positive semi-definite. Sufficient conditions for a minimum of UCP: D1: ||g(x*)|| =0; D2: G(x*) is positive definite.

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Definitions IV Assume that we expand F in its Taylor series: Where x,p   m, 0   1,  is a scalar and positive Any vector p which satisfies: is called descent direction at x*

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Properties of Quadratic Functions Assume that a quadratic function is given by: For some constant vector c and a constant symmetric matrix G (the Hessian matrix of  ). The definition of  implies the following relation between  (x+  p) and  (x) for any vector x and scalar  :

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Properties of Quadratic Functions I The function  has a stationary point when:   (x*) = Gx*+c = 0 Consequently a stationary point must satisfy the system of the linear equations: Gx* = -c The system might have: No solutions (c is not a linear combination of columns of G) Many solutions (if G is singular) A unique solution (if G is non-singular)

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Properties of Quadratic Functions II If x* is a stationary point it follows that: Hence the behaviour of  in a neighbourhood of x* is determiend by matrix G. Let j and u j denote the j-th eigenvalue and eigenvector of G. By definition: G u j = j u j The symmetry of G implies that the set of {u j }, j=1..m are orthonormal. So, when p is equal to u j :

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Properties of Quadratic Functions III Thus the change in  when moving away from x* along the direction of u j depends on the sign of j : If j >0  strictly increases as |  | increases If j <0  is monotonically decreasing as |  | increases If j =0 the value of  remain constant when moving along any direction parallel to u j If G is positive definite, x* is the global minimum of 

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Properties of Quadratic Functions IV If G is positive definite, x* is the global minimum of  If G is positive semi-definite a stationary point (if exists) is a weak local minimum. If G is indefinite and non-singular, x* is a saddle point and  is unbounded above and below

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Model Algorithm for Smooth Functions Algorithm U(Model algorithm for m-dimensional unconstrained minimisation) Let x k be the current estimate of x* U1: [Test for convergence] If the conditions are satisfied, the algorithm terminates with x k as the solution; U2: [Compute a search direction] Compute a non- zero m-vector p k the direction of search

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Model Algorithm for Smooth Functions I U3: [Compute a step length] Compute a positive scalar  k, the step length, for which it holds that: F(x k +  k p k ) < F(x k ) U4: [Update the estimate of the minimum] Set: X k+1  x k +  k p k, k  k+1 and go back to step U1. To satisfy the descent condition (F k+1 <F k ) p k should be a descent direction: g k T p k < 0

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods: 1 st derivative 1 st Derivative Methods: A linear approximation to F about x k is: F(x k +p) = F(x k ) + g k T p Steepest Descent Method: Select p k as: p = - g k It can be shown that:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods I: 1 st derivative  is the spectral condition number of G The steepest descent method can be very slow if  is large! Other 1 st derivative methods include: Discrete Newton method: Approximates the Hessian, G, with finite differences of the gradient g Quasi-Newton methods: Approximate the curvature: s k T Gs k  (g(x k +s k ) – g(x k )) T s k

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods II: 2 nd derivative 2 nd Derivative Methods: A quadratic approximation to F about x k is: F(x k +p) = F(x k ) + g k T p+(1/2) p T G k p The function is minimised my finding the minimum of the quadratic function  : This has a solution that satisfies the linear system: G k p k = -g k

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods III: 2 nd derivative The vector p k in the previous equation is called the Newton direction and the method is called the Newton method Conjugate Gradient Descent: Another 2 nd derivative method

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods IV: Methods for Quadratics In many problems the function F(x) is a sum of squares: The i-component of the m-vector is the function f i (x), and ||f(x)|| is called the residual at x. Problems of this type appear in nonlinear parameter estimation. Assume that x   n is a parameter vector and t i is the set of independent variables. Then the least-squares problem is: minimise, x   n

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods V: Methods for Quadratics Where: f i (x)=  (x,t i )-y i and Y=F(t) is the “true” function y i are the desired responses In the Least Squares Problem the gradient, g, and the Hessian, G, have a special structure. Assume that the Jacobian matrix of f(x) is denoted by J(x) (a mxn matrix) and let the matrix G i (x) denote the Hessian of f i (x). Then:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods VI: Methods for Quadratics g(x)=J(x) T f(x) and G(x)=J(x) T J(x)+Q(x) where Q is: We observe that the Hessian is a special combination of first and second order information. Least-square methods are based on the premise that eventually the first order term J(x) T J(x) dominates the second order one, Q(x)

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods VII: Methods for Quadratics The Gauss-Newton Method: Let x k denote the current estimate of the solution; a quantity subscripted by k will denote that quantity evaluated at x k. From the Newton direction we get: (J k T J k +Q)p k = -J k T f k Let the vector p N denote the Newton direction. If ||f k ||  0 as x k  x* the matrix Q k  0. Thus the Newton direction can be approximated by the solution of the equations: J k T J k p k = -J k T f k

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods VIII: Methods for Quadratics The solution of the above problem is given by the solution to the linear least square problem: minimise, p   n The solution is unique if J k has full rank. The vector p GN which solves the linear problem is called the Gauss-Newton direction. This vector approximates the Netwon direction p N as ||Q k ||  0

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods VIII: Methods for Quadratics The Levenberg-Marquardt Method: In this method the search direction is defined as the solution to the equations: (J k T J k + k I)p k = -J k T f k Where k is a non-negative integer. A unit step is taken along p k, i.e. x k+1  x k + p k It can be shown that for some scalar , related to k, the vector p k is the solution to the constrained subproblem:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Classical Optimisation Methods VIIII: Methods for Quadratics minimise, p   n subject to ||p|| 2   If k =0 p k is the Gauss Newton direction; If k  , || p k ||  0 and p k becomes parallel to the steepest descent direction

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Other Methods Other methods are based on function values only. This category includes methods such as: Genetic Algorithms Simulated Annealing Tabu Search Guided Local Search etc

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Optimisation Additional References Practical Optimisation, P. Gill, W. Murray, M. Wright, Academic Press, 1981. Numerical Recipes in C/C++, Press et al, Cambridege University Press, 1988

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron The model was created by Rosenblatt It uses the nonlinear neuron of McCulloch-Pitts (which uses the Sgn as transfer function)

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Perceptron Output The output y is calculated by: Where sgn(  ) is defined as: The perceptron classifies an input vector x   m to one of two classes C 1 or C 2

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Decision Boundary The case present above is called linearly separable classes

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Learning Rule Assume that vectors are drawn from two classes C 1 and C 2, i.e. x 1 (1), x 1 (2), x 1 (3),…. belong to C 1 ; and x 2 (1), x 2 (2), x 2 (3),…. belong to C 2 Assume also that we redefine vectors x(n) and w(n) such as to include the bias, i.e. x(n)=[+1,x 1 (n),…,x m (n)] T ; and w(n)=[b(n),w 1 (n),…,w m (n)] T x, w   m+1

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Learning Rule II Then there should exist a weight vector w such that: w T x > 0, when x belongs to class C 1 ; and w T x  0, when x belongs to class C 2 ; We have selected arbitrary to include the case of w T x=0 to class C 2 The algorithm for adapting the weights of the perceptron may be formulated as follows:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Learning Rule III 1.If the nth member of the training set, x(n) is correctly classified by the current weight set, w(n) no correction is needed, i.e. w(n+1)=w(n), if w T x(n) > 0 and x(n) belongs to class C 1 ; w(n+1)=w(n), if w T x(n)  0 and x(n) belongs to class C 2 ;

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Learning Rule IV 2.Otherwise, the weight vector is updated according to the rule: w(n+1)=w(n)-  (n)x(n), if w T x(n) > 0 and x(n) belongs to class C 2 ; w(n+1)=w(n)+  (n)x(n), if w T x(n)  0 and x(n) belongs to class C 1 ; The parameter  (n) is called learning rate and controls the adjustment to the weight vector at iteration n.

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Learning Rule V If we assume that the desired response is given by: Then we can re-write the adaptation rule in the form of an error-correction learning rule: w(n+1)=w(n)+  [d(n)-y(n)]x(n) Where the e(n)=  [d(n)-y(n)] is the error signal The learning rate is a positive constant in the range 0<   1.

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Learning Rule VI When we assign a value to  in the range (0,1] we must keep in mind two conflicting requirements: Averaging of past inputs to provide stable weights estimates, which requires a small  Fast adaptation with respect to real changes in the underlying distributions of the process responsible for the generation of the input vector x, which requires a large 

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Summary of the Perceptron Algorithm Variables and Parameters: x(n)=(m+1)-by-1 input vector = [+1, x 1 (n),…, x m (n)] T w(n)=(m+1)-by-1 weight vector = [b(n), w 1 (n),…, w m (n)] T b(n)=bias y(n)=actual response d(n)=desired response  =learning rate in (0,1]

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Summary of the Perceptron Algorithm I 1.Initialisation: Set w(0)=0. Then perform the following computations for time steps n=1,2,…. 2.Activation: At time step n, activate the perceptron by applying the input vector x(n) and desired response d(n). 3.Computation of Actual Response: Compute the actual response of the perceptron by using: y(n)=Sgn(w T (n)x(n) ) where Sgn() is the signum function.

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Perceptron Summary of the Perceptron Algorithm II 4.Adaptation of Weight Vector: Update the weight vector of the perceptron by using: w(n+1)=w(n)+  [d(n)-y(n)]x(n) where: 5.Continuation: Increment the time step n by one and go back to step 2.

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem We present a proof of the fact that the perceptron needs only a finite number of steps in order to converge (i.e. to find the correct weight vector if this exists) We assume that w(0)=0. If this is not the case the proof still stands but the number of the number of steps that are needed for convergence is increased or decreased Assume that vectors are which are drawn from two classes C 1 and C 2, form two subsets, i.e. H 1 ={x 1 (1), x 1 (2), x 1 (3),….} belong to C 1 ; and H 2 ={ x 2 (1), x 2 (2), x 2 (3),….} belong to C 2

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem I Suppose that w(n) T x(n) < 0 for n=1,2,… and the input vector belongs to H 1 Thus for  (n)=1 we can write the (actually incorrect) weight update equation as: w(n+1)=w(n)+ x(n), for x(n) belonging to class C 1 ; Given the initial condition w(0)=0, we can solve iteratively the above equation and obtain the result: w(n+1)=x(1)+x(2)+…+x(n) (E.1)

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem II Since classes C 1 and C2 are linearly separable there exists w 0 such that w T x(n) > 0 for all vectors x(1), x(2), …, x(n) belonging to H 1. For a fixed solution w 0 we can define a positive number  as: Multiplying both of E.1 with w 0 T we get: w 0 T w(n+1)= w 0 T x(1)+ w 0 T x(2)+…+ w 0 T x(n) So we have finally: w 0 T w(n+1)  n  (E.2)

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem III We use the Cauchy-Schwarz inequality for two vectors which states that: ||w 0 || 2 || w(n+1)|| 2  [w 0 T w(n+1)] 2 Where |||| denotes the Euclidean norm of the vector and the inner product w 0 T w(n+1) is a scalar. Then from E.2 we get: ||w 0 || 2 || w(n+1)|| 2  [w 0 T w(n+1)] 2  n 2  2 or alternatively:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem IV Now using: w(k+1)=w(k)+ x(k), for x(k) belonging to class C 1 ; k=1,..,n And taking the Euclidean norm we get: ||w(k+1)|| 2 =||w(k)|| 2 + ||x(k)|| 2 +2 w(k) T x(k), Which under the assumption of wrong classification (i.e. w T x(n) < 0 ) leads to: ||w(k+1)|| 2  ||w(k)|| 2 + ||x(k)|| 2

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem V Or finally to: ||w(k+1)|| 2 - ||w(k)|| 2  ||x(k)|| 2 Adding all these inequalities for k=1,…,n and using the initial condition w(0)=0 we get: Where  is a positive number defined as:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem VI E.4 states that the Euclidean norm of vector w(n+1) grows at most linearly with the number of iterations n. But this result is in conflict with E.3 for large enough n. Thus we can state that n cannot be larger than some value n max for which both E.3 and E.5 are simultaneously satisfied with the equality sign. That is n max is the solution of the equation:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Convergence Perceptron Convergence Theorem VII Solving for n max we get: This proves the perceptron algorithm will terminate after a finite number of steps. However, observe that there exists no unique solution for n max due to non- uniqueness of w 0

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Conclusions There are many optimistion methods. With decreasing power we present the methods in the following list: 2 nd derivative methods: e.g. Netwon 1 st derivative methods: e.g. Quasi-Newton Function value based methods: e.g. Genetic algorithms The perceptron is a model which classifies an input vector to one of two exclusive classes C 1 and C 2 The perceptron uses an error-correction style rule for weight update

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 Conclusions Conclusions I The perceptron learning rule converges in finite number of steps

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 WK1 - Introduction CS 476: Networks of.

Similar presentations

Presentation on theme: "Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 WK1 - Introduction CS 476: Networks of."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 WK1 - Introduction CS 476: Networks of.

Similar presentations

Presentation on theme: "Contents Optimisation Perceptron Convergence Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009 WK1 - Introduction CS 476: Networks of."— Presentation transcript:

Similar presentations

About project

Feedback