1. Network Systems Lab. Korea Advanced Institute of Science and Technology No.1 Ch. 3 Iterative Method for Nonlinear problems EE692 Parallel and Distribution Computation | Prof. Song Chong

2. Network Systems Lab. Korea Advanced Institute of Science and Technology No.2 Nonlinear Problems  Nonlinear Problems to be covered  Systems of nonlinear equations  Optimization problems - constrained or unconstrained  Variational inequality - Viewed as a generalization of the aboves

3. Network Systems Lab. Korea Advanced Institute of Science and Technology No.3 Contraction Mappings  Several nonlinear iteration algorithms can be written as where T is a mapping from a set X of R n into itself and has the property : Here, is some vector norm, and is a constant belonging to [0,1)  Any vector satisfying is called a fixed point of T.  Show that contraction mappings are continuous.

4. Network Systems Lab. Korea Advanced Institute of Science and Technology No.4 Pseudo contraction  A mapping T: X → X has a fixed point and the property  Cleary, pseudocontraction condition is weaker than contraction condition  A pseudo contraction is not necessarily continuous x X 2 T(x) 2 1 1 x X 2 21 1 contraction pseudocontraction

5. Network Systems Lab. Korea Advanced Institute of Science and Technology No.5  A mapping T could be a contraction (or a pseudocontraction) for some choice of the vector norm, and fail to be a contraction ( or pseudocontraction) under a different choice of norm. → The proper choice of norm is crucial.  A nonnegative matrix M has iff it is a contraction mapping w.r.t some weighted maximum norm ( Cor. 6.1)  Proposition 1.1 Suppose that T : X → X is a contraction with modulus and that X is a closed subset of then, (a) (Existence and Uniqueness of fixed point) The mapping T has a unique fixed point (b) (Geometric Convergence) For every initial vector, the seq {x(t)} generated by x(t+1) = T{x(t)} converges to geometrically, In particular

6. Network Systems Lab. Korea Advanced Institute of Science and Technology No.6  Proof of prop 1.1 choose some By def. of contraction, therefore, for every and choose an arbitrary then, therefore, {x(t)} is a cauch seq, and hence must converge to a limit, say ( Prop. A.5) Furthermore, since X is closed,

7. Network Systems Lab. Korea Advanced Institute of Science and Technology No.7 For all t≥1, As Thus, T(x * ) = x *, i.e. is a fixed point of T. Suppose that y * (≠x * ) is another fixed point. Then, b.We have

8. Network Systems Lab. Korea Advanced Institute of Science and Technology No.8  Proposition 1.2  Suppose that and the mapping T: X -> X is a pseudo contraction with a fixed point and modulus Then, T has no other fixed points and the sequence {x(t)} generated by x(t+1) = T{x(t)} satisfies for every choice of the initial vector, In particular, {x(t)} converges to x *. Proof)  Uniqueness of the fixed point follows as in the proof of 1.1 By definition of pseudocontraction, By induction on t, we concluded. (*)

9. Network Systems Lab. Korea Advanced Institute of Science and Technology No.9 Contractions over Cartesian Product Sets  Block-maximum Norm  Assume that, where x i is a nonempty subset of and where n 1 +…+n m =n  Any vector is decomposed as x=(x 1,…,x m ) with  Given a norm || · || i on for each i, R n is endowed with the norm, termed block-maximum norm  Let T:X -> X be a contraction with modulus α under the above block- maximum norm (termed block contraction), and let T(x) = ( T 1 (x), …, T m (x) ) where T i :X -> X i is the i-th block-component of T Then, and Why?

10. Network Systems Lab. Korea Advanced Institute of Science and Technology No.10 Gauss-Seidel Methods  An iteration x(t+1) = T(x(t)) corresponds to updating all components of x simultaneously  A Gauss-Seidel mode of implementation of T -> block components of x are updated one at a time Define a mapping, corresponding to an update of the i- th block-component only, by Update all the block components of x, one at a time in increasing order is equivalent to applying the mapping S:X -> X defined by An equivalent definition of S is given by where S i :X -> X i is the i-th block-component of S

11. Network Systems Lab. Korea Advanced Institute of Science and Technology No.11 Gauss-Seidel Methods (Cont’d)  The mapping S is called Gauss-Seidel mapping based on T and the iteration x(t+1)=S(x(t)) is called Gauss-Seidel algorithm based on T.  Any fixed point of T is also a fixed point of S, and conversely.  Proposition 1.4 (Convergence of Gauss-Seidel Block-Contracting Iterations)  If T:X -> X is a block-contraction. Then the Gauss-Seidel mapping S is also a block-contraction with the same modulus as T.  In particular, if x is closed, the seq. of vectors generated by the Gauss-Seidel algorithm based on T converges to the unique fixed point of T geometrically.

12. Network Systems Lab. Korea Advanced Institute of Science and Technology No.12 Prop 1.4 & proof  Proof) For every and i = 1,…,m, An induction on i yields

13. Network Systems Lab. Korea Advanced Institute of Science and Technology No.13 Prop 1.4 & proof  Thus, S is a block-contraction with the same modulus as T  Therefore, by Prop1.1, the iteration converges to a unique fixed point of S in X geometrically.  Need to show that T and S have the same fixed point.  Suppose that is a unique fixed point of T

14. Network Systems Lab. Korea Advanced Institute of Science and Technology No.14 Prop 1.5  If a mapping has a fixed point and is a pseudo- contraction of modulus with respect to a block-maximum norm, then the same is true for the Gauss-Seidel mapping S, that is  In Particular, the sequence generated by converges to geometrically Pf.) simple

15. Network Systems Lab. Korea Advanced Institute of Science and Technology No.15 Component Solution Methods  Consider  Decompose this into m smaller systems  Algorithm that solves individual equations, the i-th equation for, while keeping the other components fixed.  Let be the set of all solutions of the i-th equation, defined by

16. Network Systems Lab. Korea Advanced Institute of Science and Technology No.16 Component Solution Methods * x )),(( 21 xxQ ))(( 2,1 xQx ),( 21 xxx  (),(()( 21 xQxQxQ   ),(| 2111 xxTxx   ),(| 2122 xxTxx  1 x 2 x

17. Network Systems Lab. Korea Advanced Institute of Science and Technology No.17 Prop 1.6 & proof  Suppose that X is closed and is a block contraction. Then, the set has exactly one element ( i.e., singleton) for each i and for each  Proof) For some i and some, and consider the mapping defined by Then is equal to the set of fixed points of. Let and. Then by block –contraction assumption on T Therefore, is a contraction w.r.t., which implies that it has a unique point in, Therefore, is a singleton. Q.E.D.

18. Network Systems Lab. Korea Advanced Institute of Science and Technology No.18 Component solution methods  Define a mapping by letting be equal to the unique element of. Then, the component solution method is  Gauss-Seidel algorithm based on the mapping Q is referred to as “Gauss-Seidel Component Solution Method” where the block- components of x are updated one at a time.

19. Network Systems Lab. Korea Advanced Institute of Science and Technology No.19 Gauss-Seidel Component Solution Method  Proposition 1.7 If T: X -> X is a block contraction, then Q is also a block-contraction with the same modulus as T. In particular, if X is closed, then the component solution method as well as Gauss-Seidel algorithm based on Q, converges to the unique fixed point of T geometrically.  ),(| 2122 xxTxx   ),(| 2111 xxTxx 

20. Network Systems Lab. Korea Advanced Institute of Science and Technology No.20 Gauss-Seidel Component Solution Method Pf. ) Let By definition of and using the block-contraction assumption on T, Thus, Q is a block-contraction with the same modulus as T. Since x is closed, Q has a unique fixed point, i.e. is equivalent to By definition of Thus, is the unique fixed point of T. Q.E.D

21. Network Systems Lab. Korea Advanced Institute of Science and Technology No.21 Gauss-Seidel Component Solution Method  Proposition 1.8 Suppose that T:X->X is continuous and a pseudocontraction w.r.t. a block- maximum norm. If each set X i is closed and convex, then the set R i (x) is nonempty for each i and for each x in X.  Proposition 1.9 Suppose that T: X -> X has a fixed point x * and is a pseudocontraction w.r.t. a block-maximum norm. Suppose that for every i and x in X, the set R i (x) is nonempty. Then, Q is also a pseudocontraction w.r.t. the same norm, and x * is its unique fixed point. In particular, the component solution method as well as the Gauss-Seidel algorithm based on Q, converges to x * geometrically.

22. Network Systems Lab. Korea Advanced Institute of Science and Technology No.22 Some useful Contraction Mappings  Consider a mapping T: X  R n whose i-th block component T i is of the form where is a fct. from into, is some scalar and G i is an invertible symmetric matrix of dimensions  Simple case

23. Network Systems Lab. Korea Advanced Institute of Science and Technology No.23 Some useful Contraction Mappings

24. Network Systems Lab. Korea Advanced Institute of Science and Technology No.24 Some useful Contraction Mappings  Proposition 1.10

25. Network Systems Lab. Korea Advanced Institute of Science and Technology No.25 Some useful Contraction Mappings

26. Network Systems Lab. Korea Advanced Institute of Science and Technology No.26 Some useful Contraction Mappings  Proposition 1.11