1. Mathematical Prerequisites Finite Automata Markov Chain Block Matrix multiplication Solving System of linear equations Sparse Matrix 11

2. Deterministic Finite State Automata (DFA) …….. One-way, infinite tape, broken into cells One-way, read-only tape head. Finite control, i.e., –finite number of states, and –transition rules between them, i.e., –a program, containing the position of the read head, current symbol being scanned, and the current “state.” A string is placed on the tape, read head is positioned at the left end, and the DFA will read the string one symbol at a time until all symbols have been read. The DFA will then either accept or reject the string. 12 Finite Control 01100

3. The finite control can be described by a transition diagram or table: Example #1: 1 0 0 1 1 q 0 q 0 q 1 q 0 q 0 q 0 One state is final/accepting, all others are rejecting. The above DFA accepts those strings that contain an even number of 0’s, including the null string, over Sigma = {0,1} L = {all strings with zero or more 0’s} Note, the DFA must reject all other strings 13 q0q0 q1q1 0 0 1 1

4. Note: Machine is for accepting a language, language is the purpose! Many equivalent machines may accept the same language, but a machine cannot accept multiple languages! Id’s of the characters or states are irrelevant, you can call them by any names! Sigma = {0, 1} ≡ {a, b} States = {q0, q1} ≡ {u, v}, as long as they have identical (isomorphic) transition table M1M2 ….M-inf L 14

5. An equivalent machine to the previous example (DFA for even number of 0’s): 1 0 0 1 1 q 0 q 3 q 1 q 2 q 2 q 2 accept string One state is final/accepting, all others are rejecting. The above DFA accepts those strings that contain an even number of 0’s, including null string, over Sigma = {0,1} Can you draw a machine for a language by excluding the null string from the language? L = {all strings with 2 or more 0’s} 15 q0q0 q1q1 0 0 1 q2q2 1 1 0 q3q3 1 0

6. Example #2: a c c c baccepted q 0 q 0 q 1 q 2 q 2 q 2 a a crejected q 0 q 0 q 0 q 1 Accepts those strings that contain at least two c’s 16 q1q1 q0q0 q2q2 a b a b c c a/b/c

7. Formal Definition of a DFA A DFA is a five-tuple: M = (Q, Σ, δ, q 0, F) QA finite set of states ΣA finite input alphabet q 0 The initial/starting state, q 0 is in Q FA set of final/accepting states, which is a subset of Q δA transition function, which is a total function from Q x Σ to Q δ: (Q x Σ) – > Q δ is defined for any q in Q and s in Σ, and δ(q,s) = q’ is equal to some state q’ in Q, could be q’=q Intuitively, δ(q,s) is the state entered by M after reading symbol s while in state q. 17

8. Revisit example #2: Q = {q 0, q 1, q 2 } Σ = {a, b, c} Start state is q 0 F = {q 2 } δ:abc q 0 q 0 q 0 q 1 q 1 q 1 q 1 q 2 q 2 q 2 q 2 q 2 Since δ is a function, at each step M has exactly one option. It follows that for a given string, there is exactly one computation. 18 q1q1 q0q0 q2q2 a b a b c c a/b/c

9. Give a DFA M such that: L(M) = {x | x is a string of 0’s and 1’s and |x| >= 2} Prove this by induction 19 q1q1 q0q0 q2q2 0/1

10. Give a DFA M such that: L(M) = {x | x is a string of (zero or more) a’s, b’s and c’s such that x does not contain the substring aa} Logic: In Start state (q0): b’s and c’s: ignore – stay in same state q0 is also “accept” state First ‘a’ appears: get ready (q1) to reject But followed by a ‘b’ or ‘c’: go back to start state q0 When second ‘a’ appears after the “ready” state: go to reject state q2 Ignore everything after getting to the “reject” state q2 110 q2q2 q0q0 a a/b/c a q1q1 b/c

11. Give a DFA M such that: L(M) = {x | x is a string of a’s, b’s and c’s such that x contains the substring aba} Logic: acceptance is straight forward, progressing on each expected symbol However, rejection needs special care, in each state (for DFA, we will see this becomes easier in NFA, non-deterministic machine) 111 q2q2 q0q0 a a/b/c b q1q1 c b/ca q3q3 a

12. Give a DFA M such that: L(M) = {x | x is a string of a’s and b’s such that x contains both aa and bb} First do, for a language where ‘aa’ comes before ‘bb’ Then do its reverse; and then parallelize them. Remember, you may have multiple “final” states, but only one “start” state 112 q0q0 b q7q7 q5q5 q4q4 q6q6 b b b a q2q2 q1q1 q3q3 a a a b a/b b a a ab

13. Problem: Third symbol from last is 1 113 0/1 q1q1 q0q0 q3q3 1 q2q2 Is this a DFA? No, but it is a Non-deterministic Finite Automaton

14. Nondeterministic Finite State Automata (NFA) An NFA is a five-tuple: M = (Q, Σ, δ, q 0, F) QA finite set of states ΣA finite input alphabet q 0 The initial/starting state, q 0 is in Q FA set of final/accepting states, which is a subset of Q δA transition function, which is a total function from Q x Σ to 2 Q δ: (Q x Σ) – > 2 Q :2 Q is the power set of Q, the set of all subsets of Q δ(q,s):The set of all states p such that there is a transition labeled s from q to p δ(q,s) is a function from Q x S to 2 Q (but not only to Q) 114

15. Example #1: one or more 0’s followed by one or more 1’s Q = {q 0, q 1, q 2 } Σ = {0, 1} Start state is q 0 F = {q 2 } δ:01 q 0 q 1 q 2 115 {q 0, q 1 }{} {q 1, q 2 } {q2}{q2}{q2}{q2} q1q1 q0q0 q2q2 01 0 1 0/1

16. Example #2: pair of 0’s or pair of 1’s as substring Q = {q 0, q 1, q 2, q 3, q 4 } Σ = {0, 1} Start state is q 0 F = {q 2, q 4 } δ:01 q 0 q 1 q 2 q 3 q 4 116 {q 0, q 3 }{q 0, q 1 } {}{q2}{q2} {q2}{q2}{q2}{q2} {q4}{q4} {q4}{q4}{q4}{q4} q0q0 0/1 0 0 q3q3 q4q4 q1q1 q2q2 1 1

17. Let Σ = {a, b, c}. Give an NFA M that accepts: L = {x | x is in Σ * and x contains ab} Is L a subset of L(M)? Or, does M accepts all string in L? Is L(M) a subset of L? Or, does M rejects all strings not in L? Is an NFA necessary? Can you draw a DFA for this L? Designing NFAs is not as trivial as it seems: easy to create bug accepting string outside language 117 q1q1 q0q0 q2q2 a a/b/c b

18. Problem: Third symbol from last is 1 118 0/1 q1q1 q0q0 q3q3 1 q2q2 Now, can you convert this NFA to a DFA?

19. Discrete-Time Markov Chains Topics State-transition matrix Network diagrams Examples: gambler’s ruin, brand switching, IRS, craps Transient probabilities Steady-state probabilities 119

20. Many real-world systems contain uncertainty and evolve over time. Stochastic processes (and Markov chains) are probability models for such systems. Discrete – Time Markov Chains 120 Origins: Galton-Watson process  When and with what probability will a family name become extinct? A discrete-time stochastic process is a sequence of random variables X 0, X 1, X 2,... typically denoted by { X n }.

21. Components of Stochastic Processes 121 The state space of a stochastic process is the set of all values that the X n ’s can take. (we will be concerned with stochastic processes with a finite # of states ) Time: n = 0, 1, 2,... State: v -dimensional vector, s = ( s 1, s 2,..., s v ) In general, there are m states, s 1, s 2,..., s m or s 0, s 1,..., s m -1 Also, X n takes one of m values, so X n  s.

22. At time 0 I have X 0 = $2, and each day I make a $1 bet. I win with probability p and lose with probability 1– p. I’ll quit if I ever obtain $4 or if I lose all my money. State space is S = { 0, 1, 2, 3, 4 } Let X n = amount of money I have after the bet on day n. Gambler’s Ruin 122 If X n = 4, then X n +1 = X n +2 = = 4. If X n = 0, then X n +1 = X n +2 = = 0.

23. A stochastic process { X n } is called a Markov chain if Pr { X n +1 = j | X 0 = k 0,..., X n -1 = k n -1, X n = i } = Pr { X n +1 = j | X n = i }  transition probabilities for every i, j, k 0,..., k n -1 and for every n. Discrete time means n  N = { 0, 1, 2,... }. The future behavior of the system depends only on the current state i and not on any of the previous states. Markov Chain Definition 123

24. Pr{ X n +1 = j | X n = i } = Pr{ X 1 = j | X 0 = i } for all n (They don’t change over time) We will only consider stationary Markov chains. The one-step transition matrix for a Markov chain with states S = { 0, 1, 2 } is where p ij = Pr{ X 1 = j | X 0 = i } Stationary Transition Probabilities 124

25. If the state space S = { 0, 1,..., m –1} then we have  j p ij = 1  i and p ij  0  i, j (we must (each transition go somewhere) has probability  0) Gambler’s Ruin Example 01234 0 10000 11- p 0 p 00 201- p 0 p 0 3001- p 0 p 400001 Properties of Transition Matrix 125

26. Computer Repair Example 126 Two aging computers are used for word processing. When both are working in morning, there is a 30% chance that one will fail by the evening and a 10% chance that both will fail. If only one computer is working at the beginning of the day, there is a 20% chance that it will fail by the close of business. If neither is working in the morning, the office sends all work to a typing service. Computers that fail during the day are picked up the following morning, repaired, and then returned the next morning. The system is observed after the repaired computers have been returned and before any new failures occur.

27. States for Computer Repair Example 127 IndexStateState definitions 0s = (0)No computers have failed. The office starts the day with both computers functioning properly. 1s = (1)One computer has failed. The office starts the day with one working computer and the other in the shop until the next morning. 2s = (2)Both computers have failed. All work must be sent out for the day.

28. Events and Probabilities for Computer Repair Example 128 Index Current state EventsProb- ability Next state 0s 0 = (0)Neither computer fails.0.6s' = (0) One computer fails.0.3s' = (1) Both computers fail.0.1s' = (2) 1s 1 = (1)Remaining computer does not fail and the other is returned. 0.8s' = (0) Remaining computer fails and the other is returned. 0.2s' = (1) 2s 2 = (2)Both computers are returned. 1.0s' = (0)

29. State-Transition Matrix and Network 129 The events associated with a Markov chain can be described by the m  m matrix: P = (p ij ). For computer repair example, we have: State-Transition Network Node for each state Arc from node i to node j if p ij > 0. For computer repair example:

30. Procedure for Setting Up a DTMC 1.Specify the times when the system is to be observed. 2.Define the state vector s = (s 1, s 2,..., s v ) and list all the states. Number the states. 3.For each state s at time n identify all possible next states s ' that may occur when the system is observed at time n + 1. 4.Determine the state-transition matrix P = (p ij ). 5.Draw the state-transition diagram. 130

31. Repair Operation Takes Two Days 131 One repairman, two days to fix computer.  new state definition required: s = (s 1, s 2 ) s 1 = day of repair of the first machine s 2 = status of the second machine (working or needing repair) For s 1, assign0 if 1 st machine has not failed 1 if today is the first day of repair 2 if today is the second day of repair For s 2, assign0 if 2 nd machine has not failed 1 if it has failed

32. State Definitions for 2-Day Repair Times 132 IndexStateState definitions 0s 0 = (0, 0)No machines have failed. 1s 1 = (1, 0)One machine has failed and today is in the first day of repair. 2s 2 = (2, 0)One machine has failed and today is in the second day of repair. 3s 3 = (1, 1)Both machines have failed; today one is in the first day of repair and the other is waiting. 4s 4 = (2, 1)Both machines have failed; today one is in the second day of repair and the other is waiting.

33. State-Transition Matrix for 2-Day Repair Times 133 0 1 2 3 4 For example,p 14 = 0.2 is probability of going from state 1 to state 4 in one day,where s 1 = (1, 0) and s 4 = (2, 1)

34. Brand Switching Example 134 Brand(j)1(j)123Total (i)(i) 19073100 2520540250 33018102150 Total125230145500 Number of consumers switching from brand i in week 26 to brand j in week 27 This is called a contingency table.  Used to construct transition probabilities.

35. Empirical Transition Probabilities for Brand Switching, p ij 135 Brand(j)1(j)123 (i)(i) 1 2 3 Steady state  

36. Markov Analysis 136 State variable, X n = brand purchased in week n {X n } represents a discrete state and discrete time stochastic process, where S = {1, 2, 3} and N = {0, 1, 2,...}. If {X n } has Markovian property and P is stationary, then a Markov chain should be a reasonable representation of aggregate consumer brand switching behavior. Potential Studies - Predict market shares at specific future points in time. - Assess rates of change in market shares over time. - Predict market share equilibriums (if they exist). - Evaluate the process for introducing new products.

37. Transform a Process to a Markov Chain 137 Sometimes a non-Markovian stochastic process can be transformed into a Markov chain by expanding the state space. Example: Suppose that the chance of rain tomorrow depends on the weather conditions for the previous two days (yesterday and today). Specifically, Pr{ rain tomorrow  rain last 2 days (RR) }= 0.7 Pr{ rain tomorrow  rain today but not yesterday (NR) }= 0.5 Pr{ rain tomorrow  rain yesterday but not today (RN) }= 0.4 Pr{ rain tomorrow  no rain in last 2 days (NN) }= 0.2 Does the Markovian Property Hold ?

38. The Weather Prediction Problem 138 How to model this problem as a Markov Process ? The state space: 0 = (RR) 1 = (NR) 2 = (RN) 3 = (NN) 0 (RR)0.7 00.3 0 P =1 (NR)0.5 00.5 0 2 (RN) 00.4 0 0.6 3 (NN) 00.2 0 0.8 0(RR) 1(NR) 2(RN) 3(NN) The transition matrix: This is a discrete-time Markov process.

39. Multi-step ( n -step) Transitions 139 The P matrix is for one step: n to n + 1. How do we calculate the probabilities for transitions involving more than one step? Consider an IRS auditing example: Two states: s 0 = 0 (no audit), s 1 = 1 (audit) Transition matrix Interpretation:p 01 = 0.4, for example, is conditional probability of an audit next year given no audit this year.

40. Two-step Transition Probabilities 140 Let p ij be probability of going from i to j in two transitions. In matrix form, P (2) = P  P, so for IRS example we have (2) The resultant matrix indicates, for example, that the probability of no audit 2 years from now given that the current year there was no audit is p 00 = 0.56. (2)

41. n -Step Transition Probabilities 141 This idea generalizes to an arbitrary number of steps. For n = 3: P (3) = P (2) P = P 2 P = P 3 or more generally, P ( n ) = P ( m ) P ( n - m ) Interpretation: RHS is the probability of going from i to k in m steps & then going from k to j in the remaining n  m steps, summed over all possible intermediate states k. The ij th entry of this reduces to p ij ( n ) =  p ik ( m ) p kj ( n - m ) 1  m  n  1 m k =0 Chapman - Kolmogorov Equations

42. n -Step Transition Matrix for IRS Example 142 Time, nTransition matrix, P (n) 1 2 3 4 5

43. Gambler’s Ruin Revisited for p = 0.75 143 01234 0 10000 1 0.250 0.7500 20 0.250 0.750 300 0.250 0.75 400001 State-transition matrix State-transition network

44. Gambler’s Ruin with p = 0.75, n = 30 144 0 1 2 3 4 0 1 0 0 0 0 1 0.325  0  0.675 2 0.1 0  0 0.9 3 0.025  0  0.975 40 0 0 0 1 What does matrix mean?  is very small nonunique number) P (30) = A steady state probability does not exist.

45. DTMC Add-in for Gambler’s Ruin 145

46. 30-Step Transition Matrix for Gambler’s Ruin 146 Limiting probabilities

47. Conditional vs. Unconditional Probabilities 147 Let state space S = {1, 2,..., m }. Let p ij be conditional n -step transition probability  P ( n ). Let q ( n ) = ( q 1 ( n ),..., q m ( n ) ) be vector of all unconditional probabilities for all m states after n transitions. (n)(n) Perform the following calculations: q ( n ) = q (0) P ( n ) or q ( n ) = q ( n –1) P where q (0) is initial unconditional probability. The components of q ( n ) are called the transient probabilities.

48. Brand Switching Example   148 We approximate q i (0) by dividing total customers using brand i in week 27 by total sample size of 500: q(0) = (125/500, 230/500, 145/500) = (0.25, 0.46, 0.29) To predict market shares for, say, week 29 (that is, 2 weeks into the future), we simply apply equation with n = 2: q(2) = q(0)P (2) = (0.327, 0.406, 0.267) = expected market share from brands 1, 2, 3

49. Transition Probabilities for n Steps 149 Property 1:Let {X n : n = 0, 1,...} be a Markov chain with state space S and state- transition matrix P. Then for i and j  S, and n = 1, 2,... Pr{X n = j | X 0 = i} = p ij where the right-hand side represents the ij th element of the matrix P (n). (n)(n)

50. Steady-State Probabilities 150 Property 2:Let π = (π 1, π 2,..., π m ) is the m-dimensional row vector of steady-state (unconditional) probabilities for the state space S = {1,…,m}. To find steady-state probabilities, solve linear system: π = πP,  j=1,m π j = 1, π j  0, j = 1,…,m Brand switching example: π 1 + π 2 + π 2 = 1, π 1  0, π 2  0, π 3  0

51. Steady-State Equations for Brand Switching Example 151 π 1 = 0.90π 1 + 0.02π 2 + 0.20π 3 π 2 = 0.07π 1 + 0.82π 2 + 0.12π 3 π 3 = 0.03π 1 + 0.16π 2 + 0.68π 3 π 1 + π 2 + π 3 = 1 π 1  0, π 2  0, π 3  0  Discard 3 rd equation and solve the remaining system to get : π 1 = 0.474, π 2 = 0.321, π 3 = 0.205  Recall:q 1 (0) = 0.25, q 2 (0) = 0.46, q 3 (0) = 0.29 Total of 4 equations in 3 unknowns

52. Comments on Steady-State Results 152 1.Steady-state predictions are never achieved in actuality due to a combination of (i)errors in estimating P (ii)changes in P over time (iii)changes in the nature of dependence relationships among the states. 2.Nevertheless, the use of steady-state values is an important diagnostic tool for the decision maker. 3.Steady-state probabilities might not exist unless the Markov chain is ergodic.

53. Existence of Steady-State Probabilities 153 A Markov chain is ergodic if it is aperiodic and allows the attainment of any future state from any initial state after one or more transitions. If these conditions hold, then For example,State-transition network 1 3 2 Conclusion: chain is ergodic.   Craps

54. Game of Craps 154 The game of craps is played as follows. The player rolls a pair of dice and sums the numbers showing. Total of 7 or 11 on the first rolls wins for the player Total of 2, 3, 12 loses Any other number is called the point. The player rolls the dice again. If she rolls the point number, she wins If she rolls number 7, she loses Any other number requires another roll The game continues until he/she wins or loses

55. Game of Craps as a Markov Chain 155 All the possible states Start Win Lose P4P5P6P8P9P10 Continue

56. Game of Craps Network 156 Start Win Lose P4 P5 P6 P8 P9 P10 not (4,7)not (5,7)not (6,7)not (8,7)not (9,7)not (10,7) 4 5 6 8 9 10 4 5 6 8 9 7 7 7 7 7 7 (7, 11) (2, 3, 12)

57. Game of Craps 157 Probability of win = Pr{ 7 or 11 } = 0.167 + 0.056 = 0.223 Probability of loss = Pr{ 2, 3, 12 } = 0.028 + 0.056 + 0.028 = 0.112 Sum 23456789101112 Prob. 0.0280.0560.0830.1110.1390.1670.1390.1110.0830.0560.028

58. Transient Probabilities for Craps 158 This is not an ergodic Markov chain so where you start is important.ergodic

59. Absorbing State Probabilities for Craps 159

60. Interpretation of Steady-State Conditions 160 3.The probability that the process is in state j after a large number of steps is also equals the long-run proportion of time that the process will be in state j. 4.When the Markov chain is finite, irreducible and periodic, we still have the result that the  j, j  S, uniquely solve the steady-state equations, but now π j must be interpreted as the long-run proportion of time that the chain is in state j. 1.Just because an ergodic system has steady-state probabilities does not mean that the system “settles down” into any one state. 2.The limiting probability  j is simply the likelihood of finding the system in state j after a large number of steps.

61. What You Should Know About Markov Chains How to define states of a discrete time process? How to construct a state-transition matrix? How to find the n-step state-transition probabilities ? How to determine the unconditional probabilities after n steps? How to determine steady-state probabilities? 161

62. 162 Block Matrix Multiplication And Solving System of equations

63. Matrix multiplication, C = A x B 163

64. Assume throughout that the matrices are square (n x n matrices). The sequential code to compute A x B : for (i = 0; i < n; i++) for (j = 0; j < n; j++) { c[i][j] = 0; for (k = 0; k < n; k++) c[i][j] = c[i][j] + a[i][k] * b[k][j]; } Requires n 3 multiplications and n 3 additions T seq = (n 3 ) (Very easy to parallelize!) Sequential Code 164

65. One PE to compute each element of C - n 2 processors would be needed. Each PE holds one row of elements of A and one column of elements of B. Direct Implementation (P=n 2 ) P = n 2  T par = O(n) 165

66. P = n 3  T par = O(log n) n processors collaborate in computing each element of C - n 3 processors are needed. Each PE holds one element of A and one element of B. Performance Improvement (P=n 3 ) 166

67. P = nT par = O(n 2 ) Each instance of inner loop is independent and can be done by a separate processor. Cost optimal since O(n 3 ) = n * O(n 2 ) P = n 2 T par = O(n) One element of C (c ij ) is assigned to each processor. Cost optimal since O(n 3 ) = n 2 x O(n) P = n 3 T par = O(log n) n processors compute one element of C (c ij ) in parallel (O(log n)) Not cost optimal since O(n 3 ) < n 3 * O(log n) O(log n) lower bound for parallel matrix multiplication. Parallel Matrix Multiplication - Summary 167

68. Block Matrix Multiplication 168

69. 169

70. Cannon’s algorithm Systolic array All involve using processors arranged into a mesh (or torus) and shifting elements of the arrays through the mesh. Partial sums are accumulated at each processor. Mesh Implementations 170

71. Solve Ax = b Consists of two phases: –Forward elimination –Back substitution Forward Elimination reduces Ax = b to an upper triangular system Tx = b’ Back substitution can then solve Tx = b’ for x 71 Forward Elimination Back Substitution Gauss Elimination

72. Solve Ax = b Consists of two phases: – Forward elimination – Back substitution Forward Elimination reduces Ax = b to an upper triangular system Tx = b’ Back substitution can then solve Tx = b’ for x 172 Forward Elimination Back Substitution

73. Gaussian Elimination 173 Forward Elimination x 1 - x 2 + x 3 = 6 3x 1 + 4x 2 + 2x 3 = 9 2x 1 + x 2 + x 3 = 7 x 1 - x 2 + x 3 = 6 0 +7x 2 - x 3 = -9 0 + 3x 2 - x 3 = -5 x 1 - x 2 + x 3 = 6 0 7x 2 - x 3 = -9 0 0 -(4/7)x 3 =-(8/7) -(3/1) Solve using BACK SUBSTITUTION: x 3 = 2 x 2 =-1 x 1 =3 -(2/1) -(3/7) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

74. Forward Elimination 174

75. 4x04x0 +6x 1 +2x 2 – 2x 3 =8 2x02x0 +5x 2 – 2x 3 =4 –4x0–4x0 – 3x 1 – 5x 2 +4x 3 =1 8x08x0 +18x 1 – 2x 2 +3x 3 =40 -(2/4) MULTIPLIERSMULTIPLIERS -(-4/4) -(8/4) Gaussian Elimination 175

76. 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 +3x 1 – 3x 2 +2x 3 =9 +6x 1 – 6x 2 +7x 3 =24 – 3x 1 -(3/-3) MULTIPLIERSMULTIPLIERS -(6/-3) Gaussian Elimination 176

77. 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 1x21x2 +1x 3 =9 2x22x2 +5x 3 =24 – 3x 1 ?? MULTIPLIERMULTIPLIER Gaussian Elimination 177

78. 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 1x21x2 +1x 3 =9 3x33x3 =6 – 3x 1 Gaussian Elimination 178

79. Gaussian Elimination 179 Operation count in Forward Elimination 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2n TOTAL 1 st column: 2n(n-1)  2n 2 2(n-1) 2 2(n-2) 2 ……. b 11 b 22 b 33 b 44 b 55 b 66 b 77 b 66

80. Parallel implementation 180

81. Eigen-values and Eigenvectors 81

82. Definitions 182 Definition 1: A nonzero vector x is an eigenvector (or characteristic vector) of a square matrix A if there exists a scalar λ such that Ax = λx. Then λ is an eigenvalue (or characteristic value) of A. Note: The zero vector can not be an eigenvector even though A0 = λ0. But λ = 0 can be an eigenvalue. Example:

83. Geometric interpretation of Eigenvalues and Eigenvectors 183 An n×n matrix A multiplied by n×1 vector x results in another n×1 vector y=Ax. Thus A can be considered as a transformation matrix. In general, a matrix acts on a vector by changing both its magnitude and its direction. However, a matrix may act on certain vectors by changing only their magnitude, and leaving their direction unchanged (or possibly reversing it). These vectors are the eigenvectors of the matrix. A matrix acts on an eigenvector by multiplying its magnitude by a factor, which is positive if its direction is unchanged and negative if its direction is reversed. This factor is the eigenvalue associated with that eigenvector.

84. Eigen-values 184 Let x be an eigenvector of the matrix A. Then there must exist an eigenvalue λ such that Ax = λx or, equivalently, Ax - λx = 0or (A – λI)x = 0 If we define a new matrix B = A – λI, then Bx = 0 If B has an inverse then x = B -1 0 = 0. But an eigenvector cannot be zero. Thus, it follows that x will be an eigenvector of A if and only if B does not have an inverse, or equivalently det(B)=0, or det(A – λI) = 0 This is called the characteristic equation of A. Its roots determine the eigenvalues of A.

85. Eigen-values: examples Example 1: Find the eigenvalues of two eigenvalues:  1,  2 Note: The roots of the characteristic equation can be repeated. That is, λ 1 = λ 2 =…= λ k. If that happens, the eigenvalue is said to be of multiplicity k. Example 2: Find the eigenvalues of λ = 2 is an eigenvector of multiplicity 3. 185

86. Eigenvectors Example 1 (cont.): 186 To each distinct eigenvalue of a matrix A there will correspond at least one eigenvector which can be found by solving the appropriate set of homogenous equations. If λ i is an eigenvalue then the corresponding eigenvector x i is the solution of (A – λ i I)x i = 0

87. Eigenvectors Example 2 (cont.): Find the eigenvectors of Recall that λ = 2 is an eigenvector of multiplicity 3. Solve the homogeneous linear system represented by Let. The eigenvectors of = 2 are of the form s and t not both zero. 187

88. Properties of Eigen-values and Eigenvectors 188 Definition: The trace of a matrix A, designated by tr(A), is the sum of the elements on the main diagonal. Property 1: The sum of the eigenvalues of a matrix equals the trace of the matrix. Property 2: A matrix is singular if and only if it has a zero eigenvalue. Property 3: The eigenvalues of an upper (or lower) triangular matrix are the elements on the main diagonal. Property 4: If λ is an eigenvalue of A and A is invertible, then 1/λ is an eigenvalue of matrix A -1.

89. Properties of Eigen-values and Eigenvectors 189 Property 5: If λ is an eigenvalue of A then kλ is an eigenvalue of kA where k is any arbitrary scalar. Property 6: If λ is an eigenvalue of A then λk is an eigenvalue of Ak for any positive integer k. Property 8: If λ is an eigenvalue of A then λ is an eigenvalue of AT. Property 9: The product of the eigenvalues (counting multiplicity) of a matrix equals the determinant of the matrix.

90. Linearly independent eigenvectors 190 Theorem: Eigenvectors corresponding to distinct (that is, different) eigenvalues are linearly independent. Theorem: If λ is an eigenvalue of multiplicity k of an n  n matrix A then the number of linearly independent eigenvectors of A associated with λ is given by m = n - r(A- λI). Furthermore, 1 ≤ m ≤ k. Example 2 (cont.): The eigenvectors of = 2 are of the form s and t not both zero. = 2 has two linearly independent eigenvectors