1. Linear Transformations
這邊所謂的linear Transformations指的就是前面的Weight轉換，但沒有bias
因為如果有bias就不會是linear
7,10

2. Hopfield Network Questions

3. Hopfield Network Questions
The network output is repeatedly multiplied by the weight matrix W.
What is the effect of this repeated operation?
Will the output converge, go to infinity, oscillate?
In this chapter we want to investigate matrix multiplication, which represents a general linear transformation.
輸出是藉由W一再倍增
重覆運算的影響是什麼
輸出會收斂還是無窮，或是振盪
這章要研究矩陣乘法所代表的線性轉換

4. Linear Transformations
A transformation consists of three parts:
1. A set of elements X = {x i}, called the domain,
2. A set of elements Y = {y i}, called the range, and
3. A rule relating each x i Î X to an element y i Î Y.
A transformation is linear if:
1. For all x 1, x 2 Î X, A (x 1 + x 2 ) = A (x 1) + A (x 2 ),
2. For all x Î X, a Î Â , A (a x ) = a A (x ) .

5. Example - Rotation
Is rotation linear? θ

6. Example - Rotation
Is rotation linear? 1. 2.

7. Example - Translation Is translation linear? 𝒳+𝒯 𝒳 𝒯 𝒜 𝒳 =𝒳+𝒯
𝒜 𝒳1+𝒳2 =𝒳1+𝒳2+𝒯
𝒜 𝒳1 +𝒜 𝒳2 =𝒳1+𝒳2+2𝒯
𝒜 𝒳1+𝒳2 ≠𝒜 𝒳1 +𝒜 𝒳2

8. Matrix Representation - (1)
Any linear transformation between two finite-dimensional vector spaces can be represented by matrix multiplication.
Let {v1, v2, ..., vn} be a basis for X, and let {u1, u2, ..., um} be a basis for Y.
𝜒= 𝑖=1 𝑛 𝑥𝑖𝑣𝑖
𝒴= 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
Let 𝒜:X®Y
𝒜 𝑗=1 𝑛 𝑥𝑗𝑣𝑗 = 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
𝒜 𝒳 =𝒴

9. Matrix Representation - (2)
𝑗=1 𝑛 𝑥𝑗𝒜(𝑣𝑗 )= 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
Since 𝒜 is a linear operator,
𝒜 𝑣𝑗 = 𝑖=1 𝑚 𝑎𝑖𝑗𝑢𝑖
Since the ui are a basis for Y,
𝑗=1 𝑛 𝑥𝑗 𝑖=1 𝑚 𝑎𝑖𝑗𝑢𝑖 = 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
𝑖=1 𝑚 𝑢𝑖 𝑗=1 𝑛 𝑎𝑖𝑗𝑥𝑗 = 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
當m和n有限時，可以交換順序
(The coefficients a i j will make up the matrix representation of the transformation.)

10. Matrix Representation - (3)
𝑖=1 𝑚 𝑢𝑖 𝑗=1 𝑛 𝑎𝑖𝑗𝑥𝑗 = 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
𝑖=1 𝑚 𝑢𝑖 𝑗=1 𝑛 𝑎𝑖𝑗𝑥𝑗 −𝑦𝑖 =0
𝑗=1 𝑛 𝑎𝑖𝑗𝑥𝑗 =𝑦𝑖
Because the u i are independent,
X為input，y為output a 11 12 1 n 21 22 2 m x y =
This is equivalent to matrix multiplication.

11. Summary
A linear transformation can be represented by matrix multiplication.
To find the matrix which represents the transformation we must transform each basis vector for the domain and then expand the result in terms of the basis vectors of the range.
𝒜 𝑣𝑗 = 𝑖=1 𝑚 𝑎𝑖𝑗𝑢𝑖
Each of these equations gives us one column of the matrix.

12. To find the matrix we need to transform each of the basis vectors.
Example - (1)
To find the matrix we need to transform each of the basis vectors.
𝒜 𝑣𝑗 = 𝑖=1 𝑚 𝑎𝑖𝑗𝑢𝑖
We will use the standard basis vectors for both the domain and the range.
𝒜 𝑠𝑗 = 𝑖=1 2 𝑎𝑖𝑗𝑠𝑖 =𝑎1𝑗𝑠1+𝑎2𝑗𝑠2

13. Example - (1) We begin with s1:
If we rotate s1 counterclockwise by the angle 𝜃 we obtain
𝒜 𝑠1 =𝑐𝑜𝑠𝜃 𝑠1+𝑠𝑖𝑛𝜃 𝑠2= 𝑖=1 2 𝑎𝑖1𝑠𝑖 =𝑎11𝑠1+𝑎21𝑠2
This gives us the first column of the matrix.

14. Example - (1) Next, we skew s2:
If we rotate s2 counterclockwise by the angle 𝜃 we obtain
𝒜 𝑠1 =−𝑠𝑖𝑛𝜃 𝑠1+𝑐𝑜𝑠𝜃 𝑠2= 𝑖=1 2 𝑎𝑖1𝑠𝑖 =𝑎12𝑠1+𝑎22𝑠2
This gives us the second column of the matrix.

15. The matrix of the transformation is:
Example - (1)
The matrix of the transformation is:
𝐀= cosθ −sinθ sinθ cosθ
𝐳=𝑟 𝑐𝑜𝑠𝜌+𝑖𝑠𝑖𝑛𝜌 = 𝑟𝑐𝑜𝑠𝜌 𝑟𝑠𝑖𝑛𝜌
𝐳′=𝑟 cos⁡(𝜌+𝜃)+𝑖𝑠𝑖𝑛(𝜌+𝜃) = 𝑐𝑜𝑠𝜃 −𝑠𝑖𝑛𝜃 𝑠𝑖𝑛𝜃 𝑐𝑜𝑠𝜃 𝑟𝑐𝑜𝑠𝜌 𝑟𝑠𝑖𝑛𝜌 =Az

16. The matrix representation is:
Change of Basis
Consider the linear transformation A:X®Y. Let {v1, v2, ..., vn} be a basis for X, and let {u1, u2, ..., um} be a basis for Y.
𝜒= 𝑖=1 𝑛 𝑥𝑖𝑣𝑖
𝒴= 𝑖=1 𝑚 𝑦𝑖𝑢𝑖
𝒜 𝒳 =𝒴
The matrix representation is: a 11 12 1 n 21 22 2 m x y =
𝐀𝐱=𝐲

17. The new matrix representation is:
New Basis Sets
Now let’s consider different basis sets. Let {t1, t2, ..., tn} be a basis for X, and let {w1, w2, ..., wm} be a basis for Y.
𝜒= 𝑖=1 𝑛 𝑥′𝑖𝑡𝑖
𝒴= 𝑖=1 𝑚 𝑦′𝑖𝑤𝑖
𝒜 𝒳 =𝒴
The new matrix representation is: a ' 11 12 1 n 21 22 2 m x y =
𝐀′𝐱′=𝐲′

18. How are A and A' related?
Expand ti in terms of the original basis vectors for X.
𝐭𝑖= 𝑡1𝑖 𝑡2𝑖 ⋮ 𝑡𝑛𝑖
𝑡𝑖= 𝑗=1 𝑛 𝑡𝑗𝑖𝑣𝑗
Expand wi in terms of the original basis vectors for Y.
𝐰𝑖= 𝑤1𝑖 𝑤2𝑖 ⋮ 𝑤𝑚𝑖
𝑤𝑖= 𝑗=1 𝑚 𝑤𝑗𝑖𝑢𝑗

19. How are A and A' related? 𝐁t= 𝐭1 𝐭2 ⋯ 𝐭n
𝐱= 𝑥 ′ 1𝐭1+ 𝑥 ′ 2𝐭2+⋯ 𝑥 ′ n𝐭n=𝐁t𝐱′
𝐁w= 𝐰1 𝐰2 ⋯ 𝐰m
𝐲= 𝑦 ′ 1𝐰1+ 𝑦 ′ 2𝐰2+⋯ y ′ m𝐰m=𝐁w𝐲′
𝐀𝐱=𝐲
𝐀𝐁t𝐱′=𝐁w𝐲′
𝐁w−1𝐀𝐁t 𝐱 ′ =𝐲′
𝐀 ′ = 𝐁w−1𝐀𝐁t
𝐀 ′ 𝐱 ′ =𝐲′
Similarity
Transform

20. Example(2)
Consider a transformation A :Ｒ3→Ｒ2 whose matrix representation relative to the standard basis sets is
Find the matrix for this transformation relative to the basis sets :

21. Example(2)
Step 1 : form the matrices
Step 2 : Use

22. Eigenvalues and Eigenvectors
Let A:X®X be a linear transformation. Those vectors
z Î X, which are not equal to zero, and those scalars
l which satisfy
A(z) = l z
are called eigenvectors and eigenvalues, respectively.

23. Computing the Eigenvalues
𝐀𝐳=𝜆𝐳
[𝐀−λ𝐈]𝐳=0
𝐀−λ𝐈=0
𝐴= 𝑐𝑜𝑠𝜃 −𝑠𝑖𝑛𝜃 𝑠𝑖𝑛𝜃 𝑐𝑜𝑠𝜃
𝑐𝑜𝑠𝜃−𝜆 −𝑠𝑖𝑛𝜃 𝑠𝑖𝑛𝜃 𝑐𝑜𝑠𝜃−𝜆 =0
𝜆2−2𝜆𝑐𝑜𝑠𝜃+cos2θ+sin2θ=𝜆2−2𝜆𝑐𝑜𝑠𝜃+1=0
−𝑖𝑠𝑖𝑛𝜃 −𝑠𝑖𝑛𝜃 𝑠𝑖𝑛𝜃 −𝑖𝑠𝑖𝑛𝜃 𝑧11 𝑧21 = 0 0
𝜆1=𝑐𝑜𝑠𝜃+𝑖𝑠𝑖𝑛𝜃
𝐳1= 𝑧11 𝑧21 = 1 𝑖
𝑖𝑠𝑖𝑛𝜃 −𝑠𝑖𝑛𝜃 𝑠𝑖𝑛𝜃 𝑖𝑠𝑖𝑛𝜃 𝑧11 𝑧21 = 0 0
𝜆2=𝑐𝑜𝑠𝜃−𝑖𝑠𝑖𝑛𝜃
𝐳2= 𝑧12 𝑧22 = 𝑖 1

24. Diagonalization
Perform a change of basis (similarity transformation) using
the eigenvectors as the basis vectors. If the eigenvalues are
distinct, the new matrix will be diagonal.
Eigenvectors
Eigenvalues B 1 – A [ ] l 2 = n

25. Example(3) 𝐀= 1 1 1 1 1−𝜆 1 1 1−𝜆 =0 𝜆2−2𝜆=𝜆(𝜆-2)=0 𝜆1=0𝐀=
1−𝜆 1 1 1−𝜆 =0
𝜆2−2𝜆=𝜆(𝜆-2)=0
𝜆1=0
1−𝜆 1 1 1−𝜆 𝐳= 0 0
𝜆2=2
𝑧11 𝑧21 = 0 0
𝐳1= 𝑧11 𝑧21 = 1 −1
𝜆1=0
−1 1 1 −1 𝑧12 𝑧22 = 0 0
𝐳2= 𝑧12 𝑧22 = 1 1
𝜆2=2
𝐀 ′ = 𝐁−1𝐀𝐁 = − −1 1 =
Diagonal Form:

26. Example - (4)
Take the skewing problem described previously, and find the
new matrix representation using the basis set {s1, s2}. t 1
0.5 = t 2 1 – =
(Same basis for
domain and range.)

27. Example - (4)
For q = 45°:

28. Example - (4)
Try a test vector:
Check using reciprocal basis vectors: