1. Signal , Weight Vector Spaces and Linear Transformations

2. Notation Vectors in Ân. Generalized Vectors. 𝐱 𝐱=𝑎nxn+…+𝑎1x+𝑎02 n = 𝐱
𝐱=𝑎nxn+…+𝑎1x+𝑎0
𝐱= 𝑎1,1 𝑎1,2 𝑎2,1 𝑎2,2

3. Vector Space
1. An operation called vector addition is defined such that if x Î X and y Î X then x + y Î X.
2. x + y = y + x
3. (x + y) + z = x + (y + z)
4. There is a unique vector 0 Î X, called the zero vector, such that x + 0 = x for all x Î X.
5. For each vector there is a unique vector in X, to be called (-x ), such that x + (-x ) = 0 .

4. Vector Space (Cont.)
6. An operation, called multiplication, is defined such that for all scalars a Î F, and all vectors x  Î X, a x  Î X.
7. For any x  Î X , 1x = x (for scalar 1).
8. For any two scalars a Î F and b Î F, and any x Î X, a (bx) = (a b) x .
9. (a + b) x = a x + b x .
10. a (x + y)  = a x + a y

5. Examples (Decision Boundaries)
Is the p2, p3 plane a vector space?

6. Examples (Decision Boundaries)
Is the line p1 + 2p2 - 2 = 0 a vector space?

7. Other Vector Spaces Polynomials of degree 2 or less. 𝐶[0,1]:
Continuous functions in the interval [0,1].
𝜒=sin⁡(𝑡)
𝒴=𝑒−2𝑡

8. is a set of linearly independent vectors.
Linear Independence If
implies that each
then
is a set of linearly independent vectors.

9. Therefore the vectors are independent.
Example
Let
This can only be true if
Therefore the vectors are independent.

10. Problem P5.3(Gramian) i. 𝐱1= 1 1 1 𝐱2= 1 0 1 𝐱3= 1 2 1
= −
=−2+0+2=0
G= (𝐱1,𝐱1) (𝐱1,𝐱2) (𝐱1,𝐱3) (𝐱2,𝐱1) (𝐱2,𝐱2) (𝐱2,𝐱3) (𝐱3,𝐱1) (𝐱3,𝐱2) (𝐱3,𝐱3) =
= − =24−8−16=0

11. Problem P5.3(Gramian)(cont.)
𝐢𝐢𝐢. 𝐱1= 𝐱2= 𝐱3=
G= (𝐱1,𝐱1) (𝐱1,𝐱2) (𝐱1,𝐱3) (𝐱2,𝐱1) (𝐱2,𝐱2) (𝐱2,𝐱3) (𝐱3,𝐱1) (𝐱3,𝐱2) (𝐱3,𝐱3) =
= − =48−18−30=0
2 𝐱1- 𝐱2 −𝐱3=0
G= =4≠0

12. Gramian
𝐏=[𝐱1,𝐱2,⋯,𝐱𝑛]
G 𝐱1,𝐱2,⋯,𝐱𝑛 = 𝐏T×𝐏 = (𝐱1,𝐱1) (𝐱1,𝐱2) (𝐱2,𝐱1) (𝐱2,𝐱2) ⋯ (𝐱1,𝐱𝑛) (𝐱2,𝐱𝑛) ⋮ ⋱ ⋮ (𝐱𝑛,𝐱1) (𝐱𝑛,𝐱2) ⋯ (𝐱𝑛,𝐱𝑛)
G 𝐱1,𝐱2,⋯,𝐱𝑛 ≠0 iff 𝐱1,𝐱2,⋯,𝐱𝑛 are linearly independent
G 𝐱1,𝐱2,⋯,𝐱𝑛 =0 iff 𝐱1,𝐱2,⋯,𝐱𝑛 are linearly dependent

13. Spanning a Space
A subset spans a space if every vector in the space can be written as a linear combination of the vectors in the subspace.

14. Basis Vectors
A set of basis vectors for the space X is a set of vectors which spans X and is linearly independent.
The dimension of a vector space, Dim(X), is equal to the number of vectors in the basis set.
Let X be a finite dimensional vector space, then every basis set of X has the same number of elements.

15. Example Polynomials of degree 2 or less. Basis A: Basis B:
(Any three linearly independent vectors
in the space will work.)
How can you represent the vector x = 1+2t using both basis sets?
𝜒= 𝐮1+𝐮2 2 +𝐮2−𝐮1= 3𝐮2−𝐮1 2
𝐮1+𝐮2=2,𝐮2−𝐮1=2t

16. Inner Product / Norm
A scalar function of vectors x and y can be defined as
an inner product, (x , y ), provided the following are
satisfied (for real inner products):
(x , y ) = (y , x ) .
(x , ay1+by2) = a( x , y1) + b( x , y2) .
(x , x) ≧ 0 , where equality holds iff x = 0 .
A scalar function of a vector x is called a norm, ||x ||
provided the following are satisfied:
||x || ≧0 .
||x || = 0 iff x = 0 .
||a x || = |a| ||x || for scalar a .
||x + y || ≤ ||x || + ||y || .

17. Example Angle: Standard Euclidean Inner Product
Standard Euclidean Norm
||x || = (x , x)1/2
||x|| = (xTx)1/2 = x12 + x xn2
𝐶[0,1] Inner Product(see Problem P5.6):
𝑥, 𝑦 = 0 1 𝑥 𝑡 𝑦 𝑡 𝑑𝑡
cosq = (x , y) ||x || ||y ||
Angle:

18. Problem P5.6 An inner product must satisfy the following properties.
1. 𝑥,𝑦 =(𝑦,𝑥)
𝑥,𝑦 = 0 1 𝑥 𝑡 𝑦 𝑡 𝑑𝑡 = 0 1 𝑦 𝑡 𝑥 𝑡 𝑑𝑡 =(𝑦,𝑥)
2. 𝑥,𝑎𝑦1+𝑏𝑦2 =𝑎 𝑥,𝑦1 +𝑏(𝑥,𝑦2)
𝑥,𝑎𝑦1+𝑏𝑦2 = 0 1 𝑥 𝑡 𝑎𝑦1 𝑡 +𝑏𝑦2 𝑡 𝑑𝑡
= 0 1 𝑥 𝑡 𝑎𝑦1 𝑡 𝑑𝑡 𝑥 𝑡 𝑏𝑦2 𝑡 𝑑𝑡= 𝑎 𝑥,𝑦1 +𝑏(𝑥,𝑦2)

19. Problem P5.6(cont.)
3. 𝑥,𝑥 ≥0,where equality holds iff x is the zero vector
𝑥,𝑥 = 0 1 𝑥 𝑡 𝑥 𝑡 𝑑𝑡 0 1 𝑥 𝑡 2𝑑𝑡 ≥0
Equality holds here only if 𝑥 𝑡 =0 for 0≤𝑡≤1
,which is the zero vector

20. Two vectors x , y ÎX are orthogonal if (x , y) = 0 .
Orthogonality
Two vectors x , y ÎX are orthogonal if (x , y) = 0 .
Example
Any vector in the p2,p3 plane is
orthogonal to the weight vector.

21. Gram-Schmidt Orthogonalization
Independent Vectors
Orthogonal Vectors
𝑦1,𝑦2,…,𝑦𝑛
𝑣1,𝑣2,…,𝑣𝑛
Step 1: Set first orthogonal vector to first independent vector.
𝑣1=𝑦1
Step 2: Subtract the portion of y2 that is in the direction of v1.
𝑣2=𝑦2−𝑎𝑣1
Where a is chosen so that v2 is orthogonal to v1:
𝑣1,𝑣2 = 𝑣1,𝑦2−𝑎𝑣1 = 𝑣1,𝑦2 −𝑎 𝑣1,𝑣1 =0
𝑎= (𝑣1,𝑦2) (𝑣1,𝑣1)

22. Gram-Schmidt (Cont.) Projection of y2 on v1: (𝑣1,𝑦2) (𝑣1,𝑣1) 𝑣1
Step k: Subtract the portion of yk that is in the direction of all
previous vi .
𝑣𝑘=𝑦𝑘− 𝑖=1 𝑘−1 (𝑣𝑖,𝑦𝑘) (𝑣𝑖,𝑣𝑖) 𝑣𝑖

23. Example
𝐲1= ,𝐲2=

24. Example
Step 1：
𝐯1=𝐲1= 2 1

25. Example (Cont.)
Step 2.

26. Vector Expansion
If a vector space X has a basis set {v1, v2, ..., vn},
then any x ÎX has a unique vector expansion:
𝜒= 𝑖=1 𝑛 𝑥𝑖𝑣𝑖 =𝑥1𝑣1+𝑥2𝑣2+…+𝑥𝑛𝑣𝑛
If the basis vectors are orthogonal, and we
take the inner product of vj and x :
𝑣j,𝜒 = 𝑣j, 𝑖=1 𝑛 𝑥i𝑣i = 𝑖=1 𝑛 𝑥i(𝑣j,𝑣i) =𝑥j(𝑣j,𝑣j)
Therefore the coefficients of the expansion can be computed:
𝑥j= (𝑣j,𝜒) (𝑣j,𝑣j)

27. Column of Numbers The vector expansion provides a meaning for
writing a vector as a column of numbers.
𝜒= 𝑖=1 𝑛 𝑥𝑖𝑣𝑖 =𝑥1𝑣1+𝑥2𝑣2+…+𝑥𝑛𝑣𝑛
𝑥= 𝑥1 𝑥2 ⋮ 𝑥𝑛
To interpret x, we need to know what basis was used
for the expansion.

28. Reciprocal Basis Vectors
Definition of reciprocal basis vectors, ri:
𝑟𝑖,𝑣𝑗 = 𝑖≠𝑗
= 𝑖=𝑗
where the basis vectors are {v1, v2, ..., vn}, and
the reciprocal basis vectors are {r1, r2, ..., rn}.
For vectors in Ân we can use the following inner product:
𝑟𝑖,𝑣 𝑗 =𝑟 𝑖 𝑇𝑣𝑗
倒數基底
Therefore, the equations for the reciprocal basis vectors become:

29. Vector Expansion 𝜒=𝑥1𝑣1+𝑥2𝑣2+…+𝑥𝑛𝑣𝑛
Take the inner product of the first reciprocal basis vector
with the vector to be expanded:
𝑟1,𝜒 =𝑥1 𝑟1,𝑣1 +𝑥2 𝑟2,𝑣2 +…+𝑥𝑛 𝑟𝑛,𝑣𝑛
By definition of the reciprocal basis vectors:
𝑟1,𝑣2 = 𝑟1,𝑣3 =…= 𝑟1,𝑣𝑛 =0
𝑟1,𝑣1 =1
Therefore, the first coefficient in the expansion is:
𝑥1=(𝑟1,𝜒)
In general, we then have (even for nonorthogonal basis vectors):
𝑥𝑗=(𝑟𝑗 ,𝜒)

30. Example
𝑣1𝑠= ,𝑣2𝑠= 1 2
Basis Vectors:
𝑥𝑠=
Vector to Expand:

31. Example (Cont.) Reciprocal Basis Vectors: Expansion Coefficients:
𝐫1= − 𝐫2= −
Expansion Coefficients:
Matrix Form:
𝐱1𝑣=𝐫1𝑡𝐱𝑠= − =− 1 2
𝐱𝑣=𝐑𝑇𝐱𝑠=𝐁−1𝐱s
= − 1 3 − = −
𝐱2𝑣=𝐫2𝑡𝐱𝑠= − =1

32. Example (Cont.) χ=0𝑠1+ 3 2 𝑠2=− 1 2 𝑣1+1𝑣2
The interpretation of the column of numbers
depends on the basis set used for the expansion.

33. Linear Transformation
𝑣1 𝑣2 = 𝑠1 𝑠 = 𝑠1 𝑠2 𝐁
𝑠1 𝑠2 =𝐁−1 v1 𝑣2
3 2 𝑠2= 𝑠1 𝑠 = 𝑣1 𝑣2 𝐁− = 𝑣1 𝑣 −1 3 −
= 𝑣1 𝑣2 − = − 1 2 𝑣1+𝑣2
𝐱𝑣=𝐁−1𝐱𝑠 . This operation, call a change of basis