1. Signal & Weight Vector Spaces

2. Notation
Vectors in Ân.
Generalized Vectors. x 1 2 n =

3. Vector Space
1. An operation called vector addition is defined such that if x Î X and y Î X then x+y Î X.
2. x + y = y + x
3. (x + y) + z = x + (y + z)
4. There is a unique vector 0 Î X, called the zero vector, such that x + 0 = x for all x Î X.
5. For each vector there is a unique vector in X, to be called (-x ), such that x + (-x ) = 0 .

4. Vector Space (Cont.)
6. An operation, called multiplication, is defined such that for all scalars a Î F, and all vectors x Î X, a x Î X.
7. For any x Î X , 1x = x (for scalar 1).
8. For any two scalars a Î F and b Î F, and any x Î X, a (bx) = (a b) x .
9. (a + b) x = a x + b x .
10. a (x + y)  = a x + a y

5. Examples (Decision Boundaries)
Is the p2, p3 plane a vector space?
Is the line p1 + 2p2 - 2 = 0 a vector space?

6. Other Vector Spaces Polynomials of degree 2 or less.
Continuous functions in the interval [0,1].

7. is a set of linearly independent vectors.
Linear Independence If
implies that each
then
is a set of linearly independent vectors.

8. Example (Banana and Apple)
Let
This can only be true if
Therefore the vectors are independent.

9. Spanning a Space
A subset spans a space if every vector in the space can be written as a linear combination of the vectors in the subspace.

10. Basis Vectors
A set of basis vectors for the space X is a set of vectors which spans X and is linearly independent.
The dimension of a vector space, Dim(X), is equal to the number of vectors in the basis set.
Let X be a finite dimensional vector space, then every basis set of X has the same number of elements.

11. Example Polynomials of degree 2 or less. Basis A: Basis B:
(Any three linearly independent vectors
in the space will work.)
How can you represent the vector x = 1+2t using both basis sets?

12. Inner Product / Norm
A scalar function of vectors x and y can be defined as
an inner product, (x,y), provided the following are
satisfied (for real inner products):
(x,y) = (y,x) .
(x,ay1+by2) = a(x ,y1) + b(x ,y2) .
(x , x)  0 , where equality holds iff x = 0 .
A scalar function of a vector x is called a norm, ||x||,
provided the following are satisfied:
||x||  0 .
||x|| = 0 iff x = 0 .
||a x|| = |a| ||x|| for scalar a .
||x + y|| Š ||x|| + ||y|| .

13. Example Angle Standard Euclidean Inner Product Standard Euclidean Norm
||x|| = (x , x)1/2
||x|| = (xTx)1/2 = (x12 + x xn2) 1/2
Angle
cos(q) = (x ,y)/(||x|| ||y||)

14. Two vectors x,y ÎX are orthogonal if (x,y) = 0 .
Orthogonality
Two vectors x,y ÎX are orthogonal if (x,y) = 0 .
Example
Any vector in the p2,p3 plane is
orthogonal to the weight vector.

15. Gram-Schmidt Orthogonalization
Independent Vectors
Orthogonal Vectors
Step 1: Set first orthogonal vector to first independent vector.
Step 2: Subtract the portion of y2 that is in the direction of v1.
Where a is chosen so that v2 is orthogonal to v1:

16. Step k: Subtract the portion of yk that is in the direction of all
Gram-Schmidt (Cont.)
Projection of y2 on v1:
Step k: Subtract the portion of yk that is in the direction of all
previous vi .

17. Example
Step 1.

18. Example (Cont.)
Step 2.

19. Vector Expansion
If a vector space X has a basis set {v1, v2, ..., vn},
then any xÎX has a unique vector expansion:
If the basis vectors are orthogonal, and we
take the inner product of vj and x :
Therefore the coefficients of the expansion can be computed:

20. Column of Numbers The vector expansion provides a meaning for
writing a vector as a column of numbers. x 1 2 n =
To interpret x, we need to know what basis was used
for the expansion.

21. Reciprocal Basis Vectors
Definition of reciprocal basis vectors, ri: r i v j ( , ) = 1
where the basis vectors are {v1, v2, ..., vn}, and
the reciprocal basis vectors are {r1, r2, ..., rn}.
For vectors in Ân we can use the following inner product:
Therefore, the equations for the reciprocal basis vectors become:

22. Vector Expansion
Take the inner product of the first reciprocal basis vector
with the vector to be expanded:
By definition of the reciprocal basis vectors: r 1 v 2 ( , ) 3 n = r v 1 ( , ) =
Therefore, the first coefficient in the expansion is:
In general, we then have (even for nonorthogonal basis vectors):

23. Example
Basis Vectors:
Vector to Expand:

24. Example (Cont.) Reciprocal Basis Vectors: Expansion Coefficients:
Matrix Form:

25. Example (Cont.) x s v The interpretation of the column of numbers1 – ( ) s 2 + v
1.5 - = x v
1.5 – 2 =
The interpretation of the column of numbers
depends on the basis set used for the expansion.