Download presentation
1
Chapter 5 Orthogonality
2
Outline Scalar Product in Rn Orthogonal Subspaces
Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt Orthogonalization Process
3
Scalar product in Rn
4
Def: Let and be vectors in either R2 or R3.
The distance between and is defined to be the number
5
Theorem 5.1.1 If and are two nonzero vectors in either R2 or
R3 and is the angle between them , then
6
Proof: By the law of cosines,
7
Corollary 5.1.2(Cauchy-Schwarz Inequality)
If and are vectors in either R2 or R3, then With equality holding if and only if one of the vectors is or one vector is a multiple of the other.
8
Note: If is the angle between , then
Thus
9
Def: The vectors and in R2(or R3)are said to
be orthogonal if
10
Scalar and Vector Projections
12
Example: Find the point
on the line that is closest to the point (1,4) Sol: Note that the vector is on the line Thus the desired point is
13
Example: Find the equation of the plane
passing through and normal to Sol:
14
Example: Find the distance form
to the plane Sol: a normal vector to the plane is The distance
15
Application 1: Information Retrieval Revisited
Table 1 Frequency of Key words Modules Key Words M1 M2 M3 M4 M5 M6 M7 M8 determines 6 3 1 eignvalues 5 2 linear 4 matrices numerical orthogonality spaces systems transformations vector
16
Application I: Information Retrieval Revisited
A is the matrix corresponding to Table I, then the columns of the database matrix Q are determined by setting To do a search for the key words orthogonality, spaces, vector, we form a unit search vector whose entries are all zero except for the three rows(be put in each of the rows) corresponding to the search rows.
18
Application I: Information Retrieval Revisited
Since is the entry of that is closest to 1,this indicates that the direction of the search vector is closest to the direction of and hence that Module 5 is the one that best matches our search criteria.
19
Application 2: Correlation And Covariance Matrices
Table 2 Math Scores Fall 1996 Scores Student Assignment Exams Final S1 198 200 196 S2 160 165 S3 158 133 S4 150 91 S5 175 182 151 S6 134 135 101 S7 152 136 80 Average 161 163 131
20
Application 2: Correlation And Covariance Matrices
The column vectors of X represent the deviations from the mean for each of the three sets of scores. The three sets of translated data specified by the column vectors of X all have mean 0 and all sum to 0. A cosine value near 1 indicates that the two sets of scores are highly correlated. Scale to make them unit vectors
22
Application 2: Correlation And Covariance Matrices
The matrix C is referred to as a correlation matrix. The three sets of scores in our example are all positively correlated since the correlation coefficients are all positive. A negative coefficient would indicate that two data sets were negatively correlated. A coefficient of 0 would indicate that they were uncorrelated.
23
5-2 Orthogonal Subspaces
Def: Two subspaces X and Y of are said to be orthogonal if = 0 for every and If X and Y are orthogonal, we write
24
Def: Let Y be a subspace of . The set of all vectors in
that are orthogonal to every vector in Y will be denoted Thus = { for every } The set is called the orthogonal complement of Y
25
Remarks: If X and Y are orthogonal subspaces of , then If Y is a subspace of , then is also a subspace of
26
Four Fundamental Subspaces
Let
27
Theorem 5.2.1(Fundamental Subspace Theorem)
pf: Let and Also, if Similarly,
28
Example: Let Clearly,
29
Theorem 5.2.2 If S is a subspace of , then
Furthermore, if { } is a basis for S and { }is a basis for , then { , } is a basis for
30
Proof: If The result follows
Suppose Let and
31
To show that is a basis for ,
It remains to show their independency. Let Then Similarly,
32
Def: If U and V are subspaces of a vector space W
and each can be written uniquely as a sum , where and ,then we say that W is a direct sum of U and V, and we write
33
then pf: By Theorem5.2.2, To show uniqueness, Suppose where
Theorem5.2.3: If is a subspace of , then pf: By Theorem5.2.2, To show uniqueness, Suppose where
34
Theorem5.2.4: If is a subspace of ,
then pf: Let If
35
Remark: Let i.e. , Since and are bijections .
36
Let bijection bijection
37
Cor5.2.5: Let and Then either (i) or (ii) pf:
38
Example: Let Find The basic idea is that the row space and the sol. of are invariant under row operations. Sol: (i) (Why?) (ii) (iii) Similarly, and (iv) Clearly,
39
Example: Let (i) and (ii) The mapping (iv) What is the matrix representation for ?
40
5-4 Inner Product Spaces A tool to measure the
orthogonality of two vectors in general vector space
41
Def: An inner product on a
vector space is a function Satisfying the following conditions: (i) with equality iff (ii) (iii)
42
Example: (i) Let Then is an inner product of (ii) Let , Then is an inner product of (iii) Let and then is an inner product of (iv) Let , is a positive function and are distinct real numbers. Then is an inner product of
43
Def: Let be an inner product of a
vector space and we say The length or norm of is given by
44
Theorem5.4.1: (The Pythagorean Law)
pf:
45
Example 1: Consider with inner product
(ii) (iii) (iv) (Pythagorean Law) or
46
Example 2: Consider with inner product
It can be shown that (i) (ii) (iii) Thus is an orthonormal set.
47
Remark Remark: The inner product in example 2 plays a key
role in Fourier analysis application involving trigo- nometric approximation of functions.
48
Example 3: Let and let Then is not orthogonal to
49
Def: Let be two vectors in an
inner product space Then the scalar projection of onto is defined as The vector projection of onto is
50
Lemma: Let be the vector projection
of onto Then for some pf:
51
Theorem5.4.2: (Cauchy-Schwarz Inequality) Let be two vectors in an
inner product space Then Moreover, equality holds are linear dependent. pf: If If Equality holds i.e., equality holds iff are linear dependent.
52
Note: From Cauchy-Schwarz Inequality for This, we can define as the angle between the two nonzero vectors
53
Def: Let be a vector space a function
is said to be a norm if it satisfies Remark: Such a vector space is called a normed linear space.
54
Theorem5.4.3: If is an inner product
space, then defines a norm on pf: trivial Def: The distance between is defined as
55
Example: Let , then
56
Example: Let Thus, However, (Why?)
Remark: In the case of a norm that is not derived from an inner product, the Pythagorean Law will not hold. Example: Let Thus, However, (Why?)
57
Example: Let , then
58
Example: Let Then
59
5-3 Least Squares Problems
60
Least squares problems
A typical example: Given Find the best line to fit the data . or or find such that is minimum Geometrical meaning :
61
Least squares problems:
Given then the equation may not have solutions The objective of least square problem is trying to find such that is minimum value i.e., find satisfying
62
Preview of the results:
It will be shown that If columns of are linear independent .
63
Theorem5.3.1: Let be a subspace of , then (i) for all (ii) pf: (i)
where If (ii) follows directly from (i) by noting that unique expression
64
Question: How to find which solves
Ans.: From previous Theorem , we know that Definition:
65
Remark: one solution to the normal equation.
In general, it is possible to have more than one solution to the normal equation. If is a solution, then the general solution is of the form
66
Theorem5.3.2: Let and Then the normal equation has an unique solution and is the unique least squares solution to pf: To show that is nonsingular
67
Note: The projection vector
is the element of that is closet to in the least squares sense . Thus, The matrix is called the projection matrix (that project any vector of to )
68
Application 2: Spring Constants
Suppose a spring obeys the Hook’s law and a series of data are taken (with measurement error) as How to determine ? sol: Note that is inconsistent The normal equation is so,
69
Example 2: Given the data
Find the best least squares fit by a linear function. sol: Let the desired linear function be The problem becomes to find the least squares solution of is the unique solution. Thus, the best linear least square fit is ∵ rank(A)=2
70
Example3: Find the best quadratic least squares fit to the data
sol: Let the desired quadratic function be The problem becomes to find the least square solution of is the unique solution. Thus, the best quadratic least square fit is ∵ rank(A)=3
71
5-5 Orthonormal Sets
72
Orthonormal Set Simplify the least squares solution
(avoid computing inverse) Numerical computational stability
73
Def: is said to be an orthogonal set in
an inner product space if Moreover, if , then is said to be orthonormal.
74
Example 2: is an orthogonal set but not orthonormal. However , is orthonormal.
75
Theorem5.5.1: Let be an orthogonal
set of nonzero vectors in an inner product space Then they are linear independent. pf: Suppose that
76
Example: is an orthonormal set of with inner product Note: Now you know the meaning what one says that
77
Theorem5.5.2: Let be an orthonormal
basis for an inner product space . If , then pf:
78
Cor: Let be an orthonormal basis for
an inner product space . If and , then pf:
79
Cor: (Parseval’s Formula)
If is an orthonormal basis for an inner product space and , then pf: By Corollary ,
80
Example 4: and form an orthonormal basis for If , then and
81
Example 5: Determine without computing
antiderivatives . sol:
82
Def: is said to be an orthogonal matrix if the column vectors of form an
orthonormal set in Example 6: The rotational matrix and the elementary reflection matrix are orthogonal matrix .
83
Properties of orthogonal matrices:
If is orthogonal, then
84
Theorem 5.5.6: If the columns of form an orthonormal set in , then and the least squares solution to is This avoid computing matrix inverse .
85
Theorem & 5.5.8: Let be a subspace of an inner product space and let Let be an orthonormal basis for . If , where , then
86
Cor5.5.9: Let be a subspace of and If be an orthonormal basis for and then the projection of onto is pf:
87
Note: Let columns of be an
orthonormal set
88
Example 7: Let Find the vector in that is closet to Sol:
89
Approximation of functions
Example 8: Find the best least squares approximation to on by a linear function . Sol:
90
Sol:
91
Approximation of trigonometric polynomials
FACT: forms an orthonormal set in with respect to the inner product Problem: Given a continuous 2π-periodic function , find a trigonometric polynomial of degree n which is a best least squares approximation to
92
Sol: It suffices to find the projection of onto
the subspace The best approximation of has coefficients
93
Example: Consider with inner product of
(i) Check that is orthonormal (ii) Let
94
(iii) (iv)
95
5-6 Gram-Schmidt Orthogonalization Process
96
Cram-Schmidt Orthogonalization Process
Question: Given an ordinary basis , how to transform them into an orthonormal basis ?
97
Given ,Clearly Clearly, Similarly, We have the next result
98
Theorem5.6.1: (The Gram-Schmidt process)
H. (i) Let be a basis for an inner product space (ii) C is an orthonormal basis.
99
Example: Find an orthonormal basis for with
inner product given by , where Sol: Starting with a basis
100
Theorem5.6.2: (QR Factorization)
If A is an m×n matrix of rank n, then A can be factored into a product QR, where Q is an m×n matrix with orthonormal columns and R is an n×n matrix that is upper triangular and invertible.
101
Proof. of QR-Factorization
102
Proof. of QR-Factorization (cont.)
103
Theorem5.6.3: If A is an m×n matrix of rank n, then the solution to the least squares problem is given by , where Q and R are the matrices obtained from Thm The solution may be obtained by using back substitution to solve
104
Proof. of Thm.5.6.3
105
Example 3: Solve By direct calculation,
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.