1. Chapter 2-OPTIMIZATION
G.Anuradha

2. Contents Derivative-based Optimization Derivative-free Optimization
Descent Methods
The Method of Steepest Descent
Classical Newton’s Method
Step Size Determination
Derivative-free Optimization
Genetic Algorithms
Simulated Annealing
Random Search
Downhill Simplex Search

3. What is Optimization?
Choosing the best element from some set of available alternatives
Solving problems in which one seeks to minimize or maximize a real function

4. Notation of Optimization
Optimize
y=f(x1,x2….xn)
subject to
gj(x1,x2…xn) ≤ / ≥ /= bj
where j=1,2,….n
Eqn:1 is objective function
Eqn:2 a set of constraints imposed on the solution.
x1,x2…xn are the set of decision variables
Note:- The problem is either to maximize or minimize the value of objective function.

7. Complicating factors in optimization
Existence of multiple decision variables
Complex nature of the relationships between the decision variables and the associated income
Existence of one or more complex constraints on the decision variables

8. Types of optimization
Constraint:- Solution is arrived at by maximizing or minimizing the objective function
Unconstraint:- No constraints are imposed on the decision variables and differential calculus can be used to analyze them

9. Least Square Methods for System Identification
System Identification:- Determining a mathematical model for an unknown system by observing the input-output data pairs
System identification is required
To predict a system behavior
To explain the interactions and relationship between inputs and outputs
To design a controller
System identification
Structure identification
Parameter identification

10. Structure identification
Apply a priori knowledge about the target system to determine a class of models within which the search for the most suitable model is conducted
y=f(u;θ)
y – model’s output
u – Input Vector
θ – parameter vector

11. Parameter Identification
Structure of the model is known and optimization techniques are applied to determine the parameter vector θ= θ

12. Block diagram of parameter identification

13. Parameter identification
An input ui is applied to both the system and the model
Difference between the target system’s output yi and model’s output yi is used to update a parameter vector θ to minimize the difference
System identification is not a one-pass process; it needs to do both structure and parameter identification repeatedly

14. Classification of Optimization algorithms
Derivative-based algorithms:-
Derivative-free algorithms

15. Characteristics of derivative free algorithm
Derivative freeness:- repeated evaluation of objective function
Intuitive guidelines:- concepts are based on nature’s wisdom, such as evolution and thermodynamics
Slower
Flexibility
Randomness:- global optimizers
Analytic Opacity:-knowledge about them are based on empirical studies
Iterative nature:-

16. Characteristics of derivative free algorithm
Stopping condition of iteration:- let k denote an iteration count and fk denote the best objective function obtained at count k. stopping condition depends on
Computation time
Optimization goal;
Minimal Improvement
Minimal relative improvement

17. Basics of Matrix Manipulation and Calculus

18. Basics of Matrix Manipulation and Calculus

19. Gradient of a Scalar Function

20. Jacobian of a Vector Function

21. Least Square Estimator
Method of least squares is a standard approach to approximate solution of overdetermined systems.
Least Squares- Overall solution minimizes the sum of the squares of the errors made in solving every single equation
Application—Data Fitting

22. Types of Least Squares Least Squares
Linear:- It is a linear combination of parameters.
The model may represent a straight line, a parabola or any other linear combination of functions
Non-Linear:- the parameters appear as functions, such as β2,eβx. If the derivatives are either constant or depend only on the values of the independent variable, the model is linear else non-linear.

23. Differences between Linear and Non-Linear Least Squares
Algorithms Does not require initial values
Algorithms Require Initial values
Globally concave; Non convergence is not an issue
Non convergence is a common issue
Normally solved using direct methods
Usually an iterative process
Solution is unique
Multiple minima in the sum of squares
Yields unbiased estimates even when errors are uncorrelated with predictor values
Yields biased estimates

24. Model representation Linear regression with one variable
Machine Learning

25. Housing Prices (Portland, OR)
(in 1000s of dollars)
Size (feet2)
Supervised Learning
Given the “right answer” for each example in the data.
Regression Problem
Predict real-valued output

26. Size in feet2 (x) Price ($) in 1000's (y) 2104 460 1416 232 1534 315
Training set of
housing prices
(Portland, OR)
Size in feet2 (x)
Price ($) in 1000's (y)
2104
460
1416
232
1534
315
852
178 …
Notation:
m = Number of training examples
x’s = “input” variable / features
y’s = “output” variable / “target” variable

27. Training Set How do we represent h ? Learning Algorithm Size of house
Estimated price
Linear regression with one variable.
Univariate linear regression.

29. Linear regression with one variable
Cost function
Machine Learning

30. Training Set Size in feet2 (x) Price ($) in 1000's (y) 2104 460 1416
232
1534
315
852
178 …
Hypothesis:
‘s: Parameters
How to choose ‘s ?

32. Idea: Choose so that is close to for our training examplesy x
Idea: Choose so that is close to for our training examples

34. Cost function intuition I
Linear regression with one variable
Cost function intuition I
Machine Learning

35. Simplified
Hypothesis:
Parameters:
Cost Function:
Goal:

36. (for fixed , this is a function of x)
(function of the parameter ) y x

37. (function of the parameter )
(for fixed , this is a function of x) y x

38. (function of the parameter )
(for fixed , this is a function of x) y x

40. Cost function intuition II
Linear regression with one variable
Cost function intuition II
Machine Learning

41. Hypothesis:
Parameters:
Cost Function:
Goal:

42. (for fixed , this is a function of x)
(function of the parameters )
Price ($)
in 1000’s
Size in feet2 (x)

44. (for fixed , this is a function of x)
(function of the parameters )

45. (for fixed , this is a function of x)
(function of the parameters )

46. (for fixed , this is a function of x)
(function of the parameters )

47. (for fixed , this is a function of x)
(function of the parameters )

49. Linear regression with one variable
Gradient descent
Machine Learning

50. Have some function
Want
Outline:
Start with some
Keep changing to reduce until we hopefully end up at a minimum

51. J(0,1) 1 0

52. J(0,1) 1 0

53. Gradient descent algorithm
Correct: Simultaneous update
Incorrect:

55. Gradient descent intuition
Linear regression with one variable
Gradient descent intuition
Machine Learning

56. Gradient descent algorithm

58. If α is too small, gradient descent can be slow.
If α is too large, gradient descent can overshoot the minimum. It may fail to converge, or even diverge.

59. at local optima
Current value of

60. Gradient descent can converge to a local minimum, even with the learning rate α fixed.
As we approach a local minimum, gradient descent will automatically take smaller steps. So, no need to decrease α over time.

62. Gradient descent for linear regression
Linear regression with one variable
Gradient descent for linear regression
Machine Learning

63. Gradient descent algorithm
Linear Regression Model

65. Gradient descent algorithm
update
and
simultaneously

66. J(0,1) 1 0

67. J(0,1) 1 0

69. (for fixed , this is a function of x)
(function of the parameters )

70. (for fixed , this is a function of x)
(function of the parameters )

71. (for fixed , this is a function of x)
(function of the parameters )

72. (for fixed , this is a function of x)
(function of the parameters )

73. (for fixed , this is a function of x)
(function of the parameters )

74. (for fixed , this is a function of x)
(function of the parameters )

75. (for fixed , this is a function of x)
(function of the parameters )

76. (for fixed , this is a function of x)
(function of the parameters )

77. (for fixed , this is a function of x)
(function of the parameters )

78. “Batch”: Each step of gradient descent uses all the training examples.
“Batch” Gradient Descent
“Batch”: Each step of gradient descent uses all the training examples.

80. Multiple features Linear Regression with multiple variables
Machine Learning

81. 2104 460 1416 232 1534 315 852 178 … Multiple features (variables).
Size (feet2)
Price ($1000)
2104
460
1416
232
1534
315
852
178 …

82. Multiple features (variables).
Size (feet2)
Number of bedrooms
Number of floors
Age of home (years)
Price ($1000)
2104 5 1 45
460
1416 3 2 40
232
1534 30
315
852 36
178 …

83. Multiple features (variables).
Size (feet2)
Number of bedrooms
Number of floors
Age of home (years)
Price ($1000)
2104 5 1 45
460
1416 3 2 40
232
1534 30
315
852 36
178 …
Notation:
= number of features
= input (features) of training example.
= value of feature in training example.
Pop-up Quiz

84. Hypothesis:
Previously:

85. For convenience of notation, define .
Multivariate linear regression.

87. Gradient descent for multiple variables
Linear Regression with multiple variables
Gradient descent for multiple variables
Machine Learning

88. Hypothesis: Parameters: Cost function: Gradient descent: Repeat
(simultaneously update for every )
Repeat
Gradient descent:

89. Gradient Descent New algorithm : Repeat Previously (n=1): Repeat
(simultaneously update for )
(simultaneously update )

91. Gradient descent in practice I: Feature Scaling
Linear Regression with multiple variables
Gradient descent in practice I: Feature Scaling
Machine Learning

92. Idea: Make sure features are on a similar scale.
Feature Scaling
Idea: Make sure features are on a similar scale.
size (feet2)
E.g = size ( feet2)
= number of bedrooms (1-5)
number of bedrooms

93. Feature Scaling
Get every feature into approximately a range.

94. Mean normalization
Replace with to make features have approximately zero mean (Do not apply to ).
E.g.

96. Gradient descent in practice II: Learning rate
Linear Regression with multiple variables
Gradient descent in practice II: Learning rate
Machine Learning

97. Gradient descent
“Debugging”: How to make sure gradient descent is working correctly.
How to choose learning rate .

98. Making sure gradient descent is working correctly.
Example automatic convergence test:
Declare convergence if decreases by less than in one iteration.
No. of iterations

99. Making sure gradient descent is working correctly.
Gradient descent not working.
Use smaller .
No. of iterations
No. of iterations
No. of iterations
For sufficiently small , should decrease on every iteration.
But if is too small, gradient descent can be slow to converge.

100. Summary:
If is too small: slow convergence.
If is too large: may not decrease on every iteration; may not converge.
To choose , try

102. Features and polynomial regression
Linear Regression with multiple variables
Features and polynomial regression
Machine Learning

103. Housing prices prediction

104. Polynomial regression
Price
(y)
Size (x)

105. Choice of features
Price
(y)
Size (x)

107. Normal equation Linear Regression with multiple variables
Machine Learning

108. Gradient Descent
Normal equation: Method to solve for
analytically.

109. Intuition: If 1D
(for every )
Solve for

110. Examples:
Size (feet2)
Number of bedrooms
Number of floors
Age of home (years)
Price ($1000) 1
2104 5 45
460
1416 3 2 40
232
1534 30
315
852 36
178
Size (feet2)
Number of bedrooms
Number of floors
Age of home (years)
Price ($1000)
2104 5 1 45
460
1416 3 2 40
232
1534 30
315
852 36
178
Pop-up Quiz

111. Examples:
Size (feet2)
Number of bedrooms
Number of floors
Age of home (years)
Price ($1000) 1
2104 5 45
460
1416 3 2 40
232
1534 30
315
852 36
178
Size (feet2)
Number of bedrooms
Number of floors
Age of home (years)
Price ($1000)
2104 5 1 45
460
1416 3 2 40
232
1534 30
315
852 36
178
3000 4 38
540
Pop-up Quiz

112. examples ; features.
E.g. If

113. is inverse of matrix
Octave:
pinv(X’*X)*X’*y

114. training examples, features.
Gradient Descent
Normal Equation
Need to choose .
Needs many iterations.
No need to choose .
Don’t need to iterate.
Works well even when is large.
Need to compute
Slow if is very large.

116. Normal equation and non-invertibility (optional)
Linear Regression with multiple variables
Normal equation and non-invertibility (optional)
Machine Learning

117. What if is non-invertible? (singular/ degenerate)
Normal equation
What if is non-invertible? (singular/ degenerate)
Octave:
pinv(X’*X)*X’*y

118. What if is non-invertible?
Redundant features (linearly dependent).
E.g size in feet2
size in m2
Too many features (e.g ).
Delete some features, or use regularization.

120. Linear model
Regression Function

121. Linear model contd…
Using matrix notation Where A is a m*n matrix

122. Due to noise a small amount of error is added

123. Least Square Estimator

124. Problem on Least Square Estimator

125. Derivative Based Optimization
Deals with gradient-based optimization techniques, capable of determining search directions according to an objective function’s derivative information
Used in optimizing non-linear neuro-fuzzy models,
Steepest descent
Conjugate gradient

126. First-Order Optimality Conditionx ( ) * D + Ñ T = 1 2 -
For small Dx:
If x* is a minimum, this implies: If
then
But this would imply that x* is not a minimum. Therefore
Since this must be true for every Dx,

127. Second-Order Condition
If the first-order condition is satisfied (zero gradient), then
A strong minimum will exist at x* if
for any Dx ° 0.
Therefore the Hessian matrix must be positive definite. A matrix A is positive definite if:
for any z ° 0.
This is a sufficient condition for optimality.
A necessary condition is that the Hessian matrix be positive semidefinite. A matrix A is positive semidefinite if:
for any z.

128. Basic Optimization Algorithm
pk - Search Direction
ak - Learning Rate