1. Hazırlayan NEURAL NETWORKS Least Squares Estimation PROF. DR. YUSUF OYSAL

2. System Identification NEURAL NETWORKS – Least Squares Estimation Goal Determine a mathematical model for an unknown system (or target system) by observing its input- output data pairs Purposes To predict a system’s behavior, as in time series prediction & weather forecasting To explain the interactions & relationships between inputs & outputs of a system Example: Find a line that represent the ”best” linear relationship: b = ax 2

3. Parameter Identification NEURAL NETWORKS – Least Squares Estimation The data set composed of m desired input-output pairs (u i y i ) (i = 1,…,m) is called the training data System identification needs to do both structure & parameter identification repeatedly until satisfactory model is found: it does this as follows: 1. Specify & parameterize a class of mathematical models representing the system to be identified 2. Perform parameter identification to choose the parameters that best fit the training data set 3. Conduct validation set to see if the model identified responds correctly to an unseen data set 4. Terminate the procedure once the results of the validation test are satisfactory. Otherwise, another class of model is selected & repeat step 2 to 4 3

4. Parameter Identification NEURAL NETWORKS – Least Squares Estimation The structure of the model is known, however we need to apply optimization techniques in order to determine the parameter vector such that the resulting model describes the system appropriately: 4 General form: y =  1 f 1 (u) +  2 f 2 (u) + … +  n f n (u) u = (u 1, …, u p )T is the model input vector f 1, …, f n are known functions of u  1, …,  n are unknown parameters to be estimated

5. Parameter Identification NEURAL NETWORKS – Least Squares Estimation 5 A  = y

6. Parameter Identification NEURAL NETWORKS – Least Squares Estimation 6 A  + e = y

7. Least-Squares Estimator NEURAL NETWORKS – Least Squares Estimation 7 Theorem: Least-Squares Estimator (LSE) The squared error is minimized when (called the least-squares estimators, LSE) satisfies the normal equation If A T A is nonsingular, is unique & is given by

8. Least-Squares Estimator Example NEURAL NETWORKS – Least Squares Estimation 8 Example Spring Example The relationship is between the spring length and the force applied L = k 1 f + k 0 (linear model) Goal: Find = (k 0, k 1 ) T that best fits the data for a given force and length values. The table shows the training data for the spring example. ExperimentForce (Newtons)Length of Spring 11.11.5 21.92.1 33.22.5 44.43.3 55.94.1 67.44.6 79.25.0

9. Least-Squares Estimator Example NEURAL NETWORKS – Least Squares Estimation 9 minimizes