Computacion Inteligente Least-Square Methods for System Identification.

Slides:



Advertisements
Similar presentations
Modeling of Data. Basic Bayes theorem Bayes theorem relates the conditional probabilities of two events A, and B: A might be a hypothesis and B might.
Advertisements

Regression Eric Feigelson Lecture and R tutorial Arcetri Observatory April 2014.
Generalized Method of Moments: Introduction
11-1 Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis.
1 12. Principles of Parameter Estimation The purpose of this lecture is to illustrate the usefulness of the various concepts introduced and studied in.
Chapter 7. Statistical Estimation and Sampling Distributions
Model Building For ARIMA time series
Hazırlayan NEURAL NETWORKS Least Squares Estimation PROF. DR. YUSUF OYSAL.
6-1 Introduction To Empirical Models 6-1 Introduction To Empirical Models.
11 Simple Linear Regression and Correlation CHAPTER OUTLINE
The General Linear Model. The Simple Linear Model Linear Regression.
1 Chapter 2 Simple Linear Regression Ray-Bing Chen Institute of Statistics National University of Kaohsiung.
Visual Recognition Tutorial
The Simple Linear Regression Model: Specification and Estimation
Maximum likelihood (ML) and likelihood ratio (LR) test
Resampling techniques Why resampling? Jacknife Cross-validation Bootstrap Examples of application of bootstrap.
Point estimation, interval estimation
Maximum likelihood (ML) and likelihood ratio (LR) test
Linear and generalised linear models
Linear and generalised linear models
Basics of regression analysis
Linear and generalised linear models Purpose of linear models Least-squares solution for linear models Analysis of diagnostics Exponential family and generalised.
11-1 Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis.
Chapter 5 Transformations and Weighting to Correct Model Inadequacies
Maximum likelihood (ML)
Principles of the Global Positioning System Lecture 10 Prof. Thomas Herring Room A;
Chapter 10 Real Inner Products and Least-Square (cont.)
Introduction to Linear Regression and Correlation Analysis
2-1 MGMG 522 : Session #2 Learning to Use Regression Analysis & The Classical Model (Ch. 3 & 4)
Prof. Dr. S. K. Bhattacharjee Department of Statistics University of Rajshahi.
Computacion Inteligente Least-Square Methods for System Identification.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Deterministic vs. Random Maximum A Posteriori Maximum Likelihood Minimum.
Modern Navigation Thomas Herring
1 11 Simple Linear Regression and Correlation 11-1 Empirical Models 11-2 Simple Linear Regression 11-3 Properties of the Least Squares Estimators 11-4.
PROBABILITY AND STATISTICS FOR ENGINEERING Hossein Sameti Department of Computer Engineering Sharif University of Technology Principles of Parameter Estimation.
Business Statistics: A Decision-Making Approach, 6e © 2005 Prentice-Hall, Inc. Chap 13-1 Introduction to Regression Analysis Regression analysis is used.
Selecting Input Probability Distribution. Simulation Machine Simulation can be considered as an Engine with input and output as follows: Simulation Engine.
: Chapter 3: Maximum-Likelihood and Baysian Parameter Estimation 1 Montri Karnjanadecha ac.th/~montri.
Chapter 7 Point Estimation of Parameters. Learning Objectives Explain the general concepts of estimating Explain important properties of point estimators.
1 Standard error Estimated standard error,s,. 2 Example 1 While measuring the thermal conductivity of Armco iron, using a temperature of 100F and a power.
CHAPTER 17 O PTIMAL D ESIGN FOR E XPERIMENTAL I NPUTS Organization of chapter in ISSO –Background Motivation Finite sample and asymptotic (continuous)
1  The Problem: Consider a two class task with ω 1, ω 2   LINEAR CLASSIFIERS.
Point Estimation of Parameters and Sampling Distributions Outlines:  Sampling Distributions and the central limit theorem  Point estimation  Methods.
Machine Learning 5. Parametric Methods.
M.Sc. in Economics Econometrics Module I Topic 4: Maximum Likelihood Estimation Carol Newman.
Chapter 2-OPTIMIZATION G.Anuradha. Contents Derivative-based Optimization –Descent Methods –The Method of Steepest Descent –Classical Newton’s Method.
SYSTEMS Identification Ali Karimpour Assistant Professor Ferdowsi University of Mashhad.
Parameter Estimation. Statistics Probability specified inferred Steam engine pump “prediction” “estimation”
R. Kass/W03 P416 Lecture 5 l Suppose we are trying to measure the true value of some quantity (x T ). u We make repeated measurements of this quantity.
Week 21 Statistical Assumptions for SLR  Recall, the simple linear regression model is Y i = β 0 + β 1 X i + ε i where i = 1, …, n.  The assumptions.
Estimation Econometría. ADE.. Estimation We assume we have a sample of size T of: – The dependent variable (y) – The explanatory variables (x 1,x 2, x.
Presentation : “ Maximum Likelihood Estimation” Presented By : Jesu Kiran Spurgen Date :
STA302/1001 week 11 Regression Models - Introduction In regression models, two types of variables that are studied:  A dependent variable, Y, also called.
Chapter 4 Basic Estimation Techniques
Chapter 7. Classification and Prediction
Deep Feedforward Networks
11-1 Empirical Models Many problems in engineering and science involve exploring the relationships between two or more variables. Regression analysis.
12. Principles of Parameter Estimation
Probability Theory and Parameter Estimation I
The Simple Linear Regression Model: Specification and Estimation
Ch3: Model Building through Regression
Simultaneous equation system
The Regression Model Suppose we wish to estimate the parameters of the following relationship: A common method is to choose parameters to minimise the.
Statistical Assumptions for SLR
10701 / Machine Learning Today: - Cross validation,
Simple Linear Regression
Parametric Methods Berlin Chen, 2005 References:
12. Principles of Parameter Estimation
Applied Statistics and Probability for Engineers
Regression Models - Introduction
Presentation transcript:

Computacion Inteligente Least-Square Methods for System Identification

Content  System Identification: an Introduction  Least-Squares Estimators  Statistical Properties & the Maximum Likelihood Estimator  LSE for Nonlinear Models

3 System Identification: Introduction  Goal –Determine a mathematical model for an unknown system (or target system) by observing its input-output data pairs

4 System Identification: Introduction  Purposes –To predict a system’s behavior, –As in time series prediction & weather forecasting –To explain the interactions & relationships between inputs & outputs of a system

5 System Identification: Introduction  Context –To design a controller based on the model of a system, –as an aircraft or ship control –Simulate the system under control once the model is known

6 System Identification: Introduction  There are 2 main steps that are involved –Structure identification –Parameter identification

7 System Identification: Introduction  Structure identification  Apply a-priori knowledge about the target system to determine a class of models within which the search for the most suitable model is to be conducted This class of model is denoted by a function y = f(u,  ) where: y is the model output u is the input vector  Is the parameter vector

8 System Identification: Introduction  Structure identification  f(u,  ) depends on –the problem at hand –the designer’s experience –the laws of nature governing the target system

9 System Identification: Introduction  Parameter identification –The structure of the model is known, however we need to apply optimization techniques –In order to determine the parameter vector such that the resulting model describes the system appropriately:

10 Block diagram for parameter identification System Identification: Introduction

11 System Identification: Introduction  The data set composed of m desired input-output pairs –(u i, y i ) (i = 1,…,m) is called the training data  System identification needs to do both structure & parameter identification repeatedly until satisfactory model is found

12 System Identification: Steps –Specify & parameterize a class of mathematical models representing the system to be identified –Perform parameter identification to choose the parameters that best fit the training data set –Conduct validation set to see if the model identified responds correctly to an unseen data set –Terminate the procedure once the results of the validation test are satisfactory. Otherwise, another class of model is selected & repeat step 2 to 4

13  Least-Squares Estimators

14 Least-Squares Estimators  General form: y =  1 f 1 (u) +  2 f 2 (u) + … +  n f n (u) (14) where: –u = (u 1, …, u p ) T is the model input vector –f 1, …, f n are known functions of u –  1, …,  n are unknown parameters to be estimated

15 Least-Squares Estimators  The task of fitting data using a linear model is referred to as linear regression where: –u = (u 1, …, u p ) T is the input vector –f 1 (u), …, f n (u)regressors –  1, …,  n parameter vector

16 Least-Squares Estimators  We collect a training data set {(u i, y i ), i = 1, …, m} Equation (14) becomes: Which is equivalent to: A  = y

17 Least-Squares Estimators  Which is equivalent to: A  = y –where A  = y   = A -1 y (solution) m*n matrixn*1 vectorm*1 vector unknown

18 Least-Squares Estimators  We have –m outputs & –n fitting parameters to find  Or m equations & n unknown variables  Usually m is greater than n

19 Least-Squares Estimators  Since the model is just an approximation of the target system & the data observed might be corrupted, therefore  an exact solution is not always possible!  To overcome this inherent conceptual problem, an error vector e is added to compensate A  + e = y

20 Least-Squares Estimators  Our goal consists now of finding that reduces the errors between and  The problem: find, estimate

21 Least-Squares Estimators  If e = y - A  then: We need to compute:

22 Least-Squares Estimators  Theorem [least-squares estimator] The squared error is minimized when  satisfies the normal equation if is nonsingular, is unique & is given by is called the least-squares estimators, LSE

23  Statistical Properties of least-squares estimators

24 Statistical qualities of LSE  Definition [unbiased estimator] An estimator of the parameter  is unbiased if where E[.] is the statistical expectation

25 Statistical qualities of LSE  Definition [minimal variance] –An estimator is a minimum variance estimator if for any other estimator  *: where cov(  ) is the covariance matrix of the random vector 

26 Statistical qualities of LSE  Theorem [Gauss-Markov]: –Gauss-Markov conditions: The error vector e is a vector of m uncorrelated random variables, each with zero mean & the same variance  2. This means that:

27 Statistical qualities of LSE  Theorem [Gauss-Markov]: –Gauss-Markov conditions: The error vector e is a vector of m uncorrelated random variables, each with zero mean & the same variance  2. This means that:

28 Statistical qualities of LSE  Theorem [Gauss-Markov] LSE is unbiased & has minimum variance. Proof:

29  Maximum likelihood (ML) estimator

30 Maximum likelihood (ML) estimator –ML is one of the most widely used technique for parameter estimation of a statistical distribution  ML definition: –For a sample of n observations (of a probability density function ) x 1, x 2, …, x n, the likelihood function L is defined by:

31 Maximum likelihood (ML) estimator  The criterion for choosing  is: –“pick a value of  that provides a high probability of obtaining the actual observed data x 1, x 2, …, x n ” –Therefore, ML estimator is defined as the value of  which maximizes L: or equivalently:

32 Maximum likelihood (ML) estimator  Example: ML estimation for normal distribution –For m observations x 1, x 2, …, x m, we have:

33 Maximum likelihood (ML) estimator  Example: ML estimation for normal distribution –For m observations x 1, x 2, …, x m, we have:

34 Statistical Properties & the ML Estimator  Equivalence between LSE & MLE  Theorem –Under the Gauss conditions and if each component of the vector e follows a normal distribution then: LSE of  = MLE of 

35 Statistical Properties & the Maximum Likelihood Estimator (5.7) (cont.)  Maximum likelihood (ML) estimator (cont.) –Equivalence between LSE & MLE –Theorem: Under the Gauss conditions & if each component of the vector e follows a normal distribution then the LSE of  = MLE of 

36  LSE for Nonlinear Models

37 LSE for Nonlinear Models  Nonlinear models are divided into 2 families –Intrinsically linear –Intrinsically nonlinear Through appropriate transformations of the input- output variables & fitting parameters, an intrinsically linear model can become a linear model By this transformation into linear models, LSE can be used to optimize the unknown parameters

38 LSE for Nonlinear Models  Examples of intrinsically linear systems