1. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group TLTE.3120 Computer Simulation in Communication and Systems (5 ECTS) http://www.uva.fi/~timan/tlte3120/ Lecture 6 – 14.10.2015 Timo Mantere Professor, Communications & systems University of Vaasa http://www.uva.fi/~timan timan@uva.fi 1

2. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Outline  Some  Systems theory  Systems engineering  System identification  Parameter estimation  Probability calculus  Regression analysis  Dynamic models  Stochastics 2

3. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Systems theory 3 Systems theory is a cross-disciplinary field where aim to find those principles that can be applied to all types of systems in different areas of research. The term can be regarded as systems thinking as a special case and a generalization of the general system of science, which forms a systemic perspective. The concept of systems theory has its origins in Ludwig von Bertalanff’s general systems theory. It has been applied later in other areas such as the theory of functions and social systems theory. More limited concept of systems theory is linked to the analysis, design and control of various systems based on the use of mathematical models that describe the system variables in the cause-effect relationships and interactions.

4. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Systems engineering 4 Systems engineering is an interdisciplinary field of technology, which focuses on how to plan and manage complex technical systems over their life cycle. It handles questions such as requirements specification, reliability, logistics, coordination of the various engineering teams, testing and evaluation, maintainability, and many other disciplines necessary for successful system development, design, execution, and the finally decommission of product. Systems engineering deals with the work processes, optimization methods and risk management tools in different technical projects.

5. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Systems engineering 5 It is combination of technical and human oriented disciplines such as control engineering, industrial management, software engineering, organizational studies, and project management. Designing of systems to ensure that all aspects of the project are likely to be taken account and a system is considered and integrated into the whole. In product development, the first and most important task is to identify, understand and interpret the operational demands of the new product and technical limitations. In general, it is not enough that the product will work, but it must also meet several other requirements, e.g. one need take account also future demands for flexibility with regard for modifications and additions, and other factors such as product cost, manufacturability, usability and serviceability.

6. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Systems engineering 6 While searching for the proper solution to implement the list of requirements for new product designers take advantage of their knowledge of the technology, mathematics, and their experience, and the analysis of the problem. By creating a mathematical model of the problem in hand, different solutions can be tested. Generally, there is always several viable solutions, so the designers have to evaluate the different choices with their knowledge and to choose the most suitable solution. If a designer is talented (s)he will find the best solution among the possible solutions. Sometimes innovative designer may find a solution that completely removes the need for new product i.e. new way of doing things.

7. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System identification 7 System identification uses different statistical methods to build mathematical model of the system based on the measurement done from the system The field of system identification include the design of experiments which efficiently extracts informative information from the system. Based on the measurements one can do different kind of data analysis, e.g. correlation analysis, curve fittings and model fittings. System identification has some relations to the data analysis, i.e. in this case we try identify from the data which kind of system has produced that data These days we are also talking about Big data and Cloud computing which can also do data modelling

8. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System identification 8 Internet of things These days IoT creates a huge amount of sensory etc. data which can be analyzed with cloud computing Many companies wants to utilize that data in order to sell their products or designing new products based on that data, i.e direct marketing etc. Quite often the biggest problem at the moment is that many companies do not have, or are unable to build “earnings logic”, i.e. how to make money from that that data or data analyses.

9. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System Identification 9 To better understand what system identification means, see additional information e.g. from: System identification: http://en.wikipedia.org/wiki/System_identification Curve fitting: http://en.wikipedia.org/wiki/Curve_fitting Nonlinear regression: http://en.wikipedia.org/wiki/Nonlinear_regression

10. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System Identification - Introduction  In continuous and discrete time modeling the model was formed based on the physical knowledge about the system and then simulated  Differential equation, state-space presentation and transfer function in continuous time  Difference equation, discrete state-space presentation and discrete transfer function in discrete time 10

11. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System Identification - Introduction  Open Box: Model structure and parameters are known based on system structure and on the laws of the physics  Gray Box: We know the model structure, but we must define the parameter values by using measurements  Black Box: The measurement data is the only information we have. Both model structure and model parameters must be defined by using the measurement data. 11

12. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System Identification - Introduction  In the case of system identification we usually have just input-output data and we must identify a feasible model for the system by using it ? u(t)y(t) 12

13. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group System Identification - Introduction System identification is an iterative process which combines the estimation of parameters and the estimation of model structure 1) Defining the model structure -Basic variables and their mutual dependencies -Linear or nonlinear, model degree 2) Parameter estimation -Once the model structure is identified, we must seek such values for the model parameters that the model fits to the input-output data as well as possible 3) Model validation -Generalizability: the model must fit to the system input-output data, not just to the training data set -Valid area: in which range the model variables can vary -System stability 13

14. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Parameter Estimation  Fitting the parameter values to the data  Optimization problem: fitting error must be minimized  One can see the statistical properties of the system from the data  Expectation value, variance, etc.  Usually the parameters are identified by using computation program  MATLAB Identification Toolbox 14

15. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Parameter Estimation  In practise the measurements are always noisy  One must know some identification theory to be able to use the computation programs correctly ? u(t)y(t) Measured value: u(t) + e{u(t)}Measured value: y(t) + e{y(t)} e{u(t)}e{y(t)} ++ 15

16. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  Discrete distribution is defined just in separate points x i. Respective probability function values are p i.  Continuous distribution is defined in a continuous real axis. Variable value x gives a probability density funvction value p(x).  Estimation theory: http://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Estimation_theory  Propability: http://en.wikipedia.org/wiki/Propabilityhttp://en.wikipedia.org/wiki/Propability 16

17. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  Probability density function fills following two conditions: Continuous:Discrete: 17

18. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment For continuous distribution For discrete distribution 18

19. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  The most important continuous distribution is Gaussian Distribution. A Gaussian distribution with average  and standard deviation σ is noted by N( , σ). Its probability density function is  Gaussian Distribution is often normalized such that its avarage is 0 and standard deviation 1. Normalization is done by applying a transform 19

20. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  The probability density function of Normalized Gaussian Distribution N(0,1) is  It is often assumed that the measurement data is Gaussian distributed. As a consequence it will be fitted to the Gaussian distribution.  Easy to process further  One must be careful because the fitting to the Gaussian distribution will also bias the original data and possibly flush away some important information 20

21. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  Distribution weightpoint is its expectation value: (continuous) (discrete) 21

22. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  Standard deviation describes how much the results spread around the expectation value (continuous) (discrete) 22

23. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Probability Calculus Refreshment  A variance, which is a square of the expectation value, is often applied (continuous) (discrete) 23

24. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Discrete Data  Usually the processed data is discrete. If we have measurements x(k) in time range k = 1,...,n  Expectation value is  Covariance is 24

25. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Discrete Data  Expectation value and covariance in a matrix form: 25

26. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  We select a model structure y = f( ,  ), in which one can measure variables  and one must estimate parameters .  During the parameter estimation we select a criteria J(  ), which will be optimized with respect to the parameters. 26

27. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  Example  Present variable y by using variables x and z :  Select the model structure  Model parameters 27

28. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  Collect a sample set  Select optimization criteria  The optimal solution can be find by minimizing the criteria J 28

29. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  Developed model will be validated by using suitable method. If the model behavior is not satisfactory, the model structure can be modified.  The validation data should be completely different than the training data  There exist always random disturbances in a real-world system. As a consequence the measurement data and the model will never fit completely ion every measurement point. 29

30. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  Assume model structure Real values in the data points: Thus the estimate and the measured value are 30

31. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  There are several options to select the citerion for the optimization. Minimizing the error square sum is one of the most common criterias. 31

32. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  The criterion is minimized with respect to both model parameters by finding the zero points of the partial derivatives: 32

33. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis  Solution of parameters: 33

34. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group An Example of Regression Analysis  A line is fitted to the set of data points ( x i, y i ) in three different cases: i)There is only one data point (2,3), (N = 1) ii)There are two data points, (2, 3) and (-1, 4), (N = 2) iii)There are three data points, (2, 3), (-1, 4) and (0, 3), (N = 3) Matrix is not invertible since its rank is not full => there is no unique solution i) N = 1 34

35. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group An Example of Regression Analysis There exists a preciese unique solution. The modeling errors is zero, because the line goes through both of the data points. ii) N = 2 35

36. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group An Example of Regression Analysis There is a unique solution. The line does not go through any of the three given data points, but it is the best line fit with respect to minumum error squaresum. iii) N = 3 36

37. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group General Regression Model  In the previous example, measurable variables were x and 1 and estimated parameters k and k o. Next we apply general regression model.  Parameters and measurable variables are collected to their own matrices: 37

38. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group General Regression Model where In general, if we have N data points, we will get N equations as follows: 38

39. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group General Regression Model In a matrix form: Minimum square error criterium can now be presented as: 39

40. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group General Regression Model / Pseudoinverse The error becomes Set the error expression to the square error criterium: 40

41. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group General Regression Model / Pseudoinverse The optimal solution is find by solving the zero point of the parameter estimate derivative: 41

42. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Pseudoinverse  A general solution of the minimum least square method, so-called pseudoinverse is  If we want to give different weights to the different points of the measurement data, the weighted error square sum criterion is In Matlab, command pinv computes the pseudoinverse 42

43. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Pseudoinverse  In mathematics pseudoinverse Matrix is a matrix that has some properties of inverse matrix, but not necessarily all of them  The idea of generating pseudoinverse matrix is that it when we have a measured collection of point, they do not present actual values which can be linearly inversed, instead we need some kind of generalized inverse in order analyze what formulas explain those points  It is needed when the function group does not have exact solution, instead we try to find closest possible solution (with the least fitment error)  Eg so that the function based on pseudoinverse causes the minimum square error when compared to those points 43

44. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Weighted Pseudoinverse  A pseudoinverse for weighted minimum least square is  In the case of several variables the linear regression analysis will be done in a same way. The only difference is that we have more measurable variables then. 44

45. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group Regression Analysis / Several Variables  If we have a system we will get a respective matrix presentation (N samples): 45

46. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group TRANSFER FUNCTION - SCALAR G UY Y = G*U 46 LINEAR SYSTEMS - TRANSFER FUNCTIONS

47. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group LINEAR SYSTEMS – MIMO TRANSFER FUNCTIONS  Linear, time-invariant state equation MIMO (MULTI INPUT - MULTI OUTPUT) TRANSFER FUNCTION CONTINUOUS 47

48. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group LINEAR SYSTEMS - TRANSFER FUNCTIONS  Linear, time-invariant state equation TRANSFER FUNCTION DISCRETE 48

49. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  WHITE NOISE  Current state depends neither on history nor future values  Noise spectrum is constant 49

50. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  WHITE NOISE 50

51. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  WHITE NOISE 51

52. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  WHITE NOISE Should be constant - why is it not? 52

53. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  COLOURED (RED) NOISE White noise through first order filter 53

54. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  COLOURED (RED) NOISE 54

55. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group STOCHASTICS  COLOURED (RED) NOISE 55

56. U NIVERSITY of V AASA Communications and Systems Engineering Group U NIVERSITY of V AASA Communications and Systems Engineering Group INTERPOLATION POLYNOMIALS AND SPLINES  Noisy data, repeat curve fitting - Splines Better to filter data first! 56