Identification of Wiener models using support vector regression Stefan Tötterman and Hannu Toivonen Process Control Laboratory Åbo Akademi University Finland 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Wiener models Output error identification The dynamic linear part F consist of an orthonormal filter The static nonlinear part N consists of a support vector model 17.2.2019 Process Control Laboratory - Åbo Akademi University

 - insensitive loss function y: observation yest: estimated function y y-yest x 17.2.2019 Process Control Laboratory - Åbo Akademi University

Support Vector Regression A set of training data: A set of basis functions: Estimation of y is expanded in basis functions Minimization of L and norm of the weight (smoothness, robustness) where w is a weight parameter C is a weight 17.2.2019 Process Control Laboratory - Åbo Akademi University

Support Vector Regression The optimization problem is transformed to a dual convex optimization problem and the approximation function is given by Most of the factors (αi - αi’) will be zero, the input vectors corresponding to the nonzero factors forms the so-called support vectors (correspond to observations outside the ε-tube) K(xi,xj) is the inner-product kernel, commonly RBF Lagrange multipliers 17.2.2019 Process Control Laboratory - Åbo Akademi University

Support Vector Regression SVMs can be seen as a network, where all the important network parameters are computed automatically. bias, b K(x,x1) x1 (1-1’) yest K(x,x2) (2-2’)  x2 SV Most of the weights (i-’i) will be zero, the other will define the support vectors (m1-m1’) K(x,xm1) xN Input layer RBF with centers x1,...,xm1 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Some properties of SVR No need to compute (xi), enough to compute the kernel values directly (kernel trick). Convex optimization. Robust algorithm when using L. Optimal model complexity is obtained automatically as a part of the solution. Efficient optimization methods exist (high memory requirements). Hard to involve prior knowledge about the task.  and C must be chosen simultaneously by the user. 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Dynamic linear part Introducing orthonormal filters to the dynamic linear part have been found useful. Usually Laguerre or Kautz filter-types are used. Laguerre filters with a single real-valued pole are well suited for modelling well damped systems. Kautz filters with a pair of complex-valued poles are suitable for systems which have oscillatory behaviour. 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Dynamic linear part Laguerre filters q-1 is the backward-shift operator and ||  1. Outputs are calculated for k = 1, 2, ..., l where l is the filter order. 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Dynamic linear part The filter output xk can be derived from the previous filter output xk-1 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Wiener models General Wiener model Wiener model in this identification method 17.2.2019 Process Control Laboratory - Åbo Akademi University

Identification of Wiener models The identified systems dynamics are unknown Design parameters: Dynamic linear part:  (filter pole) l (filter order) Static nonlinear part:  (insensitivity margin) C (weight) γ (RBF kernel) 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model* The input u(t) is a pneumatic control signal The output y(t) is a flow through a valve The simulated model is described by the following equations e(t) is white gaussian measurement noise, standard deviation 0.05 *T. Wigren, Recursive prediction error identification using the nonlinear Wiener model, Automatica 29(4) (1993) *A Hagenblad, Aspects of the Identification of Wiener Models, Linköping Studies in Science and Technology, Thesis No. 793, 1999 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model Training data 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model Test data 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model Laguerre filter of order l = 5 and with the pole  = 0.4 was found to be a proper choice Optimal SVR parameters γ = 0.1  = 0.08 C = 2000 This choice of parameters results in a model consisting of 146 support vectors RMSE 0.0541 (train) RMSE 0.0556 (test) 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model Last 100 samples of the test data set Measured output (solid) Model output (dashed) Noisefree output (dotted) 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model Output errors (test data) y-ŷ yNF-ŷ Samples Filter SVR RMSE #SV train test l   C γ y-ŷ yNF- ŷ 5 0.4 0.08 2000 0.1 0.0541 0.0218 0.0556 0.0191 146 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model Laguerre filter parmeter sensitivity table Filter SVR RMSE #SV train test l   C γ y-ŷ yNF- ŷ 3 0.4 0.08 2000 0.1 0.0697 0.0468 0.0680 0.0427 212 4 0.0564 0.0261 0.0577 0.0239 165 5 0.0541 0.0218 0.0556 0.0191 146 6 0.0540 0.0242 0.0559 0.0202 161 7 0.0532 0.0250 0.0565 0.0222 168 0.2 0.0623 0.0379 0.0634 0.0356 194 0.3 0.0561 0.0256 0.0575 0.0232 160 0.5 0.0233 0.0217 152 0.7 0.0528 176 17.2.2019 Process Control Laboratory - Åbo Akademi University

Example – Control valve model SVR parmeter sensitivity table Filter SVR RMSE #SV train test l   C γ y-ŷ yNF- ŷ 5 0.4 0.04 1000 0.1 0.0540 0.0204 0.0545 0.0170 477 2000 0.0531 0.0191 0.0542 0.0162 478 4000 0.0524 0.0182 0.0155 476 0.08 0.0548 0.0223 0.0560 0.0197 148 0.0539 0.0222 0.0198 150 20000 0.0538 0.0240 0.0571 0.0229 156 0.05 0.0570 0.0252 0.0566 0.0215 149 0.0541 0.0218 0.0556 146 0.2 0.0537 0.0236 0.0567 0.12 0.0579 0.0266 0.0588 0.0253 45 0.0573 0.0263 0.0589 0.0256 41 0.0572 0.0274 0.0595 0.0265 43 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University Conclusions This identification method works well for Wiener model identification and gives accurate models The model is determined by solving a convex quadratic minimization problem (global optimum is always obtained) Robust performance w.r.t. new data is achieved since SVR is based on structural risk minimization It is straightforward to extend this method to MIMO systems 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University 17.2.2019 Process Control Laboratory - Åbo Akademi University

Process Control Laboratory - Åbo Akademi University 17.2.2019 Process Control Laboratory - Åbo Akademi University