Helsinki University of Technology Adaptive Informatics Research Centre Finland Variational Bayesian Approach for Nonlinear Identification and Control Matti Tornio and Tapani Raiko October 9, 2006

Introduction System identification and control in nonlinear state-space models Continues the work by Rosenqvist and Karlström (Automatica 2005) Our background is in machine learning Uncertainties taken explicitly into account by using Variational Bayesian treatment

Why nonlinear state-space models? System identification using a hidden state has many benefits: More resistant to noise Observations (without history) do not always carry enough information about the system state Finds a representation of the state that is more suitable for approximating the dynamics

System identification in nonlinear state-space models We use a state-of-the-art tool by Valpola and Karhunen (Neural Computation 2002) Parameters, states, and observations are modelled with Gaussian distributions Less prone to overfitting (than the prediction error method) Can determine the dimensionality of the state space etc.

Properties of the method The model scales well to higher dimensions Can model very complex dynamics Natural conjugate gradient algorithm is used for fast system identification

Nonlinearities by MLP networks f(x(t),θ)=B tanh[Ax(t)+a] + b + noise The parameters θ include the weight matrices, bias vectors, noise variances etc. Note that the policy mapping does not fix the control signal (because of the noise model)

Variational Bayesian treatment Posterior probability p(x,θ|y) is approximated by q(x,θ) q is assumed to be Gaussian with limited dependencies The fit of q to p is measured by a cost function Both identification and prediction can be done by minimising the misfit by adjusting the parameters defining q (means, variances, covariances)

Control Current state is estimated with extended Kalman filter (EKF) Control signals u(t) are selected to minimise the expected cost E{J} over the distribution q Quasi-Newton algorithm for optimisation Compare to dual control: estimation errors increase the expected cost

Control (cont.) Prediction with variances is ~5 times slower  too slow for some applications, the method can still be used for system identification Learning is done offline  online learning possible as well, leads to different exploration strategies

Optimistic inference control Alternative control scheme Observations at some point in the future are fixed and the states leading to this desired future are inferred Allows the direct use of inference algorithms Conceptually very simple, but not as versatile as NMPC  constraints hard to model

Experiments Assume the cart-pole system to be unknown Dynamics are identified from only 2500 samples 6-dimensional state space x(t) was used

Results Very high success rate was reached even under high noise Partially observed system is hard to control Low noiseHigh noise

Results (initialisation) Good initialisations are important  Local minima are the biggest problem Internal forward model can provide reasonable initialisations without significant extra computation

Conclusion Learning nonlinear state-space models seems promising when  observations about the system state are incomplete or  the dynamics of the system are not well known Variational Bayesian treatment helps  against overfitting  to determine the model order