1. Learning Chaotic Dynamics from Time Series Data A Recurrent Support Vector Machine Approach Vinay Varadan

2. Primary Motivation  Understand the biological cell as a complex dynamical system  Recent developments allow for in vivo post-translation protein modification measurements along with gene expression levels  Very expensive still, thus forcing only relatively sparse sampling of the modified protein concentrations in time  We invariably measure only a small number of variables of the system - in most cases just one or two variables  Develop modeling techniques to learn underlying dynamics with short time series without knowing the exact structure of the nonlinear differential equation  Even in the absence of noise, trajectory learning is still a difficult problem

3. Problem Statement  Given the time series of one variable in a multidimensional nonlinear differential equation (NDE)  Learn the number of dimensions, viz. number of interacting variables in the underlying NDE  Given a few samples, be able to generate all future samples exactly matching the trajectory of the variable  Do this for all possible NDEs, including ones at the edge of chaos and also chaotic systems  In this project we concentrate on chaotic systems because the rest would be easier to learn, for a given dimensionality

4. Previous Attempts At Chaotic Time Series Prediction  Taken’s delay embedding theorem (1981) – can recreate the geometry of the state-space using just delayed samples of the single observable  Thus for the time series measurement, y(t), y(t) = f(y(t-1), y(t-2), …, y(t-m))  Nonlinear functions with universal approximation capability employed for f such as RBF, polynomial functions, rational functions, local methods  One-step predictors - these methods learn to predict one time step ahead when given past samples of the observable  Not good enough – not learning to follow trajectories of the dynamical system thus not learning the geometry of the state space well  We need to learn Recurrent models

5. Recurrent Models - SVM  Consider learning models of the form  In order to estimate the function f, we use Recurrent Least Squares Support Vector Machines  We can rewrite the above equation in terms of the given data and the error variables as

6. Recurrent Training using SVM  The training of the network is formulated as  The final term of the equation to be minimized refers to the Least Squares formulation  We can now define the Lagrangian and derive the optimality conditions appropriately  Further, we can eliminate the calculation of w explicitly and use just the Kernel formulation

7. Recurrent Training using SVM  The resulting recurrent simulation model is given as  For the Recurrent SVM case, the parameter estimation problem becomes nonconvex  We thus have to use sequential quadratic programming

8. Recurrent Model Performance  Performance of different prediction algorithms on a chaotic Predator-Prey model

9. Conclusion and Pending Work  Recurrent SVM models are able to capture the underlying dynamics much better compared to other models  In the past, we have developed an Improved Least Squares (ILS) formulation for use in modeling chaotic systems  Need to explore how that can be integrated with SVMs