1 Introduction to Bio-Inspired Models During the last three decades, several efficient machine learning tools have been inspired in biology and nature: Artificial Neural Networks (ANN) are inspired in the brain to automatically learn and generalize (model) observed data. Evolutive and genetic algorithms offer a solution to standard optimization problems when no much information about the function to optimize is available. Artificial ant colonies offer an alternative solution for optimization problems. All these methods share some common properties: They are inspired in nature (not in human logical reasoning). They are automatic (no human intervention) and nonlinear. They provide efficient solutions to some hard NP-problems.

2 Introduction to Artificial Neural Networks Artificial Neural Networks are inspired in the structure and functioning of the brain, which is a collection of interconnected neurons (the simplest computing elements performing information processing): Each neuron consists of a cell body, that contains a cell nucleus. There are number of fibers, called dendrites, and a single long fiber called axon branching out from the cell body. The axon connects one neuron to others (through the dendrites). The connecting junction is called synapse.

3 Functioning of a “Neuron” The synapses releases chemical transmitter substances. The chemical substances enter the dendrite, raising or lowering the electrical potential of the cell body. When the potential reaches a threshold, an electric pulse or action potential is sent down to the axon affecting other neurons. (Therefore, there is a nonlinear activation). Excitatory and inhibitory synapses. nonlinear activation function neuron potential: mixed input of neighboring neurons weights (+ or -, excitatory or inhibitory) (threshold)

4 The neural activity (output) is given by a no linear function. 1. Init the neural weight with random values 2. Select the input and output data and train it 3. Compute the error associate with the output 4. Compute the error associate with the hidden neurons 5. Compute and update the neural weight according to these values Gradient descent Inputs Outputs Multilayer perceptron. Backpropagation algorithm

5 Time Series Modeling and Forecast Sometimes the chaotic time series have a stochastic look difficult to predict An example is Henon map

6 Given a time series with 2000 points (T=20), generated from a Lorenz system (chaotic behavior). To check modeling power different parameters are tested. Example: Supervised Fitting and Prediction Three variables (x,y,z) (x n,y n,z n ) (x n+1,y n+1,z n+1 ) Continuous System Neural Network 3:k:3 h 1 h 2 h k y 1 y i x 2 x 3 x j W ik w kj xnxn ynyn znzn x n+1 y n+1 z i z n+1 3:6:3 3:15:3

7 Dynamical Behavior A simple model doesn’t capture the complete structure of the system, then the dynamics of the system is not reproduce. 3:6:3 3:15:3 A complex system it’s overfitting the problem and the dynamics of the system is not reproduce Only a intermediate model with an appropriate amount of parameters can model the functional structure of the system and the dynamics

8 Time series from a infrared laser. X i-1 X i-2 X i-3 X i-j XiXi The Neural network reproduces laser behavior Infrared laser intensity is modeled using a neural network. Only time lagged intensities are used. Net 6:5:5:1 The Neural Network can be synchronized with the time series obtained from the laser.

9 Structural Learning: Modular Neural Networks With the aim of giving some flexibility to the network topology, modular neural networks combine different neural blocks into a global topology. Fully-connected topology (too many parameters). Combining several blocks (parameter reduction). Assigning different subnets to specific tasks we can simplify the complexity of the model. 2*4+4*4+4*1+9= 37 weights 2(2*2)+2(2*2)+4*1+9= 29 weights In most of the cases, block division is a heuristic task !!! How to obtain an optimal “block division” for a given problem ?

10 Functional Networks Functional networks are a generalization of neural networks which combine both qualitative domain knowledge and data. y f f x z u f + f -1 y I F x z u F I F F Qualitative knowledge:x 3 =F(x 1,x 2 ), Initial Topology Theorem. The simplest functional form is: Simplified Topology Learning (least squares): {    n } {a 1,..., a n } This is the optimal “block division” for this problem !!! Data:(x 1i,x 2i,x 3i ),i=1,2,...

11 Some FN Architectures Associative Model: F(x,y) is an associative operator. Sliced-Conditioned Model: where  and  are covenient basis for the x- and y-constant slices. Separable Model: A simple topology.

12 A First Example. Functional vs Neural Neural Network Functional Network ( separable model) 2:2:2:1 MLP15 parametersRMSE= :3:3:1 MLP25 parametersRMSE= parametersRMSE=  = {1,x,x 2,x 3 } Knowledge of the network structure (separable). Non-parametric approach to learn the neuron functions !!!! 100 points of Training Data with Uniform Noise in (-0.01,0.01). 25x25 points from the exact surface for Validation. Appropriate family of functions (polynomial).

13 Functional Nets & Modular Neural Nets Advantages and shortcomings of Black-box topology with no problem connection. Efficient non-parametric models for approximating functions. Neural Nets Parametric learning techniques (supply basis functions). Model driven optimal topology. Functional Nets The topology of the network is obtained from the Functional network. The neuron functions are Approximated using MLPs. Hybrid functional-neural networks (Modular networks)

14 Another example. Nonlinear Time Series Nonlinear Maps (the Lozi model) Sensitivity to initial conditions Fractal geometry Time series modeling and forecasting is an important problem with many practical applications. Goal: predicting the future using past values. x 1, x 2,…, x n ¿¿¿ x n+1 ??? Modeling methods: x n+1 = F ( x 1, x 2,…, x n) Nonlinear time series may exhibit complex seemingly stochastic behavior. There are many well-known techniques for linear time series (ARMA, etc.). Nonlinear time series modeling is a difficult task because: Trajectories starting at very close initial points split away after a few iterates. Evolve in a irregular fractal space. X 1 =0.8 X 1 = X 1 =

15 Functional Models (separation) 500 training points 1000 validation points. FN MFNN 1 MFNN 2 Separable Functional Net: ¿which basis family?  ={sin(x),…,sin(mx), cos(x),…,cos(mx)} With 4*m parameters Symmetric Modular Functional Neural Net: 1:m:1 With 6*m parameters Asymmetric Modular Functional Neural Net: 1:2m:1 1:2:1 With 2*m-2 parameters m=11 (44 pars) RMSE=5.3e-3 m=7 (42 pars) RMSE=1.5e-3 m=7 (42 pars) RMSE=4.0e-4

16 Minimum Description Length The Minimum Description Length (MDL) algorithm has proved to be simple and efficient in several problems about Model Selection. Description Length for a model