1. Regularization of Evolving Polynomial Models
IWIM 2007 workshop, Sept , 2007, Prague
Regularization of Evolving Polynomial Models
Pavel Kordík
CTU Prague, Faculty of Electrical Engineering, Department of Computer Science

2. External criteria in GMDH theory

3. GAME model

4. Encoding into chromosomes

5. Polynomial units encoded into neurons

6. Data set division Adaptive division? 2/3 1/3 A B Training data
Validation
data
Testing data
Optimize coefficients
(learning of units)
Select surviving
units
Check if model
overfits data

7. Fitness function – which units survive?
RMS Error of the unit on the training data – feedback for optimization methods
RMS Error of the unit on the validation data – used to compute fitness of units

8. External criterion Computation of error on validation set
Fitness = 1/CR

9. CRrms-r-val criterion on real data
Optimal value of R is 300 on the Antro data set

10. CRrms-r-val criterion on real data
Optimal value of R is 725 on the Building data set

11. CR should be sensitive to noise

12. How to estimate the penalization strength (1/R)?
Variance of the output variable?

13. Experiments with synthetic data
Validate just on validation set!
Regularization works, but the
difference of R300 and p-n is not
significant.

14. Theoretical and experimental aspects of regularization

15. So which criterion is the best?

16. Regularized polynomial models on Antro data set
It is evident that optimal value of R is between 100 and 1000 – the same results as in our pervious experiments with Antro data set (Ropt=300).
Linear models are still better than the best polynomial !!!

17. Conclusion Experiments with regularization of polynomial models
Every data set requires different level of penalization for complexity
It can be partially derived from the variance of the output variable
The regularization is still not sufficient, linear models perform better on highly noisy data sets!

18. Thank you!