System Identification and Curve Fitting with a Genetic Algorithm Hierarchy Alice E. Smith and Mehmet Gulsen Department of Industrial Engineering University of Pittsburgh INFORMS Fall 1997

Curve Fitting Process of approximating a closed form function to a given data set of independent variables and dependent variable (variable selection, closed form function selection, coefficient estimation). Used for: System identification Judging the strength of relationship Identifying main variables and interaction between variables Interpolate/extrapolate to new data

Conventional Approaches Various regression techniques Time series analysis Spline fitting Neural networks

Genetic Algorithm Hierarchy Function and Variable Selection Upper Module optimized coefficients for functions candidate functions Lower Module Coefficient Estimation

Search Structure Lower GA Search Data n1 n2 n 1 Upper GA Population

Genetic Search Process Top Half Selection Offspring Initial Population Initial Population best (n) Final Population Mutants Uniform Selection Offspring Mutants

Upper GA - Function Selection Explore the possible functional forms that could represent the underlying relationship between independent and dependent variables of a data set Objective Function: Minimize “adjusted” total error corresponding to the functional form. Adjustment is performed by penalizing more complex representations (more variables, higher order terms) Stopping Criteria: Search is terminated when no improvement is observed for a specific number of generations

Upper GA Function Selection - Encoding Tree Structure + * 1 cos

Upper GA Function Selection - Penalty Function Penalty Factor = 0.05 + * 1 cos

Upper GA Function Selection - Crossover Before: Parent 1 Parent 2 + * 1 cos / sin ln crossover After: Offspring 1 Offspring 2

Upper GA Function Selection - Mutation Before: Parent 1 randomly generated tree + + exp + + cos 1 * * mutation * After: Mutant

Lower GA - Coefficient Estimation Estimate the coefficients of a given closed form function which minimize the total error over the set of data points Objective Function: Minimize total squared error Minimize K: number of data points Stopping Criteria: Search is terminated when no improvement is observed for specific number of generations Detailed results are published in “International Journal of Production Research”, Vol. 33, No. 7, 1995

Lower GA Coefficient Estimation - Encoding

Lower GA - Selection/Breeding Parents are selected for breeding uniformly from the superior half of the population The values of the offspring’s coefficients are determined by calculating the arithmetic mean of the corresponding coefficients of two parents Parent A: 45.876 32.958 12.098 -3.892 0.2356 Parent B: 12.988 35.832 0.234 -12.984 2.4576 Offspring: 29.432 34.395 6.166 -8.438 1.3466

Lower GA - Mutation C1 C2 C3 C4 C5 Perturbing existing solutions to explore new regions of search space Perturbation value is obtained by multiplying the current population range with a random factor C1 C2 C3 C4 C5

Test Problem C Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Mean Sd.Dv. 1 9.986 9.998 10.002 10.000 9.996 10.001 9.997 0.005 2 9.999 10.000 10.000 10.000 10.000 10.000 10.000 0.000 3 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 4 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 5 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 6 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 7 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 8 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 9 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 10 10.000 10.000 10.000 10.000 10.000 10.000 10.000 0.000 SE. 0.0017 0.000 0.0000 0.000 0.000 0.000 0.000 -

Different Error Metrics Test Problem Different Error Metrics 1 2 3 4 5 6 7 8 500 1000 Number of Generations Log10 of Squared Error 1500 Squared Error Absolute Error Maximum Error

Test Problem Different Numbers of Data Points

Empirical Data Sets Five benchmark problems from the literature 1. onion growth 2. children growth 3. sunspots 4. chemical plant 5. slip casting Single variable/50 observations to 13 variables/1000 observations Nonlinear regression, time series analysis, model identification

Test Problem 3, Sunspot Data Sunspot data from 1700 to 1995 Highly cyclic with peak and bottom values approximately in every 11.1 years Cycle is not symmetric. The number of counts reaches to maximum value faster than it drops to a minimum Training range: 1700-1979 Validation range: 1980-1995

Functions Identified

Model D

Extrapolation of Model D

Conclusions A unique approach for curve fitting problems Provides closed form function for the given data set Can handle non-linear, discontinuous functions Flexible in terms of error metric Can be used separately for function selection and coefficient optimization Computationally intensive and needs a priori setting of search parameters and penalty function components Forthcoming paper : “A hierarchical genetic algorithm for system identification and curve fitting with a supercomputer implementation,” Mehmet Gulsen and Alice E. Smith, Institute for Mathematics and its Applications, Volumes in Mathematics and its Applications, Volume on Evolutionary Computing.