1. Improved Search for Local Optima in Particle Swarm Optimization May 6, 2015 Huidae Cho Water Resources Engineer, Dewberry Consultants Part-Time Assistant Professor, Kennesaw State University

2. 2 Overview Why Find Local Optima? Isolated-Speciation-based Particle Swarm Optimization (ISPSO) Challenges in Multi-Modal Optimization Stochastic Rainfall Generator Other Applications Conclusions

3. 3 Why Find Local Optima? Traditional model optimization tries to find “ the ” global optimum only. Is the/a global optimum always what we want? We want “ realistic ” solutions. Need a new technique to find many working solutions.

4. 4 Why Find Local Optima? (Cont.) Flood risk model Want to minimize the risk. Finds only the global optimum. If factors A and B are costly? Factor A Factor B Risk

5. 5 Isolated-Speciation-based Particle Swarm Optimization (ISPSO) New optimization method based on Species-based PSO (SPSO)! Implemented in the R language. Runs on multi-platforms: MS-Windows, UNIX Finds local optima as well as the global optimum.

6. 6 ISPSO (Cont.) Particle Swarm Optimization (PSO) PSO is a metaheuristic based on the movement of possible solutions referred to as particles. A swarm consists of multiple particles sharing information with each other. Particles fly through the search space towards optimal solutions.

7. 7 ISPSO (Cont.) Randomness vs. Uniformity

8. 8 ISPSO (Cont.) Particle ’ s Movement ~ v i ( t ) ~ x i ( t )  ~ v i ( t ) x 2 ~ p i x 1 ~ p g ~ x i ( t + 1 ) ~ v i ( t + 1 )  à 1 ~ r 1 ( t )( ~ p i ¡ ~ x i ( t ))  à 2 ~ r 2 ( t )( ~ p g ¡ ~ x i ( t ))  = 2 j 2 ¡ à ¡ p à 2 ¡ 4 à j ; à = à 1 + à 2 > 4 ~ v i ( t + 1 ) =  [ ~ v i ( t ) + à 1 ~ r 1 ( t )( ~ p i ¡ ~ x i ( t )) + à 2 ~ r 2 ( t )( ~ p g ¡ ~ x i ( t ))] ~ x i ( t + 1 ) = ~ x i ( t ) + ~ v i ( t + 1 )

9. 9 ISPSO (Cont.) Isolated Speciation

10. 10 ISPSO (Cont.) Fitness Assimilation

11. 11 ISPSO (Cont.) Preemptive Competitive Nesting

12. 12 ISPSO (Cont.) Flowchart Initial population from Sobol’ sequences Isolated speciation Update velocities Check for nesting criteria Preemptive nesting Update positions Fitness assimilation Stopping criteria? No Yes Start End

13. 13 ISPSO (Cont.) ISPSO vs. SPSO

14. 14 Challenges in Multi-Modal Optimization How to Detect the Findings of Local Optima? SPSO: error between the fitness of the global optimum and the particle ’ s fitness  In real-world problems, in most cases, the fitness of the global optimum unknown. NichePSO: the standard deviation of fitness values over a number of iteration  Cannot guarantee spatial convergence. ISPSO: SD of fitness and the normalized geometric mean of the particle ’ s half-life experience.  Guarantees spatial convergence as well as fitness convergence.

15. 15 Challenges in Multi-Modal Optimization (Cont.) When Failed to Detect Local Optima Correctly? nAnalytically counted by Cho et al. (2008), Numerically confirmed with ISPSO NichePSO by Brits et al. (2007) 195 211125 31,215625 Number of Global and Local Optima in [-28,28] n

16. 16 Challenges in Multi-Modal Optimization (Cont.) When to Stop the Algorithm? Don ’ t want to wait until the maximum number of iterations. As the number of iterations between successive discoveries increases  It becomes more difficult to find more optima.  Possibility to find another optimum decreases.  Certain threshold.

17. 17 Stochastic Rainfall Generator Modified Bartlett-Lewis Rectangular Pulse Model (MBLRP) Stochastically generates synthetic rainfall time series. Replicates statistics of observed rainfall at a given rain gage. Highly multi-modal: Need to find as many feasible solutions as possible.

18. 18 Stochastic Rainfall Generator (Cont.) ISPSO vs. NichePSO: 2D Projection of Solutions

19. 19 Stochastic Rainfall Generator (Cont.) ISPSO vs. NichePSO: Histograms of the Normalized Distance between Solutions and True Optima

20. 20 ISPSO vs. AutoCal built in SWAT Other Applications SWAT Model Calibration Clear Multi-Modality!

21. 21 Other Applications (Cont.) Uncertainty Analysis within the GLUE Framework Multi-modal optimization suitable for equifinality Relative sampling density In a SWAT case study, 4,000 model runs vs. 46,000!

22. 22 References Brits, R., Engelbrecht, A.P., van den Bergh, F., 2007. Locating Multiple Optima Using Particle Swarm Optimization. Applied Mathematics and Computation 189 (2), 1859-1883. Cho, H., Kim, D., Olivera, F., Guikema, S. D., 2011. Enhanced Speciation in Particle Swarm Optimization for Multi-Modal Problems. European Journal of Operational Research 213 (1), 15- 23. Cho, H., Olivera, F., Guikema, S. D., 2008. A Derivation of the Number of Minima of the Griewank Function. Applied Mathematics and Computation 204 (2), 694-701.

23. 23 Conclusions Mathematically best solutions are not always practical and feasible. ISPSO improved SPSO for finding local optima. More reliable criteria for finding solutions and stopping optimization were introduced. ISPSO outperformed SPSO, its predecessor, and NichePSO, another PSO-based multi-modal optimizer. Application to model calibration and GLUE.