1. Neural Networks for Optimization Bill Wolfe California State University Channel Islands

3. Neural Models Simple processing units Lots of them Highly interconnected Exchange excitatory and inhibitory signals Variety of connection architectures/strengths “Learning”: changes in connection strengths “Knowledge”: connection architecture No central processor: distributed processing

4. Simple Neural Model a i Activation e i External input w ij Connection Strength Assume: w ij = w ji (“symmetric” network)  W = (w ij ) is a symmetric matrix

5. Net Input Vector Format:

6. Dynamics Basic idea:

7. Energy

9. Lower Energy da/dt = net = -grad(E)  seeks lower energy

10. Problem: Divergence

11. A Fix: Saturation

12. Keeps the activation vector inside the hypercube boundaries Encourages convergence to corners

13. Summary: The Neural Model a i Activation e i External Input w ij Connection Strength W (w ij = w ji ) Symmetric

14. Example: Inhibitory Networks Completely inhibitory –wij = -1 for all i,j –k-winner Inhibitory Grid –neighborhood inhibition

15. Traveling Salesman Problem Classic combinatorial optimization problem Find the shortest “tour” through n cities n!/2n distinct tours

16. TSP solution for 15,000 cities in Germany

17. TSP 50 City Example

18. Random

19. Nearest-City

20. 2-OPT

21. http://www.jstor.org/view/0030364x/ap010105/01a00060/0 An Effective Heuristic for the Traveling Salesman Problem S. Lin and B. W. Kernighan Operations Research, 1973

22. Centroid

23. Monotonic

24. Neural Network Approach neuron

25. Tours – Permutation Matrices tour: CDBA permutation matrices correspond to the “feasible” states.

26. Not Allowed

27. Only one city per time stop Only one time stop per city  Inhibitory rows and columns inhibitory

28. Distance Connections: Inhibit the neighboring cities in proportion to their distances.

29. putting it all together:

30. Research Questions Which architecture is best? Does the network produce: –feasible solutions? –high quality solutions? –optimal solutions? How do the initial activations affect network performance? Is the network similar to “nearest city” or any other traditional heuristic? How does the particular city configuration affect network performance? Is there a better way to understand the nonlinear dynamics?

31. typical state of the network before convergence

32. “Fuzzy Readout”

33. Neural Activations Fuzzy Tour Initial Phase

35. Neural ActivationsFuzzy Tour Monotonic Phase

36. Neural ActivationsFuzzy Tour Nearest-City Phase

37. Fuzzy Tour Lengths tour length iteration

38. Average Results for n=10 to n=70 cities (50 random runs per n) # cities

39. DEMO 2 Applet by Darrell Long http://hawk.cs.csuci.edu/william.wolfe/TSP001/TSP1.html

40. Conclusions Neurons stimulate intriguing computational models. The models are complex, nonlinear, and difficult to analyze. The interaction of many simple processing units is difficult to visualize. The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic. Much work to be done to understand these models.

41. EXTRA SLIDES

42. E = -1/2 { ∑ i ∑ x ∑ j ∑ y a ix a jy w ixjy } = -1/2 { ∑ i ∑ x ∑ y (- d(x,y)) a ix ( a i+1 y + a i-1 y ) + ∑ i ∑ x ∑ j (-1/n) a ix a jx + ∑ i ∑ x ∑ y (-1/n) a ix a iy + ∑ i ∑ x ∑ j ∑ y (1/n 2 ) a ix a jy }

43. w ix jy = 1/n 2 - 1/ny = x, j ≠ i; (row) 1/n 2 - 1/ny ≠ x, j = i; (column) 1/n 2 - 2/ny = x, j = i; (self) 1/n 2 - d(x, y)y ≠ x, j = i +1, or j = i - 1. (distance ) 1/n 2 j ≠ i-1, i, i+1, and y ≠ x; (global )

44. Brain Approximately 10 10 neurons Neurons are relatively simple Approximately 10 4 fan out No central processor Neurons communicate via excitatory and inhibitory signals Learning is associated with modifications of connection strengths between neurons

45. Fuzzy Tour Lengths iteration tour length

46. Average Results for n=10 to n=70 cities (50 random runs per n) # cities tour length

48. with external input e = 1/2

49. Perfect K-winner Performance: e = k-1/2