Neural Networks for Optimization William J. Wolfe California State University Channel Islands

Top 10 best jobs MONEY Magazine and Salary.com researched hundreds of jobs, considering their growth, pay, stress-levels and other factors. These careers ranked highest: 1. Software Engineer6. Market research analyst 2. College professor7. Computer IT analyst 3. Financial adviser8. Real Estate Appraiser 4. Human Resources Manager9. Pharmacist 5. Physician assistant10. Psychologist By Tara Kalwarski, Daphne Mosher, Janet Paskin and Donna Rosato

Neural Models Simple processing units, and lots of them Highly interconnected Variety of connection architectures/strengths Exchange excitatory and inhibitory signals Learning: changes in connection strengths Knowledge: connection strengths/architecture No central processor: distributed processing

Simple Neural Model a i Activation e i External input w ij Connection Strength Assume: w ij = w ji (“symmetric” network)  W = (w ij ) is a symmetric matrix

Net Input

Dynamics Basic idea:

Energy

Lower Energy da/dt = net = -grad(E)  seeks lower energy

Problem: Divergence

A Fix: Saturation

Keeps the activation vector inside the hypercube boundaries Encourages convergence to corners

Summary: The Neural Model a i Activation e i External Input w ij Connection Strength W (w ij = w ji ) Symmetric

Example: Inhibitory Networks Completely inhibitory –wij = -1 for all i,j –k-winner Inhibitory Grid –neighborhood inhibition

Traveling Salesman Problem Classic combinatorial optimization problem Find the shortest “tour” through n cities n!/2n distinct tours

Neural Network Approach

Tours – Permutation Matrices tour: CDBA permutation matrices correspond to the “feasible” states.

Not Allowed

Only one city per time stop Only one time stop per city  Inhibitory rows and columns

Distance Connections: Inhibit the neighboring cities in proportion to their distances.

putting it all together:

R n 2 = F 0  E c  E r  D aix proj = aix + act avg - rowx avg - coli avg Feasible Solutions

E = -1/2 { ∑ i ∑ x ∑ j ∑ y a ix a jy w ixjy } = -1/2 { ∑ i ∑ x ∑ y (- d(x,y)) a ix ( a i+1 y + a i-1 y ) + ∑ i ∑ x ∑ j (-1/n) a ix a jx + ∑ i ∑ x ∑ y (-1/n) a ix a iy + ∑ i ∑ x ∑ j ∑ y (1/n 2 ) a ix a jy }

Research Questions Which architecture is best? Does the network produce: –feasible solutions? –high quality solutions? –optimal solutions? How do the initial activations affect network performance? Is the network similar to “nearest city” or any other traditional heuristic? How does the particular city configuration affect network performance? Is there any way to understand the nonlinear dynamics? References: –Neural Networks for Combinatorial Optimization: A Review of More Than a Decade of Research. Kate A. Smith, Informs Journal on Computing, Vol. 11, No. 1, Winter –An Analytical Framework for Optimization Problems. A. Gee, S. V. B. Aiyer, R. Prager, 1993, Neural Networks 6, –Neural Computation of Decisions in Optimization Problems. J. J. Hopfield, D. W. Tank, Biol. Cybern. 52, (1985).

typical state of the network before convergence

“Fuzzy Readout”

DEMO 1

Fuzzy Tour Lengths Tour Length Iteration

DEMO 2 Applet by Darrell Long

EXTRA SLIDES

Brain Approximately neurons Neurons are relatively simple Approximately 10 4 fan out No central processor Neurons communicate via excitatory and inhibitory signals Learning is associated with modifications of connection strengths between neurons

with external input e = 1/2

Perfect K-winner Performance: e = k-1/2