Hypercubes and Neural Networks bill wolfe 10/23/2005

Modeling

Simple Neural Model a i Activation e i External input w ij Connection Strength Assume: w ij = w ji (“symmetric” network)  W = (w ij ) is a symmetric matrix

Net Input Vector Format:

Dynamics Basic idea:

Energy

Lower Energy da/dt = net = -grad(E)  seeks lower energy

Problem: Divergence

A Fix: Saturation

Keeps the activation vector inside the hypercube boundaries Encourages convergence to corners

Summary: The Neural Model a i Activation e i External Input w ij Connection Strength W (w ij = w ji ) Symmetric

Example: Inhibitory Networks Completely inhibitory –wij = -1 for all i,j –k-winner Inhibitory Grid –neighborhood inhibition

Traveling Salesman Problem Classic combinatorial optimization problem Find the shortest “tour” through n cities n!/2n distinct tours

TSP solution for 15,000 cities in Germany

TSP 50 City Example

Random

Nearest-City

2-OPT

An Effective Heuristic for the Traveling Salesman Problem S. Lin and B. W. Kernighan Operations Research, 1973

Centroid

Monotonic

Neural Network Approach neuron

Tours – Permutation Matrices tour: CDBA permutation matrices correspond to the “feasible” states.

Not Allowed

Only one city per time stop Only one time stop per city  Inhibitory rows and columns inhibitory

Distance Connections: Inhibit the neighboring cities in proportion to their distances.

putting it all together:

Research Questions Which architecture is best? Does the network produce: –feasible solutions? –high quality solutions? –optimal solutions? How do the initial activations affect network performance? Is the network similar to “nearest city” or any other traditional heuristic? How does the particular city configuration affect network performance? Is there a better way to understand the nonlinear dynamics?

typical state of the network before convergence

“Fuzzy Readout”

Neural Activations Fuzzy Tour Initial Phase

Neural ActivationsFuzzy Tour Monotonic Phase

Neural ActivationsFuzzy Tour Nearest-City Phase

Fuzzy Tour Lengths tour length iteration

Average Results for n=10 to n=70 cities (50 random runs per n) # cities

DEMO 2 Applet by Darrell Long

Conclusions Neurons stimulate intriguing computational models. The models are complex, nonlinear, and difficult to analyze. The interaction of many simple processing units is difficult to visualize. The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic. Much work to be done to understand these models.

3 Neuron Example

Brain State:

“Thinking”

Binary Model a j = 0 or 1 Neurons are either “on” or “off”

Binary Stability a j = 1 and Net j >=0 Or a j = 0 and Net j <=0

Hypercubes

4-Cube

5-Cube

Hypercube Computer Game

2-Cube Adjacency Matrix: Hypercube Graph

Recursive Definition

Theorem 1: If v is an eigenvector of Q n-1 with eigenvalue x then the concatenated vectors [v,v] and [v,-v] are eigenvectors of Q n with eigenvalues x+1 and x-1 respectively. Eigenvectors of the Adjacency Matrix

Proof

Generating Eigenvectors and Eigenvalues

Walsh Functions for n=1, 2, 3

eigenvectorbinary number

n=3

Theorem 3: Let k be the number of +1 choices in the recursive construction of the eigenvectors of the n-cube. Then for k not equal to n each Walsh state has 2 n-k-1 non adjacent subcubes of dimension k that are labeled +1 on their vertices, and 2 n-k-1 non adjacent subcubes of dimension k that are labeled -1 on their vertices. If k = n then all the vertices are labeled +1. (Note: Here, "non adjacent" means the subcubes do not share any edges or vertices and there are no edges between the subcubes).

n=5, k= 3n=5, k= 2

Inhibitory Hypercube

Theorem 5: Each Walsh state with positive, zero, or negative eigenvalue is an unstable, weakly stable, or strongly stable state of the inhibitory hypercube network, respectively.