1. CSC2535: Computation in Neural Networks Lecture 11 Extracting coherent properties by maximizing mutual information across space or time
Geoffrey Hinton

2. The aims of unsupervised learning
We would like to extract a representation of the sensory input that is useful for later processing.
We want to do this without requiring labeled data.
Prior ideas about what the internal representation should look like ought to be helpful. So what would we like in a representation?
Hidden causes that explain high-order correlations?
Constraints that often hold?
A low-dimensional manifold that contains all the data
Properties that are invariant across space or time?

3. Temporally invariant properties
Consider a rigid object that is moving relative to the retina:
Its retinal image changes in predictable ways
Its true 3-D shape stays exactly the same. It is invariant over time.
Its angular momentum also stays the same if it is in free fall.
Properties that are invariant over time are usually interesting.

4. Spatially invariant properties
Consider a smooth surface covered in random dots that is viewed from two different directions:
Each image is just a set of random dots.
A stereo pair of images has disparity that changes smoothly over space. Nearby regions of the image pair have very similar disparities.
plane of fixation
left eye right eye
surface

5. Learning temporal invariances
maximize agreement
non-linear
features
non-linear
features
hidden layers
hidden layers
image
image
time t+1
time t

6. A new way to get a teaching signal
Each module uses the output of the other module as the teaching signal.
This does not work if the two modules can see the same data. They just report one component of the data and agree perfectly.
It also fails if a module always outputs a constant. The modules can just ignore the data and agree on what constant to output.
We need a sensible definition of the amount of agreement between the outputs.

7. Mutual information
Two variables, a and b, have high mutual information if you can predict a lot about one from the other.
Mutual Information
Individual entropies
Joint entropy
There is also an asymmetric way to define mutual information:
Compute derivatives of I w.r.t. the feature activities. Then backpropagate to get derivatives for all the weights in the network.
The network at time t is using the network at time t+1 as its teacher (and vice versa).

8. Some advantages of mutual information
If the modules output constants the mutual information is zero.
If the modules each output a vector, the mutual information is maximized by making the components of each vector be as independent as possible.
Mutual information exactly captures what we mean by “agreeing”.

9. A problem
We can never have more mutual information between the two output vectors than there is between the two input vectors.
So why not just use the input vector as the output?
We want to preserve as much mutual information as possible whilst also achieving something else:
Dimensionality reduction?
A simple form for the prediction of one output from the other?

10. A simple form for the relationship
Assume the output of module a equals the output of module b plus noise:
If we assume that a and b are both noisy versions of the same underlying signal we can use

11. Learning temporal invariances
Backpropagate derivatives
Backpropagate derivatives
maximize mutual information
non-linear
features
non-linear
features
hidden layers
hidden layers
image
image
time t+1
time t

12. Maximizing mutual information between a local region and a larger context
Contextual prediction
w w w w4
Maximize MI
hidden
hidden
hidden
hidden
hidden
left eye right eye
surface

13. How well does it work?
If we use weight sharing between modules and plenty of hidden units, it works really well.
It extracts the depth of the surface fairly accurately.
It simultaneously learns the optimal weights of -1/6, +4/6, +4/6, -1/6 for interpolating the depths of the context to predict the depth at the middle module.
If the data is noisy or the modules are unreliable it learns a more robust interpolator that uses smaller weights in order not to amplify noise.

14. But what about discontinuities?
Real surfaces are mostly smooth but also have sharp discontinuities in depth.
How can we preserve the high mutual information between local depth and contextual depth?
Discontinuities cause occasional high residual errors. The Gaussian model of residuals requires high variance to accommodate these large errors.

15. A simple mixture approach
We assume that there are “continuity” cases in which there is high MI and “discontinuity” cases in which there is no MI.
The variance of the residual is only computed on the continuity cases so it can stay small.
The residual can be used to compute the posterior probability of each type of case.
Aim to maximize the mixing proportion of the continuity cases times the MI in those cases.

16. Mixtures of expert interpolators
Instead of just giving up on discontinuity cases we can use a different interpolator that ignores the surface beyond the discontinuity
To predict the depth at c use –a + 2b
To choose this interpolator, find the location of the discontinuity.
a b c d e

17. The mixture of interpolators net
There are five interpolators, each with its own controller.
Each controller is a neural net that looks at the outputs of all 5 modules and learns to detect a discontinuity at a particular location.
Except for the controller for the full interpolator which checks that there is no discontinuity.
The mixture of expert interpolators trains the controllers and the interpolators and the local depth modules all together.

18. Mutual Information with multidimensional output
For a multidimensional Gaussian, the entropy is given by the determinant.
If we use the identity model of the relationship between the outputs of two modules we get
If we assume the outputs are jointly Gaussian we get

19. Relationship to linear dynamical system
features
linear
features
The past
Linear model (could be the identity plus noise)
image
We predict in this domain
image
time t+1
time t