1. Learning Objective
Using the generative form of Bayes’ equation, the learning objective is to find the most probable explanation, H, for the input patterns, D, presented to the network.
Thus, we want to develop a generative model that encodes the probabilities of the input data within the network’s weight structure, W.
Hence, the learning objective reduces to adapting the weight vector, W, to find the most probable explanation of the input patterns.

2. Unit Relationships
Given the generative nature of the network, for any given unit, Si, we can define up to three different unit relationships: Si Sj Sl
i. Parents (pa): Sj > Si
ii. Children (ch): Sk < Si Sk
iii. Siblings (sib): Sl = Si

3. Computing Unit Probabilities
Being a generative model, the probability of any unit’s state is directly computable from the states of its parents:  S1 Si S2 Sj u u
0.0
0.5
1.0
1 - e u
The function h specifies how the underlying causes are combined to produce the probability of Si = 1. Here, we use the “noisy OR” function:
P(Si = 1)

4. Sampling Network States
Each state of the network is updated iteratively according to the probability of each unit state, Si, given the states of the remaining units.
This conditional probability is computed as:
Thus, this sampling procedure can be interpreted in terms of a stochastic recurrent neural network where the feedback from the deeper layers influences states at the surface layers.

5. Changing Network States
One can compute the probability of a unit being active, Si = 1, given the remaining units, Sj, and the network weights, W, as
the variable Dxi indicates how much changing the unit Si to active changes the overall probability of the network state. i x = D ]) [ ( i S pa fb ]) [ ( i S ch ff +

6. Feedback
Feedback for unit Si is simply computed as the log probability of the unit being on minus the log probability of the unit being off. Si 1
This term drops out of the equation for the deepest layer of the network.
Feedback allows more surface level units to use information computable at the deeper layers.

7. Feedforward
Feedforward is computed as the probability of the unit being on given the activity of its children. Si ü + - þ ý ) ( 1
log ik i k w S u h å Î î í ì = ] [ ch ff 1 1 1 1 1
The feedforward term typically dominates the Dxi equation, but when ff(Si) = 0 (i.e., ambiguous), then fb(Si) resolves the conflict.

8. Sparse Coding Constraints
We modified the original framework to include sparse coding constraints on the unit activation probabilities. ]) [ ( i S pa fb ch ff + x = D ]) [ ( i S pa fb ch ff + x = D sp × l
sib
That is, all things being equal, sparse coding encourages the model to represent any specific input pattern with relatively few units.
Sparse coding constraints within networks have been shown to create more biologically plausible receptive fields.

9. Weight Estimation
Having sampled the activation space, we want to estimate the weights, W, to maximize the probability of generating the observed data, D.
To control the complexity of the model, we apply a weight prior based on the product of two independent gamma distributions parameterized by  and .
Thus, we want to maximize the objective function:

10. Weight Updating å å This is accomplished by using the transformations
and solving for å + - = n i j ij S g f ) ( 2 1 b a å + - = n i j ij S g f ) ( 2 1 b a
The important thing to note is that this equation is simply a form of Hebbian learning.