1. ECE 517: Reinforcement Learning in Artificial Intelligence Lecture 17: TRTRL, Implementation Considerations, Apprenticeship Learning
November 3, 2010
Dr. Itamar Arel
College of Engineering
Department of Electrical Engineering and Computer Science
The University of Tennessee
Fall 2010

2. Outline Recap on RNNs Implementation and usage issues with RTRL
Computational complexity and resources required
Vanishing gradient problem
Apprenticeship learning

3. RNNs are potentially much stronger than FFNN
Recap on RNNs
RNNs are potentially much stronger than FFNN
Can capture temporal dependencies
Embed complex state representation (i.e. memory)
Models of discrete-time dynamic systems
They are (very) complex to train
TDNN – limited performance based on window
RTRL – calculates a dynamic gradient on-line

4. RTRL reviewed RTRL is a gradient descent based method
It relies on sensitivities
expressing the impact of any weight wij on the activation of neuron k.
The algorithm then consists of computing weight changes
Let’s look at the resources involved …

5. Implementing RTRL – computations involved
The key component in RTRL is the sensitivities matrix
Must be calculated for each neuron
RTRL, however, is NOT local …
Can the calculations be efficiently distributed?
N 3 N
N 4

6. Implementing RTRL – storage requirements
Let’s assume a fully-connected network of N neurons
Memory resources
Weights matrix, wij  N 2
Activations, yk  N
Sensitivity matrix  N 3
Total memory requirements O(N 3)
Let’s go over an example:
Let’s assume we have 1000 neurons in the system
Each value requires 20 bits to represent
 ~20 Gb of storage!!

7. Possible solutions – static subgrouping
Zipser et. al (1989) suggested static grouping of neurons
Relaxing the “fully-connected” requirement
Has backing in neuroscience
Average “branching factor” in the brain ~ 1000
Reduced the complexity by simply leaving out elements of the sensitivity matrix based upon subgrouping of neurons
Neurons are subgrouped arbitrarily
Sensitivities between groups are ignored
All connections still exist in the forward path
If g is the number of subgroups then …
Storage is O(N3/g2 )
Computational speedup is g3
Communications  each node communicates with N/g nodes

8. Possible solutions – static subgrouping (cont.)
Zipser’s empirical tests indicate that these networks can solve many of the problems full RTRL solves
One caveat of the subgrouped RTRL training is that each subnet must have at least one unit for which a target exists (since gradient information is not exchanged between groups)
Others have proposed dynamic subgrouping
Subgrouping based on maximal gradient information
Not realistic for hardware realization
Open research question: how to calculate gradient without the O(N3) storage requirement?

9. Truncated Real Time Recurrent Learning (TRTRL)
Motivation:
To obtain a scalable version of the RTRL algorithm while minimizing performance degradation
How?
Limit the sensitivities of each neuron to its ingress (incoming) and egress (outgoing) links

10. Performing Sensitivity Calculations in TRTRL
For all nodes that are not in the output set, the egress sensitivity values for node i are calculated by imposing k=j in the original RTRL sensitivity equation, such that
Similarly, the ingress sensitivity values for node j are given by
For output neurons, a nonzero sensitivity element must exist in order to update the weights

11. Resource Requirements of TRTRL
The network structure remains the same with TRTRL, only the calculation of sensitivities is reduced
Significant reduction in resource requirements …
Computational load for each neuron drops to from O(N3) to O(2KN), where K denotes the number of output neurons
Total computational complexity is now O(2KN2)
Storage requirements drop from O(N3) to O(N2)
Example revisited: For N=100, 10 outputs  100k multiplications and only 20kB of storage!

12. Further TRTRL Improvements – Clustering of Neurons
TRTRL introduced localization and memory improvement
Clustered TRTRL adds scalability by reducing the number of long connection lines between processing elements
Input
Output

13. Test case #1: Frequency Doubler
Input: sin(x), target output sin(2x)
Both networks had 12 neurons

14. Vanishing Gradient Problem
Recap on goals:
Find temporal dependencies in data with a RNN
The idea behind RTRL: when an error value is found, apply it to inputs seen an indefinite number of epochs ago
In 1994 (Bengio et. al) it has been shown that both BPTT and RTRL suffer from the problem of vanishing gradient information
When using gradient based training rules, the “error signal” that is applied to previous inputs tends to vanish
Because of this, long-term dependencies in the data are often overlooked
Short-term memory is ok, long-term (>10 epochs) – lost

15. Vanishing Gradient Problem (cont.)xt yt st
RNN
A learning error yields gradients on outputs, and therefore on the state variables st
Since the weights (parameters) are shared across time

16. What is Apprenticeship Learning
Many times we want to train an agent based on a reference controller
Riding a bicycle
Flying a plane
Starting from scratch may take a very long time
Particularly for large state/action spaces
May cost a lot (e.g. helicopter crashing)
Process:
Train agent on reference controller
Evaluate trained agent
Improve trained agent
Note: reference controller can be anything (e.g. heuristic controller for Car Race problem)

17. Formalizing Apprenticeship Learning
Let’s assume we have a reference policy p from which we want our agent to learn
We would first like to learn the (approx.) value function, Vp
Once we have Vp , we can try an improve it based on the policy improvement theorem, i.e.
By following the original policy greedily we obtain a better policy!
In practice, many issues should be considered such as state space coverage and exploration/exploitation
Train on zero exploration, then explore gradually …