Deep Reinforcement Learning: Learning how to act using a deep neural network Psych 209, Winter 2019 February 12, 2019

How can we teach a neural network to act? Direct ‘policy’ supervision, or imitation learning Provide learner with a teaching signal at each step, back propagate. What problems might we encounter with this approach? What if we don’t get examples from the environment to tell us what the correct action is? Instead we only get rewards when special events occur, e.g. we stumble onto a silver dollar. Computer games can be like this, and maybe when animals forage in the wild they face this problem Core intuition Increase probability of actions that maximize the ‘expected discounted future reward’ Central concepts: V(s) and Q(s,a) Karpathy’s version: Directly calculate expected discounted future reward from gameplay rollouts The classical RL approach: Base actions on Value estimates directly Gradually update estimates of V and/or Q via the Bellman Equation

The Bellman Equation Principle of Optimality: An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. 𝑉 𝑠 = max 𝑎 𝑄 𝑠,𝑎 + 𝛾𝑉 𝑠′ Problem 1: Value depends on policy Problem 2: Value information may require exploration to obtain

Example 2-d grid world Discrete states 4 possible actions Next state based on action (no change if move into wall) Positive reward occurs when we stumble onto the silver dollar Negative reward occurs when we fall into the black hole How can we learn about this? We must explore: Softmax based exploration E-greedy exploration Estimated reward depends on exploration policy!

Discussion? Still staying with V and Q learning, how can we speed things up?

Q-learning as in Mnih et al

Value learning based on rehearsal buffer Take an action Store state, action, reward, next state tuple in rehearsal buffer Sample a batch of tuples from buffer and for each: Use stored policy parameters to estimate Q value of next state Calculate the loss as the difference between the reward and what the old parameters told you about the loss and your estimate of the value of the next state Update your weights based on the loss over a batch If buffer is full, discard oldest item in the buffer

Advantage Actor Critic (A2C) Estimate Value and policy Use Value estimate of next state to as measure of advantage Use value estimate of next state to update value estimate Solve the independent samples problem by using many independent actors simultaneously learning in the same environment