Mastering the game of Go with deep neural network and tree search David Silver, Aja Huang, et al.

The Game of Go (围棋, 囲碁, 바둑） Invented in China more than 2000 years ago Two simple rules: Players put black and white pieces on crosses of the board alternatively. The player controlling the most crosses in the end wins. If all liberties of a piece or of a set of connected pieces are taken by pieces of another color, it/they will be removed from the board. A huge search space The board has 361 (19*19) crosses. Approximately 250^150 possible sequence of moves.

Monte Carlo Tree Search (MCTS) Selection: start from root R and select successive child nodes down to a leaf node L. The section below says more about a way of choosing child nodes that lets the game tree expand towards most promising moves, which is the essence of Monte Carlo tree search. Expansion: unless L ends the game with a win/loss for either player, either create one or more child nodes or choose from them node C. Simulation: play a random playout from node C. Backpropagation: use the result of the playout to update information in the nodes on the path from C to R.

MCTS

Supervised Learning of Policy Network The SL policy network pσ(a|s) alternates between convolutional layers with weights σ, and rectifier nonlinearities. A final softmax layer outputs a probability distribution over all legal moves a. The input s to the policy network is a simple representation of the board state The policy network is trained on randomly sampled state-action pairs (s, a), using stochastic gradient ascent to maximize the likelihood of the human move a selected in state s

Reinforcement Learning of Policy Network play games between the current policy network pρ and a randomly selected previous iteration of the policy network.  Randomizing from a pool of opponents in this way stabilizes training by preventing overfitting to the current policy. use a reward function r(s) that is zero for all non-terminal time steps t < T. outcome zt = ± r(sT) is the terminal reward at the end of the game from the perspective of the current player at time step t: +1 for winning and −1 for losing.

Reinforcement Learning of Policy Network   Reinforcement Learning of Policy Network Weights are then updated at each time step t by stochastic gradient ascent in the direction that maximizes expected outcome evaluate the performance of the RL policy network in game play, sampling each move from its output probability distribution over actions.

Reinforcement Learning of Value Network estimating a value function vp(s) that predicts the outcome from position s of games played by using policy p for both players approximate the value function using a value network vθ(s) with weights θ  train the weights of the value network by regression on state- outcome pairs (s, z), using stochastic gradient descent to minimize the mean squared error (MSE) between the predicted value vθ(s), and the corresponding outcome z

Neural network training pipeline and architecture D Silver et al. Nature 529, 484–489 (2016) doi:10.1038/nature16961

MCTS in AlphaGo

Thank You