Recording a Game of Go: Hidden Markov Model Improves Weak Classifier Steven Scher

Input: video sequence Video (straightened)

Weak classifier Simple Simple Acts independently at each possible stone location Acts independently at each possible stone location Looks for Circles Looks for Circles Hough Transform Hough Transform Looks for extremely bright or dark patches Looks for extremely bright or dark patches Very Noisy Very Noisy Video (straightened) weak frame-by-frame Stone-classifier = black stone = white stone = wait until next slide

Improve Weak classifier: Ignore Motion Ignore spurious detections caused by hands Ignore spurious detections caused by hands May delete good detections May delete good detections Overall, seems to improve the classifier Overall, seems to improve the classifier Motion classifier Video (straightened) weak frame-by-frame Stone-classifier = black stone = white stone = ignored (motion)

Estimating the Sequence of moves: A Simple Algorithm Discard movie frames where too many new stones appear since the last valid frame Discard movie frames where too many new stones appear since the last valid frame Discard movie frames with inappropriate mix of classified black and white stones Discard movie frames with inappropriate mix of classified black and white stones should be equal, +/-1 should be equal, +/-1 Results Results Relies on occasional clean, noiseless frames Relies on occasional clean, noiseless frames For S > 5, many artifacts introduced For S > 5, many artifacts introduced For S < 5, algorithm cannot handle a long sequence (if there is no valid frame during the time that S stones are actually played) For S < 5, algorithm cannot handle a long sequence (if there is no valid frame during the time that S stones are actually played)

A Better Model: Hidden Markov Model Consider Each Possible Stone Location Independently Consider Each Possible Stone Location Independently Consider each transition to depend only on current state Consider each transition to depend only on current state empty Black stone White stone

HMM over time is a tree time Hidden state = {empty, black, white} Observed state = {empty, black, white, unknown/motion} time=ttime=t+1time=t-1

Transition probabilities empty Black stone White stone P new P stay = 1-2*P new 1 1 P new Estimate:

Probabilities of Observations if there is a stone empty Black stone White stone empty Black stoneWhite stone Hand moving (no information) (Black stones’ probabilities similar to white stone) Estimate:

Probabilities of Observations if there is no stone empty Black stone White stone empty Black stoneWhite stone Hand moving (no information).7.1 Estimate:

Data-Derived Estimate from Baum-Welch Algorithm empty Black stone White stone empty Black stoneWhite stone Hand moving (no information).7.1 X.5 X.3 Baum-Welsh:

Baum-Welch & Viterbi Algorithms Baum-Welch Algorithm Baum-Welch Algorithm Expectation-Maximization Expectation-Maximization Viterbi Algorithm Viterbi Algorithm Recursive (Dynamic Programming) Recursive (Dynamic Programming) Quick Quick Linear in length of sequence Linear in length of sequence Quadratic in # of hidden states Quadratic in # of hidden states

Likely sequence of hidden states: Viterbi Algorithm Motion classifier Video (straightened) weak frame-by-frame Stone-classifier Likely Sequence Of Hidden States

Algorithm provides a reasonable solution not robust enough to be fully autonomous Motion classifier Video (straightened) weak frame-by-frame Stone-classifier Likely Sequence Of Hidden States mistakes

Minimizing User Interaction Assign the first frame to be empty Assign the first frame to be empty Add e.g. 10 copies to beginning of sequence Add e.g. 10 copies to beginning of sequence Correct errors in the last frame by hand Correct errors in the last frame by hand

User Interface Not Fully Automated Not Fully Automated Useful tool, given a good user interface Useful tool, given a good user interface

Future Work Improve the Graphical Model Improve the Graphical Model Only one stone allowed to appear Only one stone allowed to appear Per image frame? Per image frame? Per t seconds of real-time? Per t seconds of real-time? Improve the stone/empty classifier Improve the stone/empty classifier More robustness against lighting changes desired More robustness against lighting changes desired Train on labeled examples Train on labeled examples Viola & Jones? Viola & Jones? Improve the motion detector Improve the motion detector Upgrade to a hand-tracking algorithm Upgrade to a hand-tracking algorithm This also be done with a graphical model This also be done with a graphical model Incorporate into one model? Incorporate into one model? GUI GUI Allow a variety of user labeling, update as new labels are made? Allow a variety of user labeling, update as new labels are made?

Questions? Weak Classifier is noisy Weak Classifier is noisy Estimating sequence of states from observations removes much of the noise Estimating sequence of states from observations removes much of the noise