1. Neural Network Implementation of Poker AI
Christopher Kramer

2. Outline of Information
The Challenge
Application, problem to be solved, motivation
Why create a poker machine with ANNE?
The Flop
The hypothesis
Can a Poker AI run using only an ANNE?
The Turn
The approach and methodology
How is this machine created?
The River
Experimentation
How should it be tested?
The Showdown
Conclusions and future work
Did it work?

3. The Challenge
The approach focused on creating an AI capable of playing poker at a reasonable level of play using Neural Networks.
The difficulty:
Incomplete knowledge
Multiple competing agents
Risk management
Opponent modeling
Deception
Unreliable information
Creating solutions to address these difficulties can be applied to other problems with similar domains.
NOISE!

4. The Fold (Hypothesis) The strengths of computers:
Quick computation
Large memory
Purely logical
Using a series of machine learning techniques including neural networks, it is predicted that a machine will be able to play at a moderate level while ignoring typical features of poker programs such as statistical analysis and pot odds.

5. The Turn (Approach) Pre-Flop Post-Flop River Showdown
Problem split into a number of stages
Hardcoded
Action chosen based off of known values of starting hands
Pre-Flop
Neural Net
5 card input
Post-Flop
6 card input
River
Player hand analyzed and given as set of input values
Showdown

6. The Turn (continued) Pre-Flop
Hardcoded
Action chosen based off of known values of starting hands
Information gathered from percentage of games won with starting hand from poker databases
All hands in top 30th percentile were analyzed for characteristics which were hardcoded into if statements
All hands below threshold were folded.
Those with characteristics in top 5th percentile had a higher random chance of being raised

7. The Turn (continued) Post-Flop
Neural Net
5 card input
Inputs the unique id of each card in hand and play
Hidden nodes utilizing sigmoid function
Output using sigmoid function between -1 and 1
Output values above .3 indicated an action to raise
Below -.3 fold
Else, Call

8. The Turn (continued)

9. The Turn (continued) River
Neural Net
6 card input
Inputs the unique id of each card in hand and play
Hidden nodes utilizing sigmoid function
Output using sigmoid function between -1 and 1
Output values above .3 indicated an action to raise
Below -.25 fold
Else, Call

10. The Turn (continued) Showdown
Neural Net
Player hand analyzed and given as set of input values
Cards analyzed for poker hands such as Flush, Fold, etc. and high card of each returned.
Input nodes for each poker hand. Value is the value of the highest card in hand.
Hidden nodes utilizing sigmoid function
Output using sigmoid function between -1 and 1
Output values above .5 indicated an action to raise
Below -.2 fold
Else, Call

11. The Turn (opponent modeling)
Opponent modeling generalized to fit into previous Neural networks
Opponents action modeling taken into account by Neural nets due to opponents actions changing the weights of neural network to account for their actions.
2 Factors
Player’s calls
For each raise by opponent, have a higher required confidence to raise again
Players treatment of poker hand values
As player actions change, the confidence values of the Neural nets should change to reflect the modeling of the player

12. The River (experimentation)
Experiment was developed to overcome short term failure due to bad luck
1000 dollars given to each player. $10 bet .
Engines initial weights were determined experimentally for each ANNE
Weights were assigned randomly and run for generations against a set of random poker hands and card tables. Winning gave an expected value of 1, loss -1, and draw 0.
Network was then checked against 1000 random validation hands.
A degree of accuracy was required for each ANNE
Required accuracy lowered as amount of information known decreases.
Once Neural Network was fit level of confidence its results were accepted as the Engines default weights.
Weights continue training during the course of game to profile player.
Tested against a number of human opponents based on skill level to measure success.

13. The River (experiment results)
Beginners
AI won 82% of games against 50 opponents
Moderate (play at least 1 game a month)
AI won 20% of games against 20 opponents
Advanced (play at least 1 game a week)
AI won 1 game out of 10 against 2 opponents.
Random (never folded)
AI won 63.2% of games out of 1000.
100% won against Random opponent who could also fold

14. The Showdown (Conclusion)
The complete reliance on generalized Neural Networks for a system with extreme noise is not a viable option for higher levels of play.
While it did not shown significant advantage over moderate and advanced opponents, it did well against beginning opponents who knew how to play poker but did not do so often.
So much noise….
While a degree of noise was expected, generalized poker domain’s noise was is in excess of % (approx.)
This led to extreme cases of overlearning regardless of an extremely low learning rate.
While this was partially handled using confidence, the amount of folds late in the game due to incorrect early predictions led to the machine still failing to moderate opponents.
Was still shown to do well against a random opponent and beginners.

15. The Showdown (future research)
There are numerous areas for improvement
Current systems adjusts slower than humans
Possible GA implementation instead of Neural Network to avoid direct noise issues.
Statistical analysis and pot odds would help as inputs for neural network.

16. Questions?