1. John A. Paulson School of Engineering and Applied Sciences
Stock Trading Using Reinforcement Learning
By Jonah Varon and Anthony Soroka
CS 182: Artificial Intelligence
Fall 2016
Introduction
Abstract: We applied Q-Learning to stock trading. Specifically we attempted to place our agent in states to find profitable strategies through technical analysis and mean reversion. Our results were mixed, leading us to question the applicability of reinforcement learning in the stock market – a space that differs in significant ways from those where reinforcement learning has proven suitable. However, our multi-agent technical analysis approach highlighted specific stocks where Q-Learning proved effective for certain time periods.
Considering the interest, there is surprisingly limited available research on reinforcement learning applied to stock trading. Hoping to either further public research or find a profitable strategy, we applied Q-Learning to stock trading.
We used Quantopian to test our strategies. Quantopian is a crowd-sourced investment firm which provides market data and a python-based development platform for algorithmic investors.
Results
Approach
Figure 1. YELP Tech-Analysis Agent
Figure 2. ATML Tech-Analysis Agent
Results Summarized: Most of our strategies did not outperform reasonable benchmarks (S&P 500 / specific stock). However, using a multi-agent approach on the technical analysis agent, we were able to scan the market for specific stocks and specific periods where the agent was profitable. Example successful agents are illustrated above.
State Representation
Q-Learning is defined by the following update function:
The state space for the stock market is vast. Hence, limiting the space to allow the agent to learn is crucial. Given such, we represented states in hopes of positioning our agent to find known trading strategies:
Technical Analysis: last 3 minutes of price and volume data {-1: Decreased, 0: Unchanged, 1: Increased}. # of States = 3^6 = 729 # of Actions per Year = 98,280
Mean Reversion: Previous week’s stock performance {-1: Stock was in the bottom X Percentile, 1: Stock was in top X percentile}
For our project, we typically applied the below:
a = {Buy, Sell}
r = Stock Performance
The stock market however has a few interesting characteristics that slightly alter Q-Learning:
a) Future State: Due to the ability to rebalance, max Q(s’,a’) is the same regardless of Q(s,a) (presuming you are a small enough investor with minimal market impact).
b) Fully Observable: Limiting the agent to actions of Buy and Sell, the reward from Q(s,a) is fully observable regardless of the actual action taken. We tried both approaches to updating Q(s,a): 1) Only for action taken
2) For both taken and not taken actions
Conclusions
Our attempts to find alpha using Q-Learning illustrate how competitive and efficient markets are. Additionally it highlights that reinforcement learning is perhaps not the best suited for stock trading given:
Stochasticity of the market
Limited amount of train data
Agent can observe rewards without taking actions
Agent has little impact on the environment
That being said the possible methods to define a state space are limitless and should still be investigated. Moreover, our multi-agent approach proved useful in highlighting specific stocks where Q-Learning might be applicable.
Lastly, we believe Deep Learning techniques such as LSTM might be better suited for this problem.
Data
Quantopian provides a wealth of technical, fundamental, and event information from thousands of stocks.
Code Snippet. Updating Technical Analysis Agent’s State
John A. Paulson School of Engineering and Applied Sciences