1. 7. Analysis of performance of trading strategies
7.1 Basic concepts
“It is hard to make predictions, especially about the future” (Mark Twain) => standard disclaimer of asset management institutions : “Past investment performance does not guarantee future returns”.
Back testing: prediction models are fitted (in-sample) and tested (out-of-sample) using past empirical data.
Walk-forward testing: for 10-year sample, 1-5-year in-sample, 6th year out-of-sample (1-5,6); then (2-6, 7), etc.
Usually, variance-stationary models. More is better? What’s S&P 500?
Regime shifts caused by macroeconomic events or changes in regulatory policies (e.g. introduction of Euro in 1999, changes in the up-tick rule, etc.) => Markov-switching models (Tsay (2005)).

2. 7. Analysis of performance of trading strategies
7.1 Basic concepts (continued)
Danger of over-fitting: “too good to be true” in-sample accuracy => disaster out-of-sample (fitting the noise).
MLE-based Akaike information criterion (AIC):
AIC = -2 ln[L(n, N)]/N + 2n/N
L(n, N) is likelihood for N points in sample and n model parameters.
For ARMA(p, q) with Gaussian noise, AIC(p, q, N) = ln(σp,q2) +2(p+q)/N.
Optimal ARMA(p, q) yields minimal AIC. Alas, optimal ARMA(p, q) in-sample may not be optimal out-of-sample (Hansen (2010)).
Data snooping bias: the best predictor in the UN database for the S&P 500 stock index is production of butter in Bangladesh (Sullivan et al (1999)).
Resampling!!!

3. 7. Analysis of performance of trading strategies
7.2 Performance measures
Performance is studied for a time interval >> round-trip transaction.
Market imperfections: bid/ask spread, finite liquidity, transaction fees.
- Realized (!) total (compound) return – ultimate benchmark
R = (1 + r1)(1+r2)… - 1
Is this just luck?
- Percentage of winning trades p
- Ratio of average winning amount to average losing amount r
- Kelly’s criterion for estimating the optimal fraction of trading capital, f, to
be used in each trade: f = (pr – 1 + p)/r – valid only asymptotically (be
careful!.. (Thorp (2006), Poundstone (2006)).

4. 7. Analysis of performance of trading strategies
7.2 Performance measures (continued)
- Multiple trades => probability distribution => comparing strategies
- If distribution ~ N(µ, σ), t-value t =
Check null hypothesis (µ = 0) using t distribution with N-1 degrees of
freedom.
- Comparing two strategies (or buys and sells (LeBrock et al (1992)):
t =
Usually not N(µ, σ)... Hence Wilcoxon (Mann-Whitney) test for medians and
Kolmogorov-Smirnov test.

5. 7. Analysis of performance of trading strategies
7.2 Performance measures (continued 2)
- Sharpe ratio => Sortino ratio (only negative returns are counted in σ).
- Informational ratio for comparing with (buy-and-hold) index strategy:
IR = (E[ri] - E[r0])/σi0
- Maximum drawdown: maximum price drop after its peak
MD = max[ max X(s) - X(t) ]
t [0, T] s [0, t]
For random walk with no drift (Magdon-Ismail & Atiya (2004)) :
E[MD] = σ
- Calmar ratio: Sharpe ratio with σ => MD
- Number of consecutive losing trades: no strategy works forever...

6. Max Drawdown