1. Model Validation For Derivative Pricing: The First Line of Risk Management Defense

2. Defining Model Risks
Model Risk is a collection of several related issues surrounding the behavior of a derivatives pricing model. The recent financial crisis and the attendant regulatory crackdown has compelled a renewed interest in this field. This has driven deeper thinking on the many possible contributors to model risk.

3. Types of Model Risk in Derivative Pricing
Implementation Errors (bugs)
Dynamic Misspecification, i.e.
Omission of important risk factors
Model performs poorly in new market regime
Poor choice of calibration instruments
Poor choice of calibration method
Bad weighting choice in objective function (e.g. absolute vs. relative price errors)
Excessively local solvers failing to find the best solution
Poor initial guess for solver
Poor choice of hedge instruments

4. Types of Model Risk, cont’d
Poor choice of hedge ratio calculation method
E.g. Continuous derivatives vs. various discrete difference methods
Poor choice of proxy data
Data Quality problems – feeding the models with the wrong data
Poor Documentation – a misunderstanding of the model can lead to passing incorrect data to the model, i.e. data quality problems

5. What can we do about these risks?
Implementation Risks
A mathematical library separate from the financial library, to be called by the financial library, instead of writing inlined or bespoke mathematical functions for each option pricer.
A strong testing regime, containing:
Mathematical identities for all pure math functions
Limiting or Special cases
Tests should be run on MORE than just a few typical cases –
A test domain should be defined and a large number of cases from the test domain should be selected at random, different ones each night. The tests need to cover high, and low values, in every possible combination, of all parameters.
Multiple methods
A strong testing regime pays you back with easier debugging, by ruling out places where bugs can be, by allowing developers and bugfixers to try out fixes on unfamiliar code more fearlessly.

6. Implementation Risks – Examples of Possible Tests
Instrument and Model Specific Tests
Parallel Reimplementation
Limiting Cases or Identities for Payoff (e.g. Knock-in + Knock-out = Vanilla, Barrier  Vanilla as Barrier  infinity)
Limiting Cases or Identities for Model (e.g. Heston  Black-Scholes, Romano-Touzi Mixing Theorem)
Convergence and convergence rate of method
Zero and infinite volatility limits, to catch payoff specification errors.

7. Model Validation Tests – Mathematical Correctness
Model and Instrument Independent, “Generic” Tests
Delta-Vega Smoothness Tests
Is there a “jitter” or oscillation mixed in with the pricing that
throws off greeks even if model price is accurate?
Calibration Round Trip Tests
Does our calibration method re-price the calibrated instruments
accurately? When a calibration problem is set up with a known solution, can the calibrator find it?
Convergence Tests
Does our pricing converge, and is the production price close to the converged value?

8. Model Validation Tests – Financial Correctness
Calibration Stability Tests
Does our model have similar calibration parameters from day-to-day? Does our model price the same vanilla similarly from day-to-day?
Calibration Error Tests
Does our model, when calibrated to vanillas from the market, price all the calibration instruments accurately?
PnL Attribution Tests
Does our model account for changes in PnL of the instrument due to changes in market data well without calibration? If we use this model to hedge, will our hedges hedge our changes in value? (Similar to Calibration Stability Test)
Variance of Hedged Portfolio Test
What is the variance of the hedged portfolio (similar to PnL Attribution Test)
Cost of Hedging Test
Does the initially calculated price accurately reflect the eventual cost of hedging?

9. Model Validation Tests – Test Definitions
Calibration Stability Tests
We calibrate a model to vanilla instruments on a series of dates, and price a test option. We then perturb the market data, and re-price the test option. The change in the price should be smaller than the perturbation. This test is to be run over market data from
Many different market regimes, preferably at least 10 years worth.
Calibration Error Tests
We calibrate a model every day, and record the sum of squared errors. We may then compare this to other models, or to the same model in other periods.
PnL Attribution Tests
We compute the price and hedge-ratios of the model every day throughout the life of the option, starting at a day in history, and record the PnL difference as reported by re-calibration and re-pricing, on the one hand, and using the Greeks and changes in underlying on the other. We graph this function, and look for spikes in this graph.

10. Model Validation Tests – Test Definitions
Variance of Hedged Portfolio Test
We compute the price of the portfolio and its hedge together, every day. This should barely move, and its movement is a measure of residual risk due to the model’s imperfect description of the market.
We compute the sum of squared errors of the one-day change in PnL of the hedged portfolio, and report this as the residual risk
Cost of Hedging Test
We compute the hedge every day, and consider the losses or gains in this trading strategy by itself. The theory of Black-Scholes says that the price of the option should be equal to its cost of hedging. We then compare the cost of hedging to the computed price of the option at inception.

11. Questions to ask in your organization
How old is my code – for what percentage of code are the original authors still in the organization?
How much time do my developers spend debugging, vs. new code development?
Are there known bugs that no one knows how to solve?
How much are you relying on “work-arounds”, instead of true bug-fixes?
How much do my traders trust the model? How much manual intervention does it require?
How much bleed are hedgers seeing? How much variance from market prices?
How volatile is my portfolio?
Are my calibrations accurately repricing the market?
Are my calibrations stable?
What kinds of data-tagging does the code do to identify things like “which volatility is coming from where”, to prevent data errors?

12. Thank you Visit us online at: News and updates:
News and updates:
blog.numerix.com
Derivatives discussion board:
forums.numerix.com