1. © National Bank of Belgium

2. Failure Prediction Models: Disagreements, Performance, and Credit Quality Janet MITCHELL and Patrick VAN ROY National Bank of Belgium “Small business banking and financing: a global perspective” Cagliari, May 25th

3. © National Bank of Belgium Motivation  The paper explores empirically a number of comparative issues relating to models assigning failure predictions (credit scores or PDs) to non-listed firms.  Failure prediction models are important in Basel II (PDs key input for the calculation of regulatory capital under IRB approach).  Focus on four models: the National Bank of Belgium (NBB) bankruptcy prediction model, two vendor models (Model 1 and Model 2) and the Z-score (Altman).

4. © National Bank of Belgium Four main issues  Disagreements between models: do different models yield significantly different "rankings" for the same firm?  Model power: are some models better at differentiating between failing and non-failing firms?  Combining models: are combinations of models more powerful than single models?  Design of internal ratings systems: does model power change as the number of rating classes and the distribution of borrowers across classes vary?

5. © National Bank of Belgium Data and sample  40,000 small to medium-sized non-listed Belgian firms.  Inputs, statistical methods and calibration differ across models.  Data are obtained from the Belgian central balance sheet office and from one vendor.  Bankruptcy data is used to estimate 1-year and 5- year credit scores or PDs. The presentation focuses on 1-year failure predictions.

6. © National Bank of Belgium 1-year bankruptcy rates (in %) Risk class% firmsNBBModel 1Model 2Z-score 101.40.00 221.50.000.010.060.05 321.50.090.060.110.25 418.80.220.160.200.45 522.00.340.430.570.40 611.61.442.001.121.28 703.37.855.526.855.46 Methodology Output of each model (PDs or credit scores) is rank- ordered before being mapped to 1 of 7 risk classes based on the output of one vendor model: low risk high risk

7. © National Bank of Belgium Disagreements (1 vs. 4,5,6 or 7) Percentage of class-1 firms (= lowest risk firms) of a given model classified in or above the median risk class (= class 4) by another model: Class 1Class 4,5,6 or 7 NBBModel 1Model 2Z-score NBB-01.436.754.2 Model 100.5-18.921.9 Model 234.329.4-42.3 Z-score07.507.744.2-

8. © National Bank of Belgium Disagreements (7 vs. 1,2,3 or 4) Percentage of class-7 firms (= highest risk firms) of a given model classified in or below the median risk class (= class 4) by another model: Class 7Class 1,2,3 or 4 NBBModel 1Model 2Z-score NBB-16.314.516.4 Model 108.7-15.633.2 Model 218.119.0-29.3 Z-score18.426.611.5-

9. © National Bank of Belgium Model power: ROC curves type 1 error 1  type 2 error

10. © National Bank of Belgium ROC curves of the 4 models

11. © National Bank of Belgium ROC areas of the 4 models ModelArea under the ROC curve NBB0.876 Model 10.868 Model 20.833 Z-score0.779 Area of model with no discriminatory power = 0.5 Area of model with acceptable discriminatory power > 0.7 Area of model with perfect discriminatory power = 1.0

12. © National Bank of Belgium ROC areas of selected combinations NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833; Z-score (Z) = 0.779 CombinationMin.Max.MedianMean N - M10.8780.8980.908 N - M20.8610.8920.898 N - Z0.8540.8550.880 N - M1 - M20.8670.9010.8990.916 N - M1 - Z0.8610.8860.8940.911 N - M2 - Z0.8450.8790.8830.899 N - M1- M2 - Z0.8520.8900.9140.917

13. © National Bank of Belgium ROC areas of selected combinations NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833; Z-score (Z) = 0.779 CombinationMin.Max.MedianMean N - M10.8780.8980.908 N - M20.8610.8920.898 N - Z0.8540.8550.880 N - M1 - M20.8670.9010.8990.916 N - M1 - Z0.8610.8860.8940.911 N - M2 - Z0.8450.8790.8830.899 N - M1- M2 - Z0.8520.8900.9140.917

14. © National Bank of Belgium ROC areas of selected combinations NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833; Z-score (Z) = 0.779 CombinationMin.Max.MedianMean N - M10.8780.8980.908 N - M20.8610.8920.898 N - Z0.8540.8550.880 N - M1 - M20.8670.9010.8990.916 N - M1 - Z0.8610.8860.8940.911 N - M2 - Z0.8450.8790.8830.899 N - M1- M2 - Z0.8520.8900.9140.917

15. © National Bank of Belgium ROC areas of selected combinations NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833; Z-score (Z) = 0.779 CombinationMin.Max.MedianMean N - M10.8780.8980.908 N - M20.8610.8920.898 N - Z0.8540.8550.880 N - M1 - M20.8670.9010.8990.916 N - M1 - Z0.8610.8860.8940.911 N - M2 - Z0.8450.8790.8830.899 N - M1- M2 - Z0.8520.8900.9140.917

16. © National Bank of Belgium ROC areas of selected combinations NBB (N) = 0.876 ; Model 1 (M1) = 0.868 ; Model 2 (M2) = 0.833; Z-score (Z) = 0.779 CombinationMin.Max.MedianMean Z - N0.8540.8550.880 Z - M10.8470.8710.890 Z - M20.8080.8580.855

17. © National Bank of Belgium ROC areas of 9 possible internal ratings systems (NBB model) Number of classes Mapping of firms based on Vendor model distribution Moody's distribution Equal distribution 70.8760.8740.858 100.8830.8820.873 170.8870.8850.883 Note: NBB continuous credit score has an ROC area of 0.889 0.018 0.010 0.004 0.011 0.025

18. © National Bank of Belgium Conclusion  High disagreements rates between models: model choice can have a significant impact on loan pricing and origination decisions.  High power of each model: the definition of failure as well as the statitical method used by the models may not matter as much as one would have expected.  Larger differences between differing combinations of models than between differing internal rating systems.

19. © National Bank of Belgium