© National Bank of Belgium

Failure Prediction Models: Disagreements, Performance, and Credit Quality Janet MITCHELL and Patrick VAN ROY National Bank of Belgium “Small business banking and financing: a global perspective” Cagliari, May 25th

© National Bank of Belgium Motivation  The paper explores empirically a number of comparative issues relating to models assigning failure predictions (credit scores or PDs) to non-listed firms.  Failure prediction models are important in Basel II (PDs key input for the calculation of regulatory capital under IRB approach).  Focus on four models: the National Bank of Belgium (NBB) bankruptcy prediction model, two vendor models (Model 1 and Model 2) and the Z-score (Altman).

© National Bank of Belgium Four main issues  Disagreements between models: do different models yield significantly different "rankings" for the same firm?  Model power: are some models better at differentiating between failing and non-failing firms?  Combining models: are combinations of models more powerful than single models?  Design of internal ratings systems: does model power change as the number of rating classes and the distribution of borrowers across classes vary?

© National Bank of Belgium Data and sample  40,000 small to medium-sized non-listed Belgian firms.  Inputs, statistical methods and calibration differ across models.  Data are obtained from the Belgian central balance sheet office and from one vendor.  Bankruptcy data is used to estimate 1-year and 5- year credit scores or PDs. The presentation focuses on 1-year failure predictions.

© National Bank of Belgium 1-year bankruptcy rates (in %) Risk class% firmsNBBModel 1Model 2Z-score Methodology Output of each model (PDs or credit scores) is rank- ordered before being mapped to 1 of 7 risk classes based on the output of one vendor model: low risk high risk

© National Bank of Belgium Disagreements (1 vs. 4,5,6 or 7) Percentage of class-1 firms (= lowest risk firms) of a given model classified in or above the median risk class (= class 4) by another model: Class 1Class 4,5,6 or 7 NBBModel 1Model 2Z-score NBB Model Model Z-score

© National Bank of Belgium Disagreements (7 vs. 1,2,3 or 4) Percentage of class-7 firms (= highest risk firms) of a given model classified in or below the median risk class (= class 4) by another model: Class 7Class 1,2,3 or 4 NBBModel 1Model 2Z-score NBB Model Model Z-score

© National Bank of Belgium Model power: ROC curves type 1 error 1  type 2 error

© National Bank of Belgium ROC curves of the 4 models

© National Bank of Belgium ROC areas of the 4 models ModelArea under the ROC curve NBB0.876 Model Model Z-score0.779 Area of model with no discriminatory power = 0.5 Area of model with acceptable discriminatory power > 0.7 Area of model with perfect discriminatory power = 1.0

© National Bank of Belgium ROC areas of selected combinations NBB (N) = ; Model 1 (M1) = ; Model 2 (M2) = 0.833; Z-score (Z) = CombinationMin.Max.MedianMean N - M N - M N - Z N - M1 - M N - M1 - Z N - M2 - Z N - M1- M2 - Z

© National Bank of Belgium ROC areas of selected combinations NBB (N) = ; Model 1 (M1) = ; Model 2 (M2) = 0.833; Z-score (Z) = CombinationMin.Max.MedianMean N - M N - M N - Z N - M1 - M N - M1 - Z N - M2 - Z N - M1- M2 - Z

© National Bank of Belgium ROC areas of selected combinations NBB (N) = ; Model 1 (M1) = ; Model 2 (M2) = 0.833; Z-score (Z) = CombinationMin.Max.MedianMean N - M N - M N - Z N - M1 - M N - M1 - Z N - M2 - Z N - M1- M2 - Z

© National Bank of Belgium ROC areas of selected combinations NBB (N) = ; Model 1 (M1) = ; Model 2 (M2) = 0.833; Z-score (Z) = CombinationMin.Max.MedianMean N - M N - M N - Z N - M1 - M N - M1 - Z N - M2 - Z N - M1- M2 - Z

© National Bank of Belgium ROC areas of selected combinations NBB (N) = ; Model 1 (M1) = ; Model 2 (M2) = 0.833; Z-score (Z) = CombinationMin.Max.MedianMean Z - N Z - M Z - M

© National Bank of Belgium ROC areas of 9 possible internal ratings systems (NBB model) Number of classes Mapping of firms based on Vendor model distribution Moody's distribution Equal distribution Note: NBB continuous credit score has an ROC area of

© National Bank of Belgium Conclusion  High disagreements rates between models: model choice can have a significant impact on loan pricing and origination decisions.  High power of each model: the definition of failure as well as the statitical method used by the models may not matter as much as one would have expected.  Larger differences between differing combinations of models than between differing internal rating systems.

© National Bank of Belgium