1. Managing model risk through effective governance
Julian Herbert
Managing model risk through effective governance
10th September 2013 #

2. Contents A view on selected topics in model governance:
Model governance - When are governance and model risk management structures effective?
Pricing and Risk model validation
The role of internal audit in model risk #

3. 1. Model Governance This section will cover:
Should Model Risk be managed as a separate risk discipline?
Who should “own” model risk? What does model ownership mean?
Will a model validation and approval fully address model risk?
The impact of organisational structure and culture on model risk #

4. (1a) Should Model Risk be a separate EWRM driver?
Model Risk impacts many risk disciplines…
CREDIT RISK
MARKET RISK
FINANCIAL REPORTING RISK
CAPITAL RISK
LIQUIDITY RISK
OPERATIONAL
RISK… 
MODEL RISK
…or can be considered as a separate risk discipline
CREDIT RISK
MARKET RISK
MODEL RISK
FINANCIAL REPORTING RISK
CAPITAL RISK
LIQUIDITY RISK
OPERATIONAL
RISK
…or a subrisk of operational risk #

5. (1a) Should Model Risk be a separate EWRM driver?
A similar questions also apply to Risk Reporting and Organisational structures –
Should model risk have a separate reporting line within Risk?
Should model risk be reported as a separate discipline to Group and Board Risk Committees?
Benefits
Can help to bring model risk “out into the open”
Increase communication and awareness of model risk at senior levels, and hence can increase the focus on managing the risks
Potential Pitfalls
Risk of detachment of model risks from the risk management or financial reporting process that the modelling relates to
Risk that model risk can be seen as a separate, “techy” discipline only
Risk that end-to-end model risk is not felt to be owned and managed by the relevant risk discipline or function (e.g. market, credit, finance) #

6. (1b) Who “owns” a model anyway?
Clear ownership is important for effective governance. But what do we mean by model “ownership”?
Possible model owners -
User?
Developer?
Risk owner / owner of the financial P&L?
What should Model Ownership incorporate?
Ownership of and responsibility for methodologies used?
Ownership of and responsibility for the decisions on how the model is used?
Ownership of end-to-end model risk? i.e. including e.g. Implementation, usage, monitoring, data quality
Potential Pitfalls
Aspects of model risk do not have a clear owner, and so are not managed for the purpose of model risk e.g. implementation, upstream data quality.
Conclusion: Ownership of end-to-end model risk need to be clearly defined, so that responsibilities for model risk management are clear and executed effectively. #

7. (1b). Responsibilities should be clear for all elements of model
(1b) Responsibilities should be clear for all elements of model risk management
Responsibility:
Methodology and Development
Model maintenance / operation
Usage
Monitoring
Data quality
Model Implementation IT
infrastructure
Policy owner
1st line /
“doing” & owning risk
2nd line /
Independent review
Approval
(More than one party?)
3rd line /
Independent assurance
Model maintenance / operation can include e.g. calibrations; Usage can include e.g. model reserving / risk management overlays, as well as day-to-day operation #

8. (1c). What does a model validation cover
(1c) What does a model validation cover? Who needs to “approve” the model?
Who needs to approve models? Is it clear which aspect of the model is being approved? Is approval always based on model validation results only?
Actions are typically put in place to address model issues arising from validations – restrictions or model enhancement plans.
Do these feed back effectively into risk management? (eg reserving, strategy setting)
How are the timelines set against other priorities and competing resources?
Should validation include review of the current or planned model performance, usage monitoring processes, implementation processes?
Potential Pitfalls:
“Model approval is the end of the story” – Risk that once a model is approved, and therefore listed as “in governance” on the model inventory, its shortcomings, usage restrictions or performance monitoring may not receive attention required.
Conclusions
The focus should be on managing model risks, not simply validating & approving models
Model inventories need to incorporate an effective way of recording, tracking and reporting significant model risks and limitations #

9. (1d). Role of organisation structure and culture in Model Risk
(1d) Role of organisation structure and culture in Model Risk management
Communication across the organisation is critical -
A close alignment between all stakeholders is important (e.g. model users, developers, IT etc).
In particular, model functions need to engage themselves on risk or system change projects
Also need good alignment and effective communication between risk modelling and pricing modelling, e.g. so that model limitations are effectively incorporated into the risk management process, beyond valuation P&L, and vice-versa. For example:
Derivative pricing models < -- > Risks Not In VaR
Credit risk rating models < -- > CVA
Stress testing model reviews < -- > process for assessing stress positions against risk appetite
The strength of independent model validation and approval functions is key.
A clear escalation process for models risks that do not reach approval.
Senior management should promote support for and acceptance of the function across the organisation.
This relates to previous discussion on approach Org design and EWRM structure.
Potential pitfall – The risk that it is seen, or sees itself, as simply a technical review function making recommendations #

10. 2. Risk versus Pricing model validation
Whilst the principals should be the same, are there cross learnings that can be taken from validation of risk and pricing models? For example:
End-to-end model risk approach –
Corporate credit rating model usage reviews are often incorporated into credit model validations.
Should usage reviews get incorporated into pricing model re-validation, e.g. an assessment of whether usage / parameter restrictions have been complied with?
How model weaknesses are dealt with -
Restrictions are often used for pricing models – effective, specific restrictions can be harder to implement for risk models, but is there some learning risk model validation can take from monitoring of these restrictions to perform monitoring, e.g. of whether model weaknesses have been factored into the risk management process? #

11. 3. The role of internal audit
This section will cover:
Role of Internal Audit in the organisation structure
Regulatory requirements relating to internal audit and model risk
Audit approaches to model risk #

12. (3a) Audit organisation structure
The Chartered Institute of Internal Auditors recently published guidance on Effective Internal Audit in the Financial Services Sector (July 2013).
It says:
“The primary role of Internal Audit should be to help the Board and Executive Management to protect the assets, reputation and sustainability of the organisation.
It does this by assessing whether all significant risks are identified and appropriately reported by management and the Risk function to the Board and Executive Management; assessing whether they are adequately controlled; and by challenging Executive Management to improve the effectiveness of governance, risk management and internal controls.”
The “Three lines of defence model” is now commonly used in the industry for Risk Management across the organisation, but is not necessarily the only model.
In this model, internal audit is the third line of defence, independent from the other parts of the organsation. #

13. (3b) Regulators on internal audit and model risk
Chartered Institute of Internal Auditors (IIA) guidance does not specifically reference model risk, but says:
“Internal Audit should include within its scope the management of the organisation’s capital and liquidity risks”
BIPRU – e.g on use of internal VaR model – requirements are specific:
“At least once a year, a firm must conduct, as part of its regular internal audit process, a review of its risk management process. … This review must include, at a minimum:
(5) The process for approving risk pricing models and valuation systems used in front and back offices;
(10 The accuracy and appropriateness of volatility and correlation assumptions;
(12) The process employed to evaluate the VaR model’s accuracy, including the programme of backtesting
[& others]” #

14. (3b) Regulators on internal audit and model risk
A comparison to the Insurance industry illustrates the direction of travel for expectations on audit in model risk.
Solvency 2 is the proposed regime for Insurance capital adequacy.
The European confederation of Institutes of Internal Auditors position paper on the Role of Internal Audit in Solvency 2 states:
“In assessing the process for designing and implementing risk models, special attention should be paid to the control activities implemented for ensuring:
The adequacy of model documentation and internal validation procedure
Compliance with reporting requirements
The degree of inclusion of the different risks in the model
The embedding of the model in risk management
The integrity of the management processesing and information systems
Quality of data sources
The quality and accuracy of the model and of the “ex posit” controls
The quality of stress testing” #

15. (3b) OCC on internal audit and model risk
“A bank’s internal audit function should assess the overall effectiveness of the model risk management framework, including the framework’s ability to address both types of model risk described in Section III [Overview of Model Risk Management], for individual models and in the aggregate.”
“Internal audit's role is not to duplicate model risk management activities. Instead, its role is to evaluate whether model risk management is comprehensive, rigorous, and effective.” #

16. (3c) Audit coverage and approach
“Do you audit around or through the model?”
Outcome < > Process
Should audit “reperform”?
Potential pitfalls
Misalignment of expectations between internal audit and risk functions on the role and approach of internal audit.
Internal audit assurance either misses the “big picture” risks, or places over-reliance on validation functions
“Audit should confirm the process,
not the decisions”
“Internal Audit doesn’t do validations…” #

17. (3c) Audit coverage and approach
IIA guidance –
On outcomes and process:
“Internal Audit should evaluate the design and operating effectiveness of the organisation’s policies and processes.
As part of this evaluation, Internal Audit should consider whether the outcomes achieved by the implementation of these policies and processes are in line with the objectives, risk appetite and values of the organisation.” #

18. (3c) OCC on internal audit approach to model risk
Policies and policy compliance
“Internal audit should verify that acceptable policies are in place and that model owners and control groups comply with those policies.”
Validation
“Internal audit should also verify records of model use and validation to test whether validations are performed in a timely manner and whether models are subject to controls that appropriately account for any weaknesses in validation activities.
Internal audit also has an important role in ensuring that validation work is conducted properly and that appropriate effective challenge is being carried out.
It should evaluate the objectivity, competence, and organizational standing of the key validation participants, with the ultimate goal of ascertaining whether those participants have the right incentives to discover and report deficiencies.” #

19. (3c) OCC on internal audit approach to model risk
Model Inventory
“Accuracy and completeness of the model inventory should be assessed.”
Usage restrictions
“In addition, processes for establishing and monitoring limits on model usage should be evaluated. Internal audit should determine whether procedures for updating models are clearly documented, and test whether those procedures are being carried out as specified. Internal audit should check that model owners and control groups are meeting documentation standards, including risk reporting.”
Systems and data
“Additionally, internal audit should perform assessments of supporting operational systems and evaluate the reliability of data used by models.”
Potential Pitfalls
Internal audit approach does not meet expectations of regulators. This could include audit coverage, depth of testing, and the level of “audit risk” taken. #

20. (3c) Audit approach to model validation
IIA guidance –
On Internal Audit interaction with Risk Management and Finance
“In evaluating the effectiveness of internal controls and risk management processes, in no circumstances should Internal Audit rely exclusively on the work of Risk Management, Compliance or Finance. Internal Audit should always examine, for itself, an appropriate sample of the activities under review.”
Relevant for audit of model validation processes. #

21. (3c). COSO is the industry standard for assessing control
(3c) COSO is the industry standard for assessing control frameworks, and can be applied to model risk at any level
Under a high level COSO view of Market Risk – models are often a “risk assessment” control
Focus in on a view of the Modelling process – and validation is a monitoring or control activity over model risk
Focus further on a validation process – specific validation tests can be seen as control activities and risk assessment controls
Information
Communication
Control Environment
Risk Assessment
Control Activities
Monitoring #