1. Model Governance Industry Evolution Beyond Model Accuracy
Bill Cember, FSA, MAAA

2. Disclaimer
Content in this presentation represent the views of the author. They do not necessarily reflect the views of ASNY, Prudential, the SOA, or other organizations

3. Agenda Models and Model Governance Guiding Principles – FAST
Cast Studies
Takeaways

4. Models and Model Governance
“A model is a quantitative method, system, or approach used to calculate or estimate value or risk that impact Prudential’s financial statements and/or assist in decision- making. A model consists of three fundamental components: (a) inputs and/or assumptions, (b) calculation routines, and (c) outputs and their adjustments. Models transform given inputs and/or assumptions into outputs with some degree of complexity and uncertainty.”
- Prudential’s Model Risk Management Group

5. Models and Model Governance
“[T]he term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information.”
- SR-11

6. Guiding Principles - FAST
Flexible
Accurate
Standardized
Testable

7. Guiding Principles - Flexible
Flexibility = How easy it is to change your model when requirements change
Why it’s important:
If it’s hard to change a model, then the model won’t get improved.
More complicated changes to models not only take up more resources but are also more error prone
When models are hard to change, they will more easily brea inadvertantly

8. Guiding Principles - Flexible
Model Component
Key Questions
Inputs & Assumptions
How easy is it to switch between assumption sets for various model functionalities?
Are the assumptions being used by the model easily accessible (and perhaps externalized) or scattered throughout the model?
Calculation Engine
How easy is it to change a calculation methodology in the model?
Is there a standard way to add new products to the model? Can existing pieces of the model be leveraged?
Outputs & External Adjustments
How easy is it to change where outputs are stored/named?
If model methodology changes, is it easily apparent how/if external adjustments should change?

9. Guiding Principles - Standardized
Standards = Common language that users and developers of the model can speak
Less Important what the standard is than a standard exists

10. Guiding Principles - Standardized
Model Component
Key Questions
Inputs & Assumptions
Where are inputs and assumptions stored? What format do they take?
Are inputs all really inputs? Are there cases where when a model gets updated the model needs to be manually changed?
Calculation Engine
Are there “code conventions” for the model?
Are common calculations done in common ways across models?
Outputs & External Adjustments
Where are outputs stored? What format do they take?
Is there a common hand-off process from output to customer?

11. Guiding Principles - Testable
Relevant Testing
Practice
Modeler should have ability to “drill-down” as needed when analyzing results
-Cell testing
-Change/rollforward analysis
-Output results at the most granular level possible
-Use pre-defined aggregations for reporting, not analysis
Changes to the model should have predictable impacts
-Implement model changes in small, testable pieces
-Changes should be explainable before models are ran in production

12. Guiding Principles - Testable
Relevant Testing
Practice
Modelers should have the ability to see intermediate steps in calculations
-Unit testing
-Cell testing
-Build models to output intermediate calculations or have the ability to output intermediate calculations as needed
-When working with “black box” actuarial software, utilize external validation tools
Modelers should have the ability to do ad hoc runs with different assumptions or methodologies.
-Change/rollforward analysis
-Parallel testing
-Build model in a flexible way

13. Guiding Principles - Accuracy
Accuracy is a line not a dot
New functionality can work when it is initially implemented but break at later dates when new “cases” arise
Simple ≠ Immune from errors
Developers should expect and be expected to make errors. All changes no matter how simple should be independently tested before being brought into production

14. Case Study
Your inforce has repeat policy numbers (that are legitimately different policies). How do you handle this?

15. Case Study
You want to run sensitivities on your model assumptions. Which of the approaches would you take to build this out?
Don’t build out anything. Run the model with new assumption tables ad hoc
Build out new assumption tables for the sensitivities. Develop model functionality to toggle which assumptions you want to use
Rebuild (if necessary) your original assumption tables such that sensitivities can be inputted as multipliers to the original tables. Develop model functionality to toggle whether to apply multipliers to the assumptions

16. Takeaways
Accuracy is a line not a dot. A model being right today does not mean it will be right tomorrow.
Model governance is more than just testing.
Develop your models in a FAST way to make your models more robust to change and easily testable to ensure changes are correct