1. Comply and Compete: Model Management Best Practices
Meet regulatory requirements in a way that helps you succeed
Andrew Jennings, Chief Analytics Officer
Neill Crossley, Principal Consultant

2. More Regulations, More Models, More at Stake
Banking regulators are focused on soundness of decision making, capital adequacy and unlawful or unfair treatment of customers
They’re scrutinizing how you use analytics to manage risk, measure capital reserve requirements, make customer decisions and ensure consistency across your operations 2
© 2014 Fair Isaac Corporation. Confidential.

3. More Regulations, More Models, More at Stake
To compete for valuable customers, you’re deploying more models of greater complexity
How well you manage your analytic resources and processes substantially affects your business performance 3
© 2014 Fair Isaac Corporation. Confidential.

4. Traditional Model Management Isn’t Up to the Job
Model management challenges include:
Tracking how and where models are being used
Prioritizing model review and redevelopment
Responding promptly to regulatory requests
Understanding how models are impacting decisions
Avoiding suboptimal decisions caused by model performance degradation
Devoting modeler time to audits vs. analytic innovations 4
© 2014 Fair Isaac Corporation. Confidential.

5. Centralized, Automated Model Management
Comply and compete with these best practices for model management:
Ensure segmentation transparency
Choose the right type of model
Validate model effectiveness
Track performance
Document thoroughly 1 2 3 4 5 5
© 2014 Fair Isaac Corporation. Confidential.

6. Ensure Segmentation Transparency1
Ensure Segmentation Transparency
Regulators Want to Know:
How you separated portfolios into subpopulations, and that your segmentation supports your business objectives
Model Management Best Practices:
Use empirical methods to define segments and prove they’re homogeneous and actionable. If using machine learning to discover subpopulations, don’t give up transparency
Validate/track segmented models as a whole (in addition to individual segments)

7. Choose the Right Type of Model2
Choose the Right Type of Model
Regulators Want to Know:
that you’re using the right model for your data and the decision you’re making
Model Management Best Practices:
Transparent models can be explained and defended
Model behavior makes sense for the business
Predictive characteristics align with regulations or customer concerns
Model can be validated for each population
Determine how model implementation will impact decision strategies

8. Choose the Right Type of Model2
Choose the Right Type of Model
This scorecard excerpt shows how points are assigned for different values within a category of information, and how each factor relates to an individual’s score:
Scorecards are a highly transparent and interpretable type of model

9. Validate Model Effectiveness3
Validate Model Effectiveness
Regulators Want to Know:
that you’re validating regularly using consistent, reliable methods and that you understand your process
Model Management Best Practices:
Use repeatable, consistent processes across model types
Validate in realistic context
Include reports to assess stability and show that model rank orders risk by score
Flag models for review when stability or performance metrics decline
Improve governance through independent supervisory review

10. Validate Model Effectiveness3
Validate Model Effectiveness
Use a variety of measures to assess model performance and stability
The Population Stability Index is a common measure of score stability, while Characteristic Analysis reports identify shifts within each model characteristic. Divergence, K-S and ROC are three popular measures of a score’s predictive power.

11. 4 Track Performance Regulators Want to Know:
that you are monitoring models regularly, and are recalibrating or rebuilding when performance degrades
Model Management Best Practices:
Scorecard tracking should include values for all model inputs, resulting score and subsequent performance to truly understand business model performance.
Get alerts when performance drops below predefined ranges

12. 4
Track Performance
A bird’s-eye view of performance across one or many models

13. 5 Document Thoroughly Regulators Want to Know:
you have adequate processes in place for compliance documentation and oversight
Model Management Best Practices:
Inventory every model in your operational environment Catalog its purpose, usage, restrictions, input types/sources
Capture enough detail and explanation that someone unfamiliar with the model can quickly understand
How it operates
Key assumptions
Limitations

14. Deploy a Model Management Foundation
Governance
The basic requirements of centralized, automated model management include:
Complete model inventory
Analytic validation of models executed on a scheduled basis
Automated report generation and notification of availability
Web-based dashboard for assessing health of all models
User-defined validation triggers and alert rules
Cumulative reviewer annotation history at performance level for audit/tracking
On-demand report generation, including ability to query custom subsegments

15. Augment with Professional Development
Governance
Next, you need the ability to construct or consume models, and manage the development lifecycle:
Complete IDE for rapidly building and testing models or consuming those built externally
Complete lifecycle management with full IT governance and white-box transparency
Central repository for rules/models and variable library
Comprehensive diagnostics and reporting
Open, extensible scripting interfaces

16. Take it to the Next Level
Governance
Finally, you need to determine how your models are performing – and what you can be doing better:
Simulation of multiple “What ifs?”
Optimization pinpointing the best customer decision given all objectives and constraints
Tracking of key performance indicators

17. Introducing the FICO® Model Central™ Solution
Governance
The three tiers of Model Central offer a scalable path for organizations of all sizes:
Foundation Tier provides core capabilities for validation, monitoring, management reporting, alerting and administration
Professional Development Tier adds model development, tuning, consumption and rapid deployment
Advanced Decisioning Tier adds decision modeling and optimization

18. Summary
You’re developing more complex models to drive business performance and customer engagement
Regulations mandate more model management diligence
Balancing business results while staying compliant requires model lifecycle management
You need the people, processes and technology in place to make it happen
FICO can help you comply and compete!

19. Make Compliance Your Competitive Advantage
Better model management is the foundation for success
Learn more:
Download the white paper on this topic:
Insights #55: Comply and Compete: Model Management Best Practices
Check out our blog:
Banking Analytics Blog