Comply and Compete: Model Management Best Practices Meet regulatory requirements in a way that helps you succeed Andrew Jennings, Chief Analytics Officer Neill Crossley, Principal Consultant

More Regulations, More Models, More at Stake Banking regulators are focused on soundness of decision making, capital adequacy and unlawful or unfair treatment of customers They’re scrutinizing how you use analytics to manage risk, measure capital reserve requirements, make customer decisions and ensure consistency across your operations 2 © 2014 Fair Isaac Corporation. Confidential.

More Regulations, More Models, More at Stake To compete for valuable customers, you’re deploying more models of greater complexity How well you manage your analytic resources and processes substantially affects your business performance 3 © 2014 Fair Isaac Corporation. Confidential.

Traditional Model Management Isn’t Up to the Job Model management challenges include: Tracking how and where models are being used Prioritizing model review and redevelopment Responding promptly to regulatory requests Understanding how models are impacting decisions Avoiding suboptimal decisions caused by model performance degradation Devoting modeler time to audits vs. analytic innovations 4 © 2014 Fair Isaac Corporation. Confidential.

Centralized, Automated Model Management Comply and compete with these best practices for model management: Ensure segmentation transparency Choose the right type of model Validate model effectiveness Track performance Document thoroughly 1 2 3 4 5 5 © 2014 Fair Isaac Corporation. Confidential.

Ensure Segmentation Transparency 1 Ensure Segmentation Transparency Regulators Want to Know: How you separated portfolios into subpopulations, and that your segmentation supports your business objectives Model Management Best Practices: Use empirical methods to define segments and prove they’re homogeneous and actionable. If using machine learning to discover subpopulations, don’t give up transparency Validate/track segmented models as a whole (in addition to individual segments)

Choose the Right Type of Model 2 Choose the Right Type of Model Regulators Want to Know: that you’re using the right model for your data and the decision you’re making Model Management Best Practices: Transparent models can be explained and defended Model behavior makes sense for the business Predictive characteristics align with regulations or customer concerns Model can be validated for each population Determine how model implementation will impact decision strategies

Choose the Right Type of Model 2 Choose the Right Type of Model This scorecard excerpt shows how points are assigned for different values within a category of information, and how each factor relates to an individual’s score: Scorecards are a highly transparent and interpretable type of model

Validate Model Effectiveness 3 Validate Model Effectiveness Regulators Want to Know: that you’re validating regularly using consistent, reliable methods and that you understand your process Model Management Best Practices: Use repeatable, consistent processes across model types Validate in realistic context Include reports to assess stability and show that model rank orders risk by score Flag models for review when stability or performance metrics decline Improve governance through independent supervisory review

Validate Model Effectiveness 3 Validate Model Effectiveness Use a variety of measures to assess model performance and stability The Population Stability Index is a common measure of score stability, while Characteristic Analysis reports identify shifts within each model characteristic. Divergence, K-S and ROC are three popular measures of a score’s predictive power.

4 Track Performance Regulators Want to Know: that you are monitoring models regularly, and are recalibrating or rebuilding when performance degrades Model Management Best Practices: Scorecard tracking should include values for all model inputs, resulting score and subsequent performance to truly understand business model performance. Get alerts when performance drops below predefined ranges

4 Track Performance A bird’s-eye view of performance across one or many models

5 Document Thoroughly Regulators Want to Know: you have adequate processes in place for compliance documentation and oversight Model Management Best Practices: Inventory every model in your operational environment Catalog its purpose, usage, restrictions, input types/sources Capture enough detail and explanation that someone unfamiliar with the model can quickly understand How it operates Key assumptions Limitations

Deploy a Model Management Foundation Governance The basic requirements of centralized, automated model management include: Complete model inventory Analytic validation of models executed on a scheduled basis Automated report generation and notification of availability Web-based dashboard for assessing health of all models User-defined validation triggers and alert rules Cumulative reviewer annotation history at performance level for audit/tracking On-demand report generation, including ability to query custom subsegments

Augment with Professional Development Governance Next, you need the ability to construct or consume models, and manage the development lifecycle: Complete IDE for rapidly building and testing models or consuming those built externally Complete lifecycle management with full IT governance and white-box transparency Central repository for rules/models and variable library Comprehensive diagnostics and reporting Open, extensible scripting interfaces

Take it to the Next Level Governance Finally, you need to determine how your models are performing – and what you can be doing better: Simulation of multiple “What ifs?” Optimization pinpointing the best customer decision given all objectives and constraints Tracking of key performance indicators

Introducing the FICO® Model Central™ Solution Governance The three tiers of Model Central offer a scalable path for organizations of all sizes: Foundation Tier provides core capabilities for validation, monitoring, management reporting, alerting and administration Professional Development Tier adds model development, tuning, consumption and rapid deployment Advanced Decisioning Tier adds decision modeling and optimization

Summary You’re developing more complex models to drive business performance and customer engagement Regulations mandate more model management diligence Balancing business results while staying compliant requires model lifecycle management You need the people, processes and technology in place to make it happen FICO can help you comply and compete!

Make Compliance Your Competitive Advantage Better model management is the foundation for success Learn more: Download the white paper on this topic: Insights #55: Comply and Compete: Model Management Best Practices Check out our blog: Banking Analytics Blog