Chapter 6: Model Assessment 6.1 Model Fit Statistics 6.2 Statistical Graphics 6.3 Adjusting for Separate Sampling 6.4 Profit Matrices
Chapter 6: Model Assessment 6.1 Model Fit Statistics 6.2 Statistical Graphics 6.3 Adjusting for Separate Sampling 6.4 Profit Matrices
Summary Statistics Summary Prediction Type Statistic Accuracy/Misclassification Profit/Loss Inverse prior threshold Decisions ROC Index (concordance) Gini coefficient Rankings Average squared error SBC/Likelihood Estimates ...
Summary Statistics Summary Prediction Type Statistic Accuracy/Misclassification Profit/Loss Inverse prior threshold Decisions ROC Index (concordance) Gini coefficient Rankings Average squared error SBC/Likelihood Estimates ...
Summary Statistics Summary Prediction Type Statistic Accuracy/Misclassification Profit/Loss Inverse prior threshold Decisions ROC Index (concordance) Gini coefficient Rankings Average squared error SBC/Likelihood Estimates
Comparing Models with Summary Statistics This demonstration illustrates the use of the Model Comparison tool, which collects assessment information from attached modeling nodes and enables you to easily compare model performance measures.
Chapter 6: Model Assessment 6.1 Model Fit Statistics 6.2 Statistical Graphics 6.3 Adjusting for Separate Sampling 6.4 Profit Matrices
Statistical Graphics – ROC Chart 0.0 1.0 captured response fraction (sensitivity) false positive fraction (1-specificity) The ROC chart illustrates a tradeoff between a captured response fraction and a false positive fraction. ...
Statistical Graphics – ROC Chart 0.0 1.0 captured response fraction (sensitivity) false positive fraction (1-specificity) The ROC chart illustrates a tradeoff between a captured response fraction and a false positive fraction. ...
Statistical Graphics – ROC Chart 0.0 1.0 Each point on the ROC chart corresponds to a specific fraction of cases, ordered by their predicted value. ...
Statistical Graphics – ROC Chart 0.0 1.0 Each point on the ROC chart corresponds to a specific fraction of cases, ordered by their predicted value. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% For example, this point on the ROC chart corresponds to the 40% of cases with the highest predicted values. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% For example, this point on the ROC chart corresponds to the 40% of cases with the highest predicted values. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% The y-coordinate shows the fraction of primary outcome cases captured in the top 40% of all cases. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% The y-coordinate shows the fraction of primary outcome cases captured in the top 40% of all cases. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% The x-coordinate shows the fraction of secondary outcome cases captured in the top 40% of all cases. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% The x-coordinate shows the fraction of secondary outcome cases captured in the top 40% of all cases. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% Repeat for all selection fractions. ...
Statistical Graphics – ROC Chart 0.0 1.0 top 40% Repeat for all selection fractions. ...
Statistical Graphics – ROC Chart 0.0 1.0 weak model strong model ...
Statistical Graphics – ROC Index 0.0 1.0 weak model ROC Index < 0.6 strong model ROC Index > 0.7 ...
Comparing Models with ROC Charts This demonstration illustrates the use of ROC charts to compare models.
Statistical Graphics – Response Chart 100% cumulative percent response 50% 0% percent selected 100% The response chart shows the expected response rate for various selection percentages. ...
Statistical Graphics – Response Chart 50% 100% 0% cumulative percent response percent selected The response chart shows the expected response rate for various selection percentages. ...
Statistical Graphics – Response Chart 50% 100% 0% Each point on the response chart corresponds to a specific fraction of cases, ordered by their predicted values. ...
Statistical Graphics – Response Chart 50% 100% 0% Each point on the response chart corresponds to a specific fraction of cases, ordered by their predicted values. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% For example, this point on the response chart corresponds to the 40% of cases with the highest predicted values. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% For example, this point on the response chart corresponds to the 40% of cases with the highest predicted values. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% 40% The x-coordinate shows the percentage of selected cases. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% 40% The x-coordinate shows the percentage of selected cases. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% 40% The y-coordinate shows the percentage of primary outcome cases found in the top 40%. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% 40% The y-coordinate shows the percentage of primary outcome cases found in the top 40%. ...
Statistical Graphics – Response Chart 50% 100% 0% top 40% 40% Repeat for all selection fractions. ...
6.01 Poll In practice, modelers often use several tools, sometimes both graphical and numerical, to choose a best model. True False Type answer here
6.01 Poll – Correct Answer In practice, modelers often use several tools, sometimes both graphical and numerical, to choose a best model. True False Type answer here
Comparing Models with Score Rankings Plots This demonstration illustrates comparing models with Score Rankings plots.
Adjusting for Separate Sampling This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.
Chapter 6: Model Assessment 6.1 Model Fit Statistics 6.2 Statistical Graphics 6.3 Adjusting for Separate Sampling 6.4 Profit Matrices
Outcome Overrepresentation A common predictive modeling practice is to build models from a sample with a primary outcome proportion different from the original population. ...
Outcome Overrepresentation A common predictive modeling practice is to build models from a sample with a primary outcome proportion different from the original population. ...
Separate Sampling secondary outcome primary outcome Target-based samples are created by considering the primary outcome cases separately from the secondary outcome cases. ...
Separate Sampling secondary outcome primary outcome Target-based samples are created by considering the primary outcome cases separately from the secondary outcome cases. ...
Separate Sampling Select some cases. Select all cases. secondary outcome primary outcome Select some cases. Select all cases. ...
Separate Sampling Select some cases. Select all cases. secondary outcome primary outcome Select some cases. Select all cases. ...
The Modeling Sample + Similar predictive power with smaller case count − Must adjust assessment statistics and graphics − Must adjust prediction estimates for bias ...
Adjusting for Separate Sampling (continued) This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.
Creating a Profit Matrix This demonstration illustrates how to create a profit matrix.
Chapter 6: Model Assessment 6.1 Model Fit Statistics 6.2 Statistical Graphics 6.3 Adjusting for Separate Sampling 6.4 Profit Matrices
Profit Matrices 15.14 -0.68 solicit ignore primary outcome secondary primary outcome secondary outcome -0.68 profit distribution for solicit decision
Profit Matrices 15.14 -0.68 solicit ignore primary outcome secondary primary outcome secondary outcome -0.68 profit distribution for solicit decision
Decision Expected Profits solicit ignore 15.14 primary outcome secondary outcome -0.68 Expected Profit Solicit = 15.14 p1 – 0.68 p0 Expected Profit Ignore = 0 Choose the larger. ^ ...
Decision Threshold 15.14 -0.68 solicit ignore primary outcome primary outcome secondary outcome -0.68 decision threshold ^ p1 ≥ 0.68 / 15.82 Solicit ^ p1 < 0.68 / 15.82 Ignore
Average Profit 15.14 -0.68 solicit ignore primary outcome secondary primary outcome secondary outcome -0.68 average profit Average profit = (15.14NPS – 0.68 NSS ) / N NPS = # solicited primary outcome cases NSS = # solicited secondary outcome cases N = total number of assessment cases
Evaluating Model Profit This demonstration illustrates viewing the consequences of incorporating a profit matrix.
Viewing Additional Assessments This demonstration illustrates several other assessments of possible interest.
Optimizing with Profit (Self-Study) This demonstration illustrates optimizing your model strictly on profit.
Exercises This exercise reinforces the concepts discussed previously.
Assessment Tools Review Compare model summary statistics and statistical graphics. Create decision data; add prior probabilities and profit matrices. Tune models with average squared error or appropriate profit matrix. Obtain means and other statistics on data source variables.