1. Chapter 6: Model Assessment
6.1 Model Fit Statistics
6.2 Statistical Graphics
6.3 Adjusting for Separate Sampling
6.4 Profit Matrices

2. Chapter 6: Model Assessment
6.1 Model Fit Statistics
6.2 Statistical Graphics
6.3 Adjusting for Separate Sampling
6.4 Profit Matrices

3. Summary Statistics Summary
Prediction Type
Statistic
Accuracy/Misclassification
Profit/Loss
Inverse prior threshold
Decisions
ROC Index (concordance)
Gini coefficient
Rankings
Average squared error
SBC/Likelihood
Estimates
...

4. Summary Statistics Summary
Prediction Type
Statistic
Accuracy/Misclassification
Profit/Loss
Inverse prior threshold
Decisions
ROC Index (concordance)
Gini coefficient
Rankings
Average squared error
SBC/Likelihood
Estimates
...

5. Summary Statistics Summary
Prediction Type
Statistic
Accuracy/Misclassification
Profit/Loss
Inverse prior threshold
Decisions
ROC Index (concordance)
Gini coefficient
Rankings
Average squared error
SBC/Likelihood
Estimates

6. Comparing Models with Summary Statistics
This demonstration illustrates the use of the Model Comparison tool, which collects assessment information from attached modeling nodes and enables you to easily compare model performance measures.

7. Chapter 6: Model Assessment
6.1 Model Fit Statistics
6.2 Statistical Graphics
6.3 Adjusting for Separate Sampling
6.4 Profit Matrices

8. Statistical Graphics – ROC Chart
0.0
1.0
captured response fraction
(sensitivity)
false positive fraction
(1-specificity)
The ROC chart illustrates a tradeoff between a captured response fraction and a false positive fraction.
...

9. Statistical Graphics – ROC Chart
0.0
1.0
captured response fraction
(sensitivity)
false positive fraction
(1-specificity)
The ROC chart illustrates a tradeoff between a captured response fraction and a false positive fraction.
...

10. Statistical Graphics – ROC Chart
0.0
1.0
Each point on the ROC chart corresponds to a specific fraction of cases, ordered by their predicted value.
...

11. Statistical Graphics – ROC Chart
0.0
1.0
Each point on the ROC chart corresponds to a specific fraction of cases, ordered by their predicted value.
...

12. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
For example, this point on the ROC chart corresponds to the 40% of cases with the highest predicted values.
...

13. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
For example, this point on the ROC chart corresponds to the 40% of cases with the highest predicted values.
...

14. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
The y-coordinate shows the fraction of primary outcome cases captured in the top 40% of all cases.
...

15. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
The y-coordinate shows the fraction of primary outcome cases captured in the top 40% of all cases.
...

16. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
The x-coordinate shows the fraction of secondary outcome cases captured in the top 40% of all cases.
...

17. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
The x-coordinate shows the fraction of secondary outcome cases captured in the top 40% of all cases.
...

18. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
Repeat for all selection fractions.
...

19. Statistical Graphics – ROC Chart
0.0
1.0
top 40%
Repeat for all selection fractions.
...

20. Statistical Graphics – ROC Chart
0.0
1.0
weak model
strong model
...

21. Statistical Graphics – ROC Index
0.0
1.0
weak model
ROC Index < 0.6
strong model
ROC Index > 0.7
...

22. Comparing Models with ROC Charts
This demonstration illustrates the use of ROC charts to compare models.

23. Statistical Graphics – Response Chart
100%
cumulative percent response
50% 0%
percent selected
100%
The response chart shows the expected response rate for various selection percentages.
...

24. Statistical Graphics – Response Chart
50%
100% 0%
cumulative percent response
percent selected
The response chart shows the expected response rate for various selection percentages.
...

25. Statistical Graphics – Response Chart
50%
100% 0%
Each point on the response chart corresponds to a specific fraction of cases, ordered by their predicted values.
...

26. Statistical Graphics – Response Chart
50%
100% 0%
Each point on the response chart corresponds to a specific fraction of cases, ordered by their predicted values.
...

27. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
For example, this point on the response chart corresponds to the 40% of cases with the highest predicted values.
...

28. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
For example, this point on the response chart corresponds to the 40% of cases with the highest predicted values.
...

29. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
40%
The x-coordinate shows the percentage of selected cases.
...

30. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
40%
The x-coordinate shows the percentage of selected cases.
...

31. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
40%
The y-coordinate shows the percentage of primary outcome cases found in the top 40%.
...

32. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
40%
The y-coordinate shows the percentage of primary outcome cases found in the top 40%.
...

33. Statistical Graphics – Response Chart
50%
100% 0%
top 40%
40%
Repeat for all selection fractions.
...

35. 6.01 Poll
In practice, modelers often use several tools, sometimes both graphical and numerical, to choose a best model.  True  False
Type answer here

36. 6.01 Poll – Correct Answer
In practice, modelers often use several tools, sometimes both graphical and numerical, to choose a best model.  True  False
Type answer here

37. Comparing Models with Score Rankings Plots
This demonstration illustrates comparing models with Score Rankings plots.

38. Adjusting for Separate Sampling
This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.

39. Chapter 6: Model Assessment
6.1 Model Fit Statistics
6.2 Statistical Graphics
6.3 Adjusting for Separate Sampling
6.4 Profit Matrices

40. Outcome Overrepresentation
A common predictive modeling practice is to build models from a sample with a primary outcome proportion different from the original population.
...

41. Outcome Overrepresentation
A common predictive modeling practice is to build models from a sample with a primary outcome proportion different from the original population.
...

42. Separate Sampling
secondary outcome
primary outcome
Target-based samples are created by considering the primary outcome cases separately from the secondary outcome cases.
...

43. Separate Sampling
secondary outcome
primary outcome
Target-based samples are created by considering the primary outcome cases separately from the secondary outcome cases.
...

44. Separate Sampling Select some cases. Select all cases.
secondary outcome
primary outcome
Select some cases.
Select all cases.
...

45. Separate Sampling Select some cases. Select all cases.
secondary outcome
primary outcome
Select some cases.
Select all cases.
...

46. The Modeling Sample + Similar predictive power with smaller case count
− Must adjust assessment
statistics and graphics
− Must adjust prediction
estimates for bias
...

47. Adjusting for Separate Sampling (continued)
This demonstration illustrates how to adjust for separate sampling in SAS Enterprise Miner.

48. Creating a Profit Matrix
This demonstration illustrates how to create a profit matrix.

49. Chapter 6: Model Assessment
6.1 Model Fit Statistics
6.2 Statistical Graphics
6.3 Adjusting for Separate Sampling
6.4 Profit Matrices

50. Profit Matrices 15.14 -0.68 solicit ignore primary outcome secondary
primary
outcome
secondary
outcome
-0.68
profit distribution
for solicit decision

51. Profit Matrices 15.14 -0.68 solicit ignore primary outcome secondary
primary
outcome
secondary
outcome
-0.68
profit distribution
for solicit decision

52. Decision Expected Profits
solicit
ignore
15.14
primary
outcome
secondary
outcome
-0.68
Expected Profit Solicit = p1 – 0.68 p0
Expected Profit Ignore = 0
Choose the larger. ^
...

53. Decision Threshold 15.14 -0.68 solicit ignore primary outcome
primary
outcome
secondary
outcome
-0.68
decision threshold ^
p1 ≥ 0.68 /  Solicit ^
p1 < 0.68 /  Ignore

54. Average Profit 15.14 -0.68 solicit ignore primary outcome secondary
primary
outcome
secondary
outcome
-0.68
average profit
Average profit = (15.14NPS – 0.68 NSS ) / N
NPS = # solicited primary outcome cases
NSS = # solicited secondary outcome cases
N = total number of assessment cases

55. Evaluating Model Profit
This demonstration illustrates viewing the consequences of incorporating a profit matrix.

56. Viewing Additional Assessments
This demonstration illustrates several other assessments of possible interest.

57. Optimizing with Profit (Self-Study)
This demonstration illustrates optimizing your model strictly on profit.

58. Exercises
This exercise reinforces the concepts discussed previously.

59. Assessment Tools Review
Compare model summary statistics and statistical graphics.
Create decision data; add prior probabilities and profit matrices.
Tune models with average squared error or appropriate profit matrix.
Obtain means and other statistics on data source variables.