1. Lecture Notes 9 Prediction Limits
Zhangxi Lin
ISQS
Texas Tech University
Note: Most slides in this file are sourced from Course Notes

2. Section 3.1
Profit Variability

3. Random Profit Consequences
Primary Decision
random
deterministic
Profiti = Yi - costsi y
Profit

4. Conditional Profits Profiti = Yi - costsi Primary Decision random
deterministic
Profiti = Yi - costsi y y

5. Expected Profit Consequence
Primary Decision
random
deterministic
Profiti = Yi - costsi
Primary
Outcome
Secondary
d(y|xi)
EPCi = E(Yi) - costsi y
= p(xi)·D(xi) - costsi ^ y

6. Predicted Profit Plots
N=96,367
Scaled
Total
Profit
Overall
Average
EPCi Σ 10 20 30 40 50 60 70 80 90
% selected
$10,000
$12,000
$14,000
$16,000
$8,000
$6,000
$4,000

7. Predicted and Observed Profit Plots
Overall
Average
Profit
EPCi Σ 10 20 30 40 50 60 70 80 90
% selected
OPi (training)
$10,000
$12,000
$14,000
$16,000
$8,000
$6,000
$4,000
Scaled
Total

8. Predicted and Observed Profit Plots
Overall
Average
Profit
EPCi Σ 10 20 30 40 50 60 70 80 90
% selected
OPi (training)
OPi (validation)
$10,000
$12,000
$14,000
$16,000
$8,000
$6,000
$4,000
Scaled
Total

9. Predicted and Observed Profit Plots
Overall
Average
Profit
Scaled
Total
Profit
Sum of independent r.v. (not i.d.)
N=96,367
Lyapunov conditions  var(Σ)=Σvari
$10,000
$12,000
$14,000
$16,000
$8,000
$6,000
$4,000
EPCi Σ
OPi (training) Σ
OPi (validation) Σ 10 20 30 40 50 60 70 80 90
% selected

10. Beyond Expectations: Variability in Profit
Profiti = Yi - costsi
EPCi = E(Yi) - costsi ^ ^
= p(xi)·D(xi) - costsi
Var( Profiti ) = Var (Yi)
= E(Yi2) – (EYi)2

11. Beyond Expectations: Variability in Profit
Profiti = Yi - costsi
E( Profiti ) = E(Yi) - costsi
= p(xi)·D(xi) - costsi ^
Var( Profiti ) = Var (Yi)
= pi·[E(Di2)-Di2·pi]
need to estimate

12. Some Second Moment Estimates
 Normal* σ2 + Di2
 Poisson Di + Di2
 Gamma Di2 ·(1+1/σshape)
 Lognormal Di2 ·exp(σ2)
Distribution Estimate ^

13. Some Profit Variance Estimates
Distribution Estimate ^ ^ ^ ^ ^
 Normal* pi·Di2 [ 1–pi + σ2/Di2 ] ^ ^ ^ ^
 Poisson pi·Di2 [ 1–pi + 1/Di ] ^ ^ ^ ^
 Gamma pi·Di2 [ 1–pi + 1/ σshape ] ^ ^ ^ ^
 Lognormal pi·Di2 [ 1–pi + exp(σ2)–1 ]

14. Profit Plots with Tolerance Limits
Overall
Average
Profit
EPCi Σ 10 20 30 40 50 60 70 80 90
% selected
OPi
EPCi ± 2 √Σ Var(Profiti)
$10,000
$12,000
$14,000
$16,000
$8,000
$6,000
$4,000
Scaled
Total

15. Profit Plots with Tolerance Limits
Overall
Average
Profit
EPCi Σ 10 20 30 40 50 60 70 80 90
% selected
OPi (training)
OPi (validation)
$10,000
$12,000
$14,000
$16,000
$8,000
$6,000
$4,000
OPi (score)
Scaled
Total

16. for “solicit everyone” model
1998 KDD-Cup Results 1. 2. 3. 4. 5. 6. 7. 8. 9.
10.
$14,712
14,662
13,954
13,825
13,794
13,598
13,040
12,298
11,423
11,276
Total
Profit
Rank
$0.153
0.152
0.145
0.143
0.141
0.135
0.128
0.119
0.117
Overall
Avg. Profit
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
$ 10,720
10,706
10,112
10,049
9,741
9,464
5,683
5,484
1,925
1,706
$ 0.111
0.111
0.105
0.104
0.101
0.098
0.059
0.057
0.020
0.018
$10,560
$ 0.110
Total profit
Avg. profit
for “solicit everyone” model

17. Prediction Limits: The Good
Quantifies uncertainty in expected
profit estimates
Lends perspective to model
comparisons
Gives insight into model fit
$ ± $

18. Prediction Limits: The Bad
Does not account for model
variability
Skewed by outlying predictions $

19. Model Variability Σ Same Model Specification Same Training Data
Overall
Average
Profit 10 20 30 40 50 60 70 80 90
% selected
Same Model Specification
Same Training Data
Different Parameter Initialization
EPCi Σ

20. Prediction Limits: The Ugly
Requires scaling adjustments for
sampling
Surprises analysts/management
¡¡¡

21. Scaling Prediction Limits (More CLT)
$100,000
$120,000
$140,000
$160,000
$80,000
$60,000
$40,000
N=963,670
Overall
Average
Profit 10 20 30 40 50 60 70 80 90
% selected
Overall Average Profit
Limits Scale by 1/√N
Total Profit
Limits Scale by √N
Scaled
Total