1. Project

2. Overview Project details Open class to begin project work
Fitness function
Equal error costs
Unequal error costs
Open class to begin project work

4. Project details
Objective (or fitness function)?

5. Equal Error Costs
If type 1 and type 2 error costs are the same we can simply minimize misclassification rates (on hold out set)

6. These two models have the same misclassification rate:
Unequal Error Costs
In many real life scenarios the costs of type 1 and type 2 error costs are not the same…
These two models have the same misclassification rate:
Which one do you prefer?

7. Example Type 2 error Type 1 error
MODEL A
MODEL B
Example
Type 2 error
expected cost of incorrectly classifying someone with cancer (actual YES) as not having cancer (predict NO) is $10000  
expected cost of incorrectly classifying someone without cancer (actual NO) as having cancer (predict YES) is $500
Type 1 error

8. Example Instead of minimizing misclassification rate...
MODEL A
MODEL B
Example
Type 2 error cost = 10000
Type 1 error cost = 500
Revenue for correctly identifying cancer exists = 4000
Revenue for correctly identifying cancer doesn’t exist = 100
Instead of minimizing misclassification rate... 
let’s maximize expected profit
revenue
correctly
predict 1
revenue
correctly
predict 0
#correct
predict 1
#correct
predict 0
cost of
type 1
error
cost of
type 2
error x + x -
#type 1
errors
#type 2
errors x - x

9. Example Instead of minimizing misclassification rate...
MODEL A
MODEL B
Example
Type 2 error cost = 10000
Type 1 error cost = 500
Revenue for correctly identifying cancer exists = 4000
Revenue for correctly identifying cancer doesn’t exist = 100
Instead of minimizing misclassification rate... 
let’s maximize expected profit
Model A:
revenue
correctly
predict 1
revenue
correctly
predict 0
4000
#correct
predict 1 5
100
#correct
predict 0 5
cost of
type 1
error
cost of
type 2
error 1 x + x -
500
#type 1
errors 9
10000
#type 2
errors x - x
= 6000

10. Example Instead of minimizing misclassification rate...
MODEL A
MODEL B
Example
Type 2 error cost = 10000
Type 1 error cost = 500
Revenue for correctly identifying cancer exists = 4000
Revenue for correctly identifying cancer doesn’t exist = 100
Instead of minimizing misclassification rate... 
let’s maximize expected profit
Model A:
revenue
correctly
predict 1
revenue
correctly
predict 0
4000
#correct
predict 1 5
100
#correct
predict 0 5
cost of
type 1
error
cost of
type 2
error x + x -
500
#type 1
errors 9
10000
#type 2
errors 1 x - x
= 6000
Model B:
revenue
correctly
predict 0
4000
#correct
predict 1 5
cost of
type 1
error 5
100
#correct
predict 0
cost of
type 2
error x + -
500
#type 1
errors 1 x
10000
#type 2
errors 9 x - x =

11. Project Work