1. Loss

2. Minimum Expected Loss/Risk
If we want to consider more than zero-one loss, then we need to define a loss matrix with elements Lkj specifying the penalty associated with assigning a pattern belonging to class Ck as class Cj (i.e. Read kj as k-> j or ‘’k classified as j’’)
Example: classify medical images as ‘cancer’ or ‘normal’
Then, to compute the minimum expected loss, we need to look at the concept of expected value.
Decision
Truth

3. Expected Value
The expected value of a function f(x), where x has the probability density/mass p(x) is
Discrete Continuous
For a finite set of data points x1 , ,xn, drawn from the distribution p(x), the expectation can be approximated by the average over the data points:

7. Reminder: Minimum Misclassification Rate
Illustration with more
general distributions,
showing different error areas.

8. Minimum Expected Loss/Risk
For two classes:
Expected loss= ∫R2 L12p(x,C1)dx + ∫R1 L21p(x,C2)dx
In general:
Regions are chosen to minimize:

9. Reject Option

10. Loss for Regression

11. Regression
For regression, the problem is a bit more complicated and we also need the concept of conditional expectation.
E[t|x] = S p(t|x) t(x) t

12. MultiVariable and Conditional Expectations
Remember the definition of the expectation of f(x) where x has the probability p(x) :
Conditional Expectation
(discrete)
E[t|x] = S p(t|x) t(x) t

13. Decision Theory for Regression
Inference step
Determine
Decision step
For given x, make optimal prediction, y(x).
Loss function:

14. The Squared Loss Function
If we used the squared loss as loss function:
Advanced
After some calculations (next slides...), we can show that:

15. ADVANCED - Explanation:
Consider the first term inside the loss:
This is equal to:
since p(x,t)=p(t|x)p(x)
since p(x) doesn’t depend on t,
we can move out of the integral; then the integral
∫p(t|x)dt
amounts to 1
as we are summing prob.s
through all possible t

16. Advanced: Explanation
Consider the second term inside the loss:
This is equal to zero:
since doesn’t depend on t,
we can move out of the integral

17. ADVANCED: Explanation for last step
E[t|x] does not vary with different values of t, so it can be moved out.
Notice that you could also immediately see that the expected value of differences from the mean for the random variable t is 0 (first line of the formula).

18. Important Hence we have:
The first term is minimized when we select y(x) as
The second term is independent of y(x) and represents the intrinsic variability of the target
It is called the intrinsic error.

20. Alternative approach/explanation
Using the squared error as the loss function:
We want to choose y(x) to minimize the expected loss:

21. Solving for y(x), we get: