1. Softmax Classifier + Generalization
Various slides from previous courses by: D.A. Forsyth (Berkeley / UIUC), I. Kokkinos (Ecole Centrale / UCL). S. Lazebnik (UNC / UIUC), S. Seitz (MSR / Facebook), J. Hays (Brown / Georgia Tech), A. Berg (Stony Brook / UNC), D. Samaras (Stony Brook) . J. M. Frahm (UNC), V. Ordonez (UVA), Steve Seitz (UW).

2. Last Class Introduction to Machine Learning
Unsupervised Learning: Clustering (e.g. k-means clustering)
Supervised Learning: Classification (e.g. k-nearest neighbors)

3. Today’s Class Softmax Classifier (Linear Classifiers)
Generalization / Overfitting / Regularization
Global Features

4. Supervised Learning vs Unsupervised Learning
𝑥 → 𝑦 𝑥
cat
dog
bear

5. Supervised Learning vs Unsupervised Learning
𝑥 → 𝑦 𝑥
cat
dog
bear

6. Supervised Learning vs Unsupervised Learning
𝑥 → 𝑦 𝑥
cat
dog
bear
Classification
Clustering

7. Supervised Learning – k-Nearest Neighbors
cat
k=3
dog
bear
cat, cat, dog
cat
cat
dog
bear
dog
bear

8. Supervised Learning – k-Nearest Neighbors
cat
dog
k=3
bear
cat
bear, dog, dog
cat
dog
bear
dog
bear

9. Supervised Learning – k-Nearest Neighbors
How do we choose the right K?
How do we choose the right features?
How do we choose the right distance metric?

10. Supervised Learning – k-Nearest Neighbors
How do we choose the right K?
How do we choose the right features?
How do we choose the right distance metric?
Answer: Just choose the one combination that works best!
BUT not on the test data.
Instead split the training data into a ”Training set” and
a ”Validation set” (also called ”Development set”)

11. Supervised Learning - Classification
Training Data
Test Data
dog
cat
bear
dog
cat
bear
cat
dog
bear

12. Supervised Learning - Classification
Training Data
Test Data
cat
dog
cat . .
bear

13. Supervised Learning - Classification
Training Data
𝑦 𝑛 =[ ]
𝑦 3 =[ ]
𝑦 2 =[ ]
𝑦 1 =[ ]
𝑥 1 =[ ]
𝑥 2 =[ ]
𝑥 3 =[ ]
𝑥 𝑛 =[ ]
cat
dog
cat .
bear

14. Supervised Learning - Classification
Training Data
inputs
targets /
labels /
ground truth
𝑦 𝑖 =𝑓( 𝑥 𝑖 ;𝜃)
We need to find a function that
maps x and y for any of them. 1 2
𝑦 𝑛 =
𝑦 3 =
𝑦 2 =
𝑦 1 =
predictions
𝑥 1 =[ 𝑥 𝑥 𝑥 𝑥 14 ] 1 2 3
𝑦 𝑛 =
𝑦 3 =
𝑦 2 =
𝑦 1 =
𝑥 2 =[ 𝑥 𝑥 𝑥 𝑥 24 ]
𝑥 3 =[ 𝑥 𝑥 𝑥 𝑥 34 ]
How do we ”learn” the parameters
of this function?
We choose ones that makes the
following quantity small:
𝑖=1 𝑛 𝐶𝑜𝑠𝑡( 𝑦 𝑖 , 𝑦 𝑖 ) .
𝑥 𝑛 =[ 𝑥 𝑛1 𝑥 𝑛2 𝑥 𝑛3 𝑥 𝑛4 ]

15. Supervised Learning –Softmax Classifier
Training Data
inputs
targets /
labels /
ground truth
𝑥 1 =[ 𝑥 𝑥 𝑥 𝑥 14 ] 1 2 3
𝑦 𝑛 =
𝑦 3 =
𝑦 2 =
𝑦 1 =
𝑥 2 =[ 𝑥 𝑥 𝑥 𝑥 24 ]
𝑥 3 =[ 𝑥 𝑥 𝑥 𝑥 34 ] .
𝑥 𝑛 =[ 𝑥 𝑛1 𝑥 𝑛2 𝑥 𝑛3 𝑥 𝑛4 ]

16. Supervised Learning –Softmax Classifier
Training Data
inputs
targets /
labels /
ground truth
[ ]
[ ]
[ ]
[ ]
𝑦 𝑛 =
𝑦 3 =
𝑦 2 =
𝑦 1 =
predictions
𝑥 1 =[ 𝑥 𝑥 𝑥 𝑥 14 ]
[ ]
[ ]
[ ]
𝑦 𝑛 =
𝑦 3 =
𝑦 2 =
𝑦 1 =
𝑥 2 =[ 𝑥 𝑥 𝑥 𝑥 24 ]
𝑥 3 =[ 𝑥 𝑥 𝑥 𝑥 34 ] .
𝑥 𝑛 =[ 𝑥 𝑛1 𝑥 𝑛2 𝑥 𝑛3 𝑥 𝑛4 ]

17. Supervised Learning –Softmax Classifier
𝑥 𝑖 =[ 𝑥 𝑖1 𝑥 𝑖2 𝑥 𝑖3 𝑥 𝑖4 ]
[ ]
𝑦 𝑖 =
𝑦 𝑖 = [ 𝑓 𝑐 𝑓 𝑑 𝑓 𝑏 ]
𝑔 𝑐 = 𝑤 𝑐1 𝑥 𝑖1 + 𝑤 𝑐2 𝑥 𝑖2 + 𝑤 𝑐3 𝑥 𝑖3 + 𝑤 𝑐4 𝑥 𝑖4 + 𝑏 𝑐
𝑔 𝑑 = 𝑤 𝑑1 𝑥 𝑖1 + 𝑤 𝑑2 𝑥 𝑖2 + 𝑤 𝑑3 𝑥 𝑖3 + 𝑤 𝑑4 𝑥 𝑖4 + 𝑏 𝑑
𝑔 𝑏 = 𝑤 𝑏1 𝑥 𝑖1 + 𝑤 𝑏2 𝑥 𝑖2 + 𝑤 𝑏3 𝑥 𝑖3 + 𝑤 𝑏4 𝑥 𝑖4 + 𝑏 𝑏
𝑓 𝑐 = 𝑒 𝑔 𝑐 / (𝑒 𝑔 𝑐 + 𝑒 𝑔 𝑑 + 𝑒 𝑔 𝑏 )
𝑓 𝑑 = 𝑒 𝑔 𝑑 / (𝑒 𝑔 𝑐 + 𝑒 𝑔 𝑑 + 𝑒 𝑔 𝑏 )
𝑓 𝑏 = 𝑒 𝑔 𝑏 / (𝑒 𝑔 𝑐 + 𝑒 𝑔 𝑑 + 𝑒 𝑔 𝑏 )

18. How do we find a good w and b?
𝑥 𝑖 =[ 𝑥 𝑖1 𝑥 𝑖2 𝑥 𝑖3 𝑥 𝑖4 ]
[ ]
𝑦 𝑖 =
𝑦 𝑖 = [ 𝑓 𝑐 (𝑤,𝑏) 𝑓 𝑑 (𝑤,𝑏) 𝑓 𝑏 (𝑤,𝑏)]
We need to find w, and b that minimize the following function L:
𝐿 𝑤,𝑏 = 𝑖=1 𝑛 𝑗=1 3 − 𝑦 𝑖,𝑗 log⁡( 𝑦 𝑖,𝑗 )
= 𝑖=1 𝑛 −log⁡( 𝑦 𝑖,𝑙𝑎𝑏𝑒𝑙 )
= 𝑖=1 𝑛 −log 𝑓 𝑖,𝑙𝑎𝑏𝑒𝑙 (𝑤,𝑏)
Why?

19. How do we find a good w and b?
Problem statement:
Find 𝑤
and 𝑏
such that
𝐿 𝑤,𝑏
is minimal.
𝜕 𝜕𝑤 𝐿 𝑤,𝑏 =0
Solution from calculus.
and solve for 𝑤
𝜕 𝜕𝑏 𝐿 𝑤,𝑏 =0 𝑏

20. https://courses. lumenlearning

21. How do we find a good w and b?
Problem statement:
Find 𝑤
and 𝑏
such that
𝐿 𝑤,𝑏
is minimal.
𝜕 𝜕𝑤 𝐿 𝑤,𝑏 =0
Solution from calculus.
and solve for 𝑤
𝜕 𝜕𝑏 𝐿 𝑤,𝑏 =0 𝑏

22. Problems with this approach:
Some functions L(w, b) are very complicated and compositions of many functions. So finding its analytical derivative is tedious.
Even if the function is simple to derivate, it might not be easy to solve for w. e.g.
𝜕 𝜕𝑤 𝐿 𝑤,𝑏 = 𝑒 𝑤 +𝑤= 0
How do you find w in that equation?

23. Solution: Iterative Approach: Gradient Descent (GD)
1. Start with a random value of w (e.g. w = 12)
𝐿 𝑤
2. Compute the gradient (derivative) of L(w) at point w = 12. (e.g. dL/dw = 6)
w=12
3. Recompute w as:
w = w – lambda * (dL / dw) 𝑤

24. Solution: Iterative Approach: Gradient Descent (GD)
𝐿 𝑤
2. Compute the gradient (derivative) of L(w) at point w = 12. (e.g. dL/dw = 6)
3. Recompute w as:
w = w – lambda * (dL / dw)
w=10 𝑤

25. Gradient Descent (GD) Problem: expensive! 𝜆=0.01
for e = 0, num_epochs do
end
Initialize w and b randomly
𝑑𝐿(𝑤,𝑏)/𝑑𝑤
𝑑𝐿(𝑤,𝑏)/𝑑𝑏
Compute:
and
Update w:
Update b:
𝑤=𝑤 −𝜆 𝑑𝐿(𝑤,𝑏)/𝑑𝑤
𝑏=𝑏 −𝜆 𝑑𝐿(𝑤,𝑏)/𝑑𝑏
Print:
𝐿(𝑤,𝑏)
// Useful to see if this is becoming smaller or not.
𝐿(𝑤,𝑏)= 𝑖=1 𝑛 −log 𝑓 𝑖,𝑙𝑎𝑏𝑒𝑙 (𝑤,𝑏)

26. Solution: (mini-batch) Stochastic Gradient Descent (SGD)
𝜆=0.01
for e = 0, num_epochs do
end
Initialize w and b randomly
𝑑𝑙(𝑤,𝑏)/𝑑𝑤
𝑑𝑙(𝑤,𝑏)/𝑑𝑏
Compute:
and
Update w:
Update b:
𝑤=𝑤 −𝜆 𝑑𝑙(𝑤,𝑏)/𝑑𝑤
𝑏=𝑏 −𝜆 𝑑𝑙(𝑤,𝑏)/𝑑𝑏
Print:
𝑙(𝑤,𝑏)
// Useful to see if this is becoming smaller or not.
𝑙(𝑤,𝑏)= 𝑖∈𝐵 −log 𝑓 𝑖,𝑙𝑎𝑏𝑒𝑙 (𝑤,𝑏)
B is a small
set of training
examples.
for b = 0, num_batches do
end

27. Source: Andrew Ng

28. Three more things How to compute the gradient Regularization
Momentum updates

29. SGD Gradient for the Softmax Function

30. SGD Gradient for the Softmax Function

31. SGD Gradient for the Softmax Function

32. Supervised Learning –Softmax Classifier
𝑦 𝑖 = [ 𝑓 𝑐 𝑓 𝑑 𝑓 𝑏 ]
Get predictions
𝑥 𝑖 =[ 𝑥 𝑖1 𝑥 𝑖2 𝑥 𝑖3 𝑥 𝑖4 ]
Extract features
𝑔 𝑐 = 𝑤 𝑐1 𝑥 𝑖1 + 𝑤 𝑐2 𝑥 𝑖2 + 𝑤 𝑐3 𝑥 𝑖3 + 𝑤 𝑐4 𝑥 𝑖4 + 𝑏 𝑐
𝑔 𝑑 = 𝑤 𝑑1 𝑥 𝑖1 + 𝑤 𝑑2 𝑥 𝑖2 + 𝑤 𝑑3 𝑥 𝑖3 + 𝑤 𝑑4 𝑥 𝑖4 + 𝑏 𝑑
𝑔 𝑏 = 𝑤 𝑏1 𝑥 𝑖1 + 𝑤 𝑏2 𝑥 𝑖2 + 𝑤 𝑏3 𝑥 𝑖3 + 𝑤 𝑏4 𝑥 𝑖4 + 𝑏 𝑏
𝑓 𝑐 = 𝑒 𝑔 𝑐 / (𝑒 𝑔 𝑐 + 𝑒 𝑔 𝑑 + 𝑒 𝑔 𝑏 )
𝑓 𝑑 = 𝑒 𝑔 𝑑 / (𝑒 𝑔 𝑐 + 𝑒 𝑔 𝑑 + 𝑒 𝑔 𝑏 )
𝑓 𝑏 = 𝑒 𝑔 𝑏 / (𝑒 𝑔 𝑐 + 𝑒 𝑔 𝑑 + 𝑒 𝑔 𝑏 )
Run features through classifier

33. Supervised Machine Learning Steps
Training
Training Labels
Training Images
Image Features
Training
Learned model
Learned model
Testing
Image Features
Prediction
Test Image
Slide credit: D. Hoiem

34. Test set (labels unknown)
Generalization
Generalization refers to the ability to correctly classify never before seen examples
Can be controlled by turning “knobs” that affect the complexity of the model
Test set (labels unknown)
Training set (labels known)

35. 𝑓 is a polynomial of degree 9
Overfitting
𝑓 is a polynomial of degree 9
𝑓 is linear
𝑓 is cubic
𝐿𝑜𝑠𝑠 𝑤 is high
𝐿𝑜𝑠𝑠 𝑤 is low
𝐿𝑜𝑠𝑠 𝑤 is zero!
Overfitting
Underfitting
High Bias
High Variance
Credit: C. Bishop. Pattern Recognition and Mach. Learning.

36. Questions?