1. cs638/838 - Spring 2017 (Shavlik©), Week 10
CS 540 Fall 2015 (Shavlik)
6/3/2018
Today’s Topics
Some Lab 3 Comments
Talk by Akshay Sood on recurrent ANNs, LSTM, etc
Talk by Luisa Polania Cabrera of American Family Insurance on some of their Deep ML projects
One-on-one Q&A on Lab3, Projects, etc
I have run turned-in Lab 3 code
Don’t redefine Vector!
Project report comments ed – lots on RL!
April 18: Google-Madison talk on TPUs
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

2. Sample, Latest CNN Results (32x32 images, batch size = 10)
ADAM? drHU/drIn ExtraTrain #FlatHUs
yes 0.00/ ,
yes 0.00/ ,
yes 0.50/ ,
no 0.00/ ,
no 0.00/
ADAM worked well
Extra examples helped a good deal
Convolution kernels of 4x4 worked better than 5x5 (3x3 in between)
‘Zero padding’ worked about the same as w/o it
Using 10 plates instead of 20 worked ok
Better test-set accuracy than I expected, given small dataset!
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

3. cs638/838 - Spring 2017 (Shavlik©), Week 10
Ensembles
Ensembles often greatly increase accuracy
Combining all models with 25 or fewer testset errors, led to 12 errors!
But this is a cheat! Why?
Correcting the cheat, led to 19 errors
Key question: how to pick N best models?
Don’t forget about ensembles!
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

4. Thought These are FLOWERS
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

5. Thought These are AIRPLANES
Some white borders!
Not centered
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

6. cs638/838 - Spring 2017 (Shavlik©), Week 10
NOT butterflies
More Ensemble Errors
NOT pianos
NOT starfish
NOT a watch
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

7. Tuneset vs. Testset Accuracies
We’d like to threshold the Y axis, but we need to threshold the X !
Testeset Errors
Tuneset Errors
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

8. Some Lab 3 Report Comments
Two senses of ‘learning curve’ (see original Lab 3 slides)
CURVES better than TABLES!
Some learning curves STEEP (next slide)
suggests value in getting more original images
Drop out worked for some, not for others
Generating ‘perturbed’ examples greatly helps
We really should replicate to get ‘error bars’ (ie, different random seeds)
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

9. An Encouraging Learning Curve!
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

10. My Learning Curve (used top-10 TUNEset models)
Testeset Errors
Number of (Original) Training Examples
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

11. cs638/838 - Spring 2017 (Shavlik©), Week 10
What Action is This?
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

12. cs638/838 - Spring 2017 (Shavlik©), Week 10
Impact of Random Seed
Be careful to avoid ‘cherry picking’!
Avoid ‘peeking at the test set’ while making decisions!
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

13. A Nice Overfitting Curve (from Lab 2)
ERROR
EPOCH
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10

14. Some Lab 3 Report Comments (2)
Should start learning curves at 0 or ‘accuracy of always guessing most common output’
Some initial weights should be negative
Too many plates might overfit?
Third CONVOLUTION layer probably hurts
WATCH predicted a lot because most common, probably not because it was LAST (reorder the enum!)
‘Rotated by 180’ – three meanings!
3/28/17
cs638/838 - Spring 2017 (Shavlik©), Week 10