CMS 165 Lecture 18 Summing up..

Most surprising/interesting things learnt in this class Bias-variance trade-off in non-classical setting Over-parametrization regime where traditional bias-variance don’t behave the same Robustness – min-max games (data augmentation + adversarial training) Decoupling the uncertainties in errors (active learning, fairness) Symmetries in non-convex optimization can be exploited Non-isolated optimal points (lives in the manifold) Generalization can be improved with robust training Data collection and importance of having good train/test set Role of local regularization and global regularization Sample complexity results and how it is not straightforward in deep learning Stability, effect of single data point in the dataset

Missing pieces (not covered in class) Interpretability Discussed applications are limited to standard ones, not many deep learning or recent applications of established works Reinforcement learning/Lifelong learning Likelihood in test after training GANs Negative results and tricks in achieving good results Explicit discussion of types of learning algorithms/ categorization Decision trees/classical ML techniques Recent properties of neural network architectures and computation scales Sequence models/autoencoders and embeddings Causal inference(besides Bayesian networks?)

Other comments.. Peer grading (not much learning from peer grading)(maybe on the open ended ones?) Maybe different application questions Paper to code kind of questions More applications/discussions of tensors Send the good homeworks to students instead of sending randomly More recitations on both theory and applications