1. A Novel Local Patch Framework for Fixing Supervised Learning Models Yilei Wang 1, Bingzheng Wei 2, Jun Yan 2, Yang Hu 2, Zhi-Hong Deng 1, Zheng Chen 2 1 Peking University 2 Microsoft Research Asia

2. Outline  Motivation & Background  Problem Definition & Algorithm Overview  Algorithm Details  Experiments - Classification  Experiments - Search Ranking  Conclusion

3. Motivation & Background  Supervised Learning:  Machine Learning task of inferring a function from labeled training data  Prediction Error:  No matter how strong a learning model is, it will suffer from prediction errors.  Noise in training data, dynamically changing data distribution, weakness of learner  Feedback from User:  Good signal for learning models to find the limitation and then improve accordingly

4. Learning to Fix Errors from Failure Cases  Automatically fix model prediction errors from failure cases in feedback data.  Input:  A well trained supervised model (we name it as Mother Model)  A collection of failure cases in feedback dataset.  Output:  Learning to automatically fix the model bugs from failure cases  Previous Works  Model Retraining  Model Aggregation  Incremental Learning

5. Local Patching: from Global to Local  Learning models are generally optimized globally  Introducing new prediction errors when fixing the old ones  Our key idea: learning to fix the model locally using patches New Error

6. Problem Definition

7. Algorithm Overview  Failure Case Collection  Learning Patch Regions/Failure Case Clustering  Clustering Failure Cases into N groups through subspace learning, compute the centroid and range for every group, then define our patches  Learning Patch Model  Learn a patch model using only the data samples that sufficiently close to the patch centroid

8. Algorithm Details

9. Learning Patch Region – Key Challenge  Failure cases may distribute diffusely  Small N = large patch range → many success cases will be patched  Big N = small patch range → high computational complexity  How to make trade-offs ?

10. Solution: Clustered Metric Learning  Our solution to diffusion: Metric Learning  Learn a distance metric, i.e. subspace, for failure cases, such that the similar failure cases will aggregate, and keep distant from the success cases. (Red circle = failure cases; blue circle = success cases) Key idea of the patch model learning (Left): The cases in original data space. (Middle): The cases mapped to the learned subspace. (Right): Repair the failure cases using a single patch.

11. Metric Learning

12. Clustered Metric Learning

13. Learning Patch Model

14. Experiments

15. Experiments - Classification  Dataset  Randomly select 3 UCI subset  Spambase, Waveform, Optical Digit Recognition  Convert to binary classification dataset  ~5000 instances in each dataset  Split to: 60% - training, 20% - feedback, 20% - test  Baseline Algorithm  SVM  Logistic Regression  SVM - retrained with training + feedback data  Logistic Regression - retrained with training + feedback data  SVM – Incremental Learning  Logistic Regression - Incremental Learning

16. Classification Accuracy  Classification accuracy on feedback dataset  Classification accuracy on test dataset SVMSVM+LPFLRLR+LPF Spam0.82300.88380.90550.9283 Wave0.72700.86700.86000.8850 Optdigit0.90660.97240.93060.9689 SVM SVM- Retain SVM-ILSVM+LPF LRLR-RetainLR-ILLR-LPF Spam 0.81960.83480.84780.85870.91520.91740.91850.9217 Wave 0.75300.77800.78500.86200.84600.86000.87700.8800 Optdigit 0.91010.91280.92170.96350.93320.93680.93880.9413

17. Classification – Case Coverage

18. Parameter Tuning  Number of Patches  Data sensitive, in our experiment the best N is 2

19. Experiments – Search Ranking  Dataset  Data from a commonly used commercial search engine  ~14, 126 pairs  With 5 grades label  Metrics  NDCG@K {1,3,5}  Baseline Algorithm  GBDT  GBDT + IL

20. Experiment Results – Ranking GBRTILGBRT + LPF nDCG@10.91150.91220.9422 nDCG@30.88370.89100.9149 nDCG@50.87900.88730.9090

21. Experiment Results – Ranking (Cont.)

22. Conclusion  We proposed  The local model fixing problem  A novel patch framework fox fixing the failure cases in feedback dataset in local view  The experiment results demonstrate the effectiveness of our proposed Local Patch Framework

23. Thank you!