1. Scaling Distributed Machine Learning with the Parameter Server By M. Li, D. Anderson, J. Park, A. Smola, A. Ahmed, V. Josifovski, J. Long E. Shekita, B. Su. EECS 582 – W161

2. Outline Motivation Parameter Server architecture Why is it special? Evaluation EECS 582 – W162

3. Motivation EECS 582 – W163

4. Motivation EECS 582 – W164

5. Motivation EECS 582 – W165

6. Parameter Server EECS 582 – W166

7. Parameter Server EECS 582 – W167 How is this different from a traditional KV store?

8. Key Value Stores EECS 582 – W168 Key-Value Store Clients

9. Diff from KV (leverage domain knowledge) Treat KVs as sparse matrices Computation occurs on PSs Flexible consistency Intelligent replication Message Compression EECS 582 – W169

10. KVs as Sparse Matrices EECS 582 – W1610

11. KVs as Sparse Matrices EECS 582 – W1611

12. User-defined Functions PSs aggregate data from the workers: In distributed gradient descent workers push gradients that PSs use to calculate how to update the parameters Users can also supply user defined functions to run on the server EECS 582 – W1612

13. Flexible Consistency EECS 582 – W1613

14. Flexible Consistency EECS 582 – W1614

15. Intelligent Replication EECS 582 – W1615

16. Intelligent Replication EECS 582 – W1616

17. Message Compression EECS 582 – W1617

18. Message Compression Training data does not change: Only send hash of training data on subsequent iterations Values are often 0 Only send non-zero values EECS 582 – W1618

19. Evaluation (logistic regression) EECS 582 – W1619

20. Evaluation (logistic regression) EECS 582 – W1620

21. Evaluation (logistic regression) EECS 582 – W1621

22. Evaluation (logistic regression) EECS 582 – W1622

23. Evaluation (Topic Modeling) EECS 582 – W1623

24. Conclusion The Parameter Server model is a much better model for machine learning computation than comparing models EECS 582 – W1624

25. Conclusion The Parameter Server model is a much better model for machine learning computation than comparing models EECS 582 – W1625 Designing a system around a particular problem exposes optimizations which dramatically improve performance