1. Closing Remarks Cyrus M. Vahid, Principal Solutions Architect,
Principal Solutions AWS Deep Learning
June 2017

2. 1014
parameters
Brain runs at 100Hz

3. Brain-sized DNN Today
Today: g2.16xlarge has 192 GB of GPU RAM. 32 bits/param. SGD needs 3x copies. => 12B parameters So you’d need 8,000x p2.16xl instances Cost: $115,000/hr

4. Amdahl’s Law
Evolution according to Amdahl's law of the theoretical speedup in latency of the execution of a program in function of the number of processors executing it, for different values of p. The speedup is limited by the serial part of the program. For example, if 95% of the program can be parallelized, the theoretical maximum speedup using parallel computing would be 20 times.

5. Moore’s Law Price/performance doubles every 12-18 months
Already on the fast side of that.
TPUs from Google / IBM / Nervana

6. Yann Le Cun
“The best neural networks have always taken 3 weeks to train.”

7. Brain-sized DNN in 2026
256x = 28, Moore’s law: 8 doublings = 8-12 years 400x p7.256xl* instances. $1,000/hr* 3 weeks to train: $500k
*Pure speculation, obviously

8. Cyrus M. Vahid