1. Next Gen Microservices
On kubernetes

2. agenda Who are we? Our unique take on microservices
How we leveraged K8S
evaluation methods
running in production
some problems we ran into (to save you some pain)

3. (or who are we?)

4. 5,000 120 325,000 19,000 192 980,000 4,800+ STORIES A DAY IN MORE THAN
COUNTRIES
Bloomberg News
325,000
SUBSCRIPTIONS GLOBALLY
Bloomberg Professional Service
19,000
EMPLOYEES IN
192
LOCATIONS AROUND THE WORLD
980,000
GLOBAL CIRCULATION IN 150 COUNTRIES
Bloomberg Businessweek
4,800+
TECHNOLOGISTS & COMPUTER ENGINEERS
Bloomberg

5. Collect real-time financial data from:
370+ exchanges
4,000+ FIX feeds
60 billion ticks
80,000+ news wires from around the world
Our subscribers exchange daily:
300 million messages
17 million chats

6. Now that’s BIG DATA

7. Our team Mikhail Kartashov Mikolaj Pawlikowski Paul McLoughlin
Sachin Kamboj

8. (This presentation will self-destruct in 32.5 minutes)
Our mission
build infrastructure that allows non-programmers (and programmers) to deploy code effortlessly
(This presentation will self-destruct in 32.5 minutes)

9. Data Technologies Pipeline
(DTP)

10. microservices
do one
thing
do it
well

11. single-click™ deployment
next generation
testing in production
single-click™ deployment

12. testing in production
All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy godmothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable: 'This time it will surely run,' or 'I just found the last bug'.
- Frederick Brooks, Jr. (The Mythical Man Month)

13. Unit + Integration Testing
typical release cycle
Development
Staging
Production
Code
Deploy
Deploy
Unit + Integration Testing
Testing
Testing
All tests pass?
Working?
Working?
Oops

14. typical release cycle Release Cycle Code Deploy Watch-it-break
Rollback

15. why oh God, why? env(development) != env(staging) != env(production)
environment
env(development) != env(staging) != env(production)
data
data(development) != data(staging) != data(production)

16. our new workflow Production Deploy Release 1 Release 1 Code
View (In/Out)put
NO!
Happy?
Promote to publish

17. our new workflow Production Deploy Release 1 Release 1 Release 2 Code
View (In/Out)put
Release 3
Release 3
NO!
Happy?
Promote to publish

18. But when I do I do it in production
I don’t always test my code
But when I do I do it in production

19. requirements microservices must be pure
no alterations to any external state

20. single-click deployment
Code
Push code to github
Release
Tag a release
Configure
Set environment variables/secrets
Deploy
Select active versions/tags
Select production version

21. How we built our platfoRm
the nuts and bolts
How we built our platfoRm

22. our shopping list Isolation Orchestration Easy deployment Self-healing
Stability
Reliability
High availability

23. kubernetes
Image credit:

24. our shopping list Docker Isolation Kubernetes Orchestration
Easy deployments
Auto-scaling
Self-healing
???
Stability
Reliability
High-availability

25. our architecture Load Balancer Datacenter - 1 Master 1 Master 2
Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
Worker 6
Worker 7
Worker 8
Worker 9
Datacenter - 2
Master 1
Master 2
Master 3
Master 4
Master 5
Worker 1
Worker 2
Worker 3
Worker 4
Worker 5
Worker 6
Worker 7
Worker 8
Worker 9

26. our architecture Master Worker reverse proxy etcd K8s api-server
µsvc pod-1
µsvc pod-2
µsvc pod-3
µsvc pod-4
K8s scheduler
kube-dns
K8s controller manager
µsvc pod-5
µsvc pod-6
µsvc pod-7
µsvc pod-8
kubelet
kube-proxy
K8s api-server proxy
kubelet
kube-proxy
K8s api-server proxy
Docker
Docker
flanneld
flanneld

27. powerful seal Our chaos monkey
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIZIIIIII7IIIIIII$IIIIIIIIIIIIIIIIIIZ?II IIIIIIIIIIIIIIIIIIIIIIIIZIIIIIIIIIIIIZD+~7NIIIIII7IIIIIII$IIIIIII7IIIIIIIIIIZII? IIIIIIIIIIIIIIIIIIIIIIIIZIIIIIIIIIIID::~~,:?8IIII7IIIIIII$IIIIIII7IIIIIIIIIIZIII IIIIIIIIIIIIIIIIIIIIIIIIZIIIIIIIII78~~~D8O~~=DIII7IIIIIII$IIIIIII7IIIIIIIIIIZIII IIIIIIIIIIIIIIIIIIIIIIIIZIIIIZNNOD8:~~~NDD:~~$ZII7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIIIIIOD?~$~8$IIIIIIIIIZIIII8~~~~~II~~~~~~~~~NII7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIIIIOZ~I=~Z8IIIIIIIIIIZIIIII$N?:~~==~~~~~~~~O$I7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIII8?~$:~O~8IIIIIIIIIIZIIIIII?I7N:~~~~~~~~:~=8I7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIIO+~=~~$~ZOD7+ZDIIIIIZIIIIIII$$~~~~~~~~~~~~~NI7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIZ7~~~~~DO:,.,,,:?87N?:IN7IIID~~~~D?~~~~~~~~~=N77D$88?~:~+N$III$IIIIIIIIIIZ7II II78:~~~~~~~~::~~~,:D:~.,:~~=NIII7$D~+~~~I~:~:~?I~~D~~,,,::~~~D$I$IIIIIIIIIIO7II IID~~~~~~~~~?~:~~~~8~~~~~~:~~~7DO?8+~I$~~I$:~~:~~~~~~~~~~~:~~~~=NN8888DDOIIIO7II I$$~~~~+~~?=Z7?~~~~~~~~~~~~~~~~~~?8~~~:~~~~~~:~=I=~~~~~~~~~~~~~~~~~~~~:,,~+8O7II ID~~~~~=+~~Z~~~~~~~~~~~~~~~~~~~~~~+8~~~~~~$8=~~~~~~~~~~~~~~~~~~~~~~~~~~~~,:O87II 7N~~~~~~8~~~~~~~=~~~~~~~~~=~~~~~~~~~~~~~~O:~~~~~:,,~~~~~~$:~~~~~788D+~~~~::+N777 IID=~~~~~+8DODI~~~~~~~~~~~Z~~~~,:~~~~~=~~~~~~~~,,,,:~~~~+~~~~~I~~~~~I~~~~~~=N$77 II7ID=:~~~~~~~~~~~~~~~~~~~~+~~:~~~~~~~8$~~~~~~~:~~~~~~~~=~~~~~?$~~~~~~~~~~~~D$77 III77ION+~~~~~~~~~~~~~~~~:D~~~~~~~~~+=O~~~~~~~~~~~~~~~~~:~~~~~7I?=:~~~~~~~~~D$ I7IZ$7DDI::~~~~~~~~:~IO:~~~~~~~~~~D?~~~~~~~~~~~~~~~I=$N7~~+~~~~~=7~~~~~=NZZO O$IIIIIII77777IIIZ8~~~~~~~~~~~=Z~~~~~~~~~~~~~~~~O=?ZI87~~~~788~:+~~~O8$ O$IIIIIIIIIIIIIIIZ8=~D7=~~~~~~~Z~~~~~~~~D8=~~~~:D~I$IIIIII$II8~:I~$:DO$ O$IIIIIIIIIIIIIIIZ?D?~~~~~~~:~~O~~~~~~~~~~~~~~7D~~8IIIIIII$IIZ++=~7D?O$7$ O$IIIIIIIIIIIIIIIZII?8Z8DD8ZI=~~~~~77$ZZZZI=~Z:~~~DIIIIIII7IIIO$~$O7IO$ OZIIIIIIIIIIIIIIIZIIID~~~:,,,~~~~,,,~~~~~~~=D~~=?$OIIIIIII7IIID8~D7IIO$ O$IIIIIIIIIIIIIIIZIIII7++~~~~~?~~~~~~~~=?=~~~~~~Z8$IIIIIII7III78DIIIIO$ OZIIIIIIIIIIIIIIIZIII7I~~~~~~~=?~~~~~~~~~~Z~7Z~ZO?$IIIIIII7IIIIIIIIIIO$ OZIIIIIIIIIIIIIIIZIIIIZ$=~~~~:~Z=~~=7$~~O7~Z?+NIII$IIIIIII7IIIIIIIIIIO$ O$IIIIIIIIIIIIIIIZIIIII$+~~~~~7N~~~~~~~~D~~~~?OIII$IIIIIII7IIIIIIIIIIO$ O$IIIIIIIIIIIIIIIZIIIIIO=~~~~~$N:~~~~~~~$~~~~$OIII$IIIIIII7IIIIIIIIII8Z O$IIIIIIIIIIIIIIIZIIIIIO=~~~~~~8:~~~~:~~~~~~:?NOZD$IIIIIII7IIIIIIIIII8Z7$ O$IIIIIIIII8DDDDDDOZ$7I$Z+=~~==$7Z8DNNDD88OZZNZ~:~?8IIIIII7IIIIIIIIII8Z O$IIIIIII8O,,::~::::::~ZOZZZZZZZZZZZZZZZZZOD=~~~~:~+8IIIII7IIIIIIIIIIOZ7$ O$IIIIIIN,~~~~~~~~~~~~~~ZOZZZZZZZZZZZZZZ88~~~~~~~~~~DIIIII$IIIIIIIIIIOZ7$ O$III7D~~~~~~~~~~~~~~~~~~DZZZZZZZZZZZZZD=:~~~~~~~~I~Z7IIII7IIIIIIIIIIOZ$$ O$IID=~~=DO8DDZ?~~~~~~~~~~NZZZZZZZZZZ8$~~~~~~~I~I:8:DIIIII$IIIIIIIIIIOZ$$ O$IZI~~~~~~~~~~~~~~~~~~~~~~NZZZZZZZO8~~~~~~~~~$7I~$8IIIIII7IIIIIIIIII8Z$$ O$I?D:~~~~~~~~~~~~~~~~~~~~~:DZZZZZD::~~~~~~~~~+??I?ZIIIIII7IIIIIIIIII8Z$$ O$IID:~~~+DN~?7$77I?+=~:~~~~~ZNZ88~~~~~==::~~~:7~~+OIIIIII7IIIIIIIIII8Z$$ O$IIN~~~~~~+:$Z=:~~~~~~~~~IN7IIIIO~~~~~~?=,~~~~~~~IZIIIIII7IIIIIIIIII8Z$$ O$IIIO~~~~~$~~~Z$Z7ODNNN8IIIIIIIIIZ:,~~~~D:,~~~~~~$7IIIIII7IIIIIIIIIIOZ$$ O$IIIOI~~~~~I~~~DZIIIIIIIIIIIIIIII8:,:~~~~D,:~~~~~OIIIIIII7IIIIIIIIIIOZ$$ O$IIIID~~~~~O~~~DZIIIIIIIIIIIIIIIIID~,~~~~:O,:~~~~8II$$III7IIIIIIIIIIOZ$$ O$IIIIID~~~~=~~+DZIIIIIIIIIIIIIIIII87~,:~~~+I~:~~~DD~~:~=N7IIIIIIIIIIOZ$$ O$IIIII$Z~~~~~~DI$IIIIIIIIIIIIIIIIIZ?N=~::~~DDZ~~?~~,,~~,,~N7IIIIIIIIOZ$$ O$IIIIIID=~~~~+DI$IIIIIIIIIIIIIIIIIZIIIZD~~~~~~~~~~~~~~~~~~~I8?II7IIIOZ$$ O$IIIIIIIN~~~~DII7IIIIIIIIIIIIIIIII$IIIIIII8~~~~~~~~~~~~~~~~~~~~~~8II8Z$$ O$$77$$$$$8~~~D$$$$$$$$ZZZZO$$$$$$$Z$$$$$ZZZN=~~~~~~~~~~~~~~~~~~~~D$$NDDD ::::::::::::::::::~~~:::::::::::::::::::::::::::::::::::::::::::::::::~~::::::::
powerful seal
Our chaos monkey

33. lessons learned
Or how we learned to stop worrying and love the bomb kubernetes
Image credit:

34. lessons learned
Kubelet (and other k8s components) can’t handle multiple masters

35. lessons learned flanneld won’t handle sigpipe
communication between pods can break

36. lessons learned

37. lessons learned Security issues:
kube-apiserver -> kubelet communication broken
can’t completely disable insecure port on K8S
certificate revocations not supported

38. Questions?