Next Gen Microservices On kubernetes

agenda Who are we? Our unique take on microservices How we leveraged K8S evaluation methods running in production some problems we ran into (to save you some pain)

(or who are we?)

5,000 120 325,000 19,000 192 980,000 4,800+ STORIES A DAY IN MORE THAN COUNTRIES Bloomberg News 325,000 SUBSCRIPTIONS GLOBALLY Bloomberg Professional Service 19,000 EMPLOYEES IN 192 LOCATIONS AROUND THE WORLD 980,000 GLOBAL CIRCULATION IN 150 COUNTRIES Bloomberg Businessweek 4,800+ TECHNOLOGISTS & COMPUTER ENGINEERS Technology @ Bloomberg

Collect real-time financial data from: 370+ exchanges 4,000+ FIX feeds 60 billion ticks 80,000+ news wires from around the world Our subscribers exchange daily: 300 million messages 17 million chats

Now that’s BIG DATA

Our team Mikhail Kartashov Mikolaj Pawlikowski Paul McLoughlin Sachin Kamboj

(This presentation will self-destruct in 32.5 minutes) Our mission build infrastructure that allows non-programmers (and programmers) to deploy code effortlessly (This presentation will self-destruct in 32.5 minutes)

Data Technologies Pipeline (DTP)

microservices do one thing do it well

single-click™ deployment next generation testing in production single-click™ deployment

testing in production All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy godmothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable: 'This time it will surely run,' or 'I just found the last bug'. - Frederick Brooks, Jr. (The Mythical Man Month)

Unit + Integration Testing typical release cycle Development Staging Production Code Deploy Deploy Unit + Integration Testing Testing Testing All tests pass? Working? Working? Oops

typical release cycle Release Cycle Code Deploy Watch-it-break Rollback

why oh God, why? env(development) != env(staging) != env(production) environment env(development) != env(staging) != env(production) data data(development) != data(staging) != data(production)

our new workflow Production Deploy Release 1 Release 1 Code View (In/Out)put NO! Happy? Promote to publish

our new workflow Production Deploy Release 1 Release 1 Release 2 Code View (In/Out)put Release 3 Release 3 NO! Happy? Promote to publish

But when I do I do it in production I don’t always test my code But when I do I do it in production

requirements microservices must be pure no alterations to any external state

single-click deployment Code Push code to github Release Tag a release Configure Set environment variables/secrets Deploy Select active versions/tags Select production version

How we built our platfoRm the nuts and bolts How we built our platfoRm

our shopping list Isolation Orchestration Easy deployment Self-healing Stability Reliability High availability

kubernetes Image credit: https://www.flickr.com/photos/55198508@N00/3772734583/

our shopping list Docker Isolation Kubernetes Orchestration Easy deployments Auto-scaling Self-healing ??? Stability Reliability High-availability

our architecture Load Balancer Datacenter - 1 Master 1 Master 2 Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Worker 6 Worker 7 Worker 8 Worker 9 Datacenter - 2 Master 1 Master 2 Master 3 Master 4 Master 5 Worker 1 Worker 2 Worker 3 Worker 4 Worker 5 Worker 6 Worker 7 Worker 8 Worker 9

our architecture Master Worker reverse proxy etcd K8s api-server µsvc pod-1 µsvc pod-2 µsvc pod-3 µsvc pod-4 K8s scheduler kube-dns K8s controller manager µsvc pod-5 µsvc pod-6 µsvc pod-7 µsvc pod-8 kubelet kube-proxy K8s api-server proxy kubelet kube-proxy K8s api-server proxy Docker Docker flanneld flanneld

powerful seal Our chaos monkey IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIZIIIIII7IIIIIII$IIIIIIIIIIIIIIIIIIZ?II IIIIIIIIIIIIIIIIIIIIIIIIZIIIIIIIIIIIIZD+~7NIIIIII7IIIIIII$IIIIIII7IIIIIIIIIIZII? IIIIIIIIIIIIIIIIIIIIIIIIZIIIIIIIIIIID::~~,:?8IIII7IIIIIII$IIIIIII7IIIIIIIIIIZIII IIIIIIIIIIIIIIIIIIIIIIIIZIIIIIIIII78~~~D8O~~=DIII7IIIIIII$IIIIIII7IIIIIIIIIIZIII IIIIIIIIIIIIIIIIIIIIIIIIZIIIIZNNOD8:~~~NDD:~~$ZII7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIIIIIOD?~$~8$IIIIIIIIIZIIII8~~~~~II~~~~~~~~~NII7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIIIIOZ~I=~Z8IIIIIIIIIIZIIIII$N?:~~==~~~~~~~~O$I7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIII8?~$:~O~8IIIIIIIIIIZIIIIII?I7N:~~~~~~~~:~=8I7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIIO+~=~~$~ZOD7+ZDIIIIIZIIIIIII$$~~~~~~~~~~~~~NI7IIIIIII$IIIIIII7IIIIIIIIIIZ7II IIIZ7~~~~~DO:,.,,,:?87N?:IN7IIID~~~~D?~~~~~~~~~=N77D$88?~:~+N$III$IIIIIIIIIIZ7II II78:~~~~~~~~::~~~,:D:~.,:~~=NIII7$D~+~~~I~:~:~?I~~D~~,,,::~~~D$I$IIIIIIIIIIO7II IID~~~~~~~~~?~:~~~~8~~~~~~:~~~7DO?8+~I$~~I$:~~:~~~~~~~~~~~:~~~~=NN8888DDOIIIO7II I$$~~~~+~~?=Z7?~~~~~~~~~~~~~~~~~~?8~~~:~~~~~~:~=I=~~~~~~~~~~~~~~~~~~~~:,,~+8O7II ID~~~~~=+~~Z~~~~~~~~~~~~~~~~~~~~~~+8~~~~~~$8=~~~~~~~~~~~~~~~~~~~~~~~~~~~~,:O87II 7N~~~~~~8~~~~~~~=~~~~~~~~~=~~~~~~~~~~~~~~O:~~~~~:,,~~~~~~$:~~~~~788D+~~~~::+N777 IID=~~~~~+8DODI~~~~~~~~~~~Z~~~~,:~~~~~=~~~~~~~~,,,,:~~~~+~~~~~I~~~~~I~~~~~~=N$77 II7ID=:~~~~~~~~~~~~~~~~~~~~+~~:~~~~~~~8$~~~~~~~:~~~~~~~~=~~~~~?$~~~~~~~~~~~~D$77 III77ION+~~~~~~~~~~~~~~~~:D~~~~~~~~~+=O~~~~~~~~~~~~~~~~~:~~~~~7I?=:~~~~~~~~~D$77 7777I7IZ$7DDI::~~~~~~~~:~IO:~~~~~~~~~~D?~~~~~~~~~~~~~~~I=$N7~~+~~~~~=7~~~~~=NZZO 7777777O$IIIIIII77777IIIZ8~~~~~~~~~~~=Z~~~~~~~~~~~~~~~~O=?ZI87~~~~788~:+~~~O8$77 7777777O$IIIIIIIIIIIIIIIZ8=~D7=~~~~~~~Z~~~~~~~~D8=~~~~:D~I$IIIIII$II8~:I~$:DO$77 7777777O$IIIIIIIIIIIIIIIZ?D?~~~~~~~:~~O~~~~~~~~~~~~~~7D~~8IIIIIII$IIZ++=~7D?O$7$ 7777777O$IIIIIIIIIIIIIIIZII?8Z8DD8ZI=~~~~~77$ZZZZI=~Z:~~~DIIIIIII7IIIO$~$O7IO$77 7777777OZIIIIIIIIIIIIIIIZIIID~~~:,,,~~~~,,,~~~~~~~=D~~=?$OIIIIIII7IIID8~D7IIO$77 7777777O$IIIIIIIIIIIIIIIZIIII7++~~~~~?~~~~~~~~=?=~~~~~~Z8$IIIIIII7III78DIIIIO$77 7777777OZIIIIIIIIIIIIIIIZIII7I~~~~~~~=?~~~~~~~~~~Z~7Z~ZO?$IIIIIII7IIIIIIIIIIO$77 7777777OZIIIIIIIIIIIIIIIZIIIIZ$=~~~~:~Z=~~=7$~~O7~Z?+NIII$IIIIIII7IIIIIIIIIIO$77 7777777O$IIIIIIIIIIIIIIIZIIIII$+~~~~~7N~~~~~~~~D~~~~?OIII$IIIIIII7IIIIIIIIIIO$77 7777777O$IIIIIIIIIIIIIIIZIIIIIO=~~~~~$N:~~~~~~~$~~~~$OIII$IIIIIII7IIIIIIIIII8Z77 7777777O$IIIIIIIIIIIIIIIZIIIIIO=~~~~~~8:~~~~:~~~~~~:?NOZD$IIIIIII7IIIIIIIIII8Z7$ 7777777O$IIIIIIIII8DDDDDDOZ$7I$Z+=~~==$7Z8DNNDD88OZZNZ~:~?8IIIIII7IIIIIIIIII8Z77 7777777O$IIIIIII8O,,::~::::::~ZOZZZZZZZZZZZZZZZZZOD=~~~~:~+8IIIII7IIIIIIIIIIOZ7$ 7777777O$IIIIIIN,~~~~~~~~~~~~~~ZOZZZZZZZZZZZZZZ88~~~~~~~~~~DIIIII$IIIIIIIIIIOZ7$ 7777777O$III7D~~~~~~~~~~~~~~~~~~DZZZZZZZZZZZZZD=:~~~~~~~~I~Z7IIII7IIIIIIIIIIOZ$$ 7777777O$IID=~~=DO8DDZ?~~~~~~~~~~NZZZZZZZZZZ8$~~~~~~~I~I:8:DIIIII$IIIIIIIIIIOZ$$ 7777777O$IZI~~~~~~~~~~~~~~~~~~~~~~NZZZZZZZO8~~~~~~~~~$7I~$8IIIIII7IIIIIIIIII8Z$$ 7777777O$I?D:~~~~~~~~~~~~~~~~~~~~~:DZZZZZD::~~~~~~~~~+??I?ZIIIIII7IIIIIIIIII8Z$$ 7777777O$IID:~~~+DN~?7$77I?+=~:~~~~~ZNZ88~~~~~==::~~~:7~~+OIIIIII7IIIIIIIIII8Z$$ 7777777O$IIN~~~~~~+:$Z=:~~~~~~~~~IN7IIIIO~~~~~~?=,~~~~~~~IZIIIIII7IIIIIIIIII8Z$$ 7777777O$IIIO~~~~~$~~~Z$Z7ODNNN8IIIIIIIIIZ:,~~~~D:,~~~~~~$7IIIIII7IIIIIIIIIIOZ$$ 7777777O$IIIOI~~~~~I~~~DZIIIIIIIIIIIIIIII8:,:~~~~D,:~~~~~OIIIIIII7IIIIIIIIIIOZ$$ 7777777O$IIIID~~~~~O~~~DZIIIIIIIIIIIIIIIIID~,~~~~:O,:~~~~8II$$III7IIIIIIIIIIOZ$$ 7777777O$IIIIID~~~~=~~+DZIIIIIIIIIIIIIIIII87~,:~~~+I~:~~~DD~~:~=N7IIIIIIIIIIOZ$$ 7777777O$IIIII$Z~~~~~~DI$IIIIIIIIIIIIIIIIIZ?N=~::~~DDZ~~?~~,,~~,,~N7IIIIIIIIOZ$$ 7777777O$IIIIIID=~~~~+DI$IIIIIIIIIIIIIIIIIZIIIZD~~~~~~~~~~~~~~~~~~~I8?II7IIIOZ$$ 7777777O$IIIIIIIN~~~~DII7IIIIIIIIIIIIIIIII$IIIIIII8~~~~~~~~~~~~~~~~~~~~~~8II8Z$$ 7777777O$$77$$$$$8~~~D$$$$$$$$ZZZZO$$$$$$$Z$$$$$ZZZN=~~~~~~~~~~~~~~~~~~~~D$$NDDD ::::::::::::::::::~~~:::::::::::::::::::::::::::::::::::::::::::::::::~~:::::::: powerful seal Our chaos monkey

lessons learned Or how we learned to stop worrying and love the bomb kubernetes Image credit: https://flic.kr/p/ebfFrK

lessons learned Kubelet (and other k8s components) can’t handle multiple masters https://github.com/kubernetes/kubernetes/issues/18174 https://github.com/kubernetes/kubernetes/issues/19152

lessons learned flanneld won’t handle sigpipe communication between pods can break

lessons learned

lessons learned Security issues: kube-apiserver -> kubelet communication broken https://github.com/kubernetes/kubernetes/issues/11816 https://github.com/kayrus/kubelet-exploit can’t completely disable insecure port on K8S https://github.com/kubernetes/kubernetes/issues/13598 certificate revocations not supported

Questions?