Lessons Learned from Microservices at Scale Mike Calvo Target Corp

about this talk Speaker: Basis: Director Engineering Target – target.com MSSE Adjunct faculty 25 Years programming and leadership experience Basis: Lessons learned from building services to support target.com and Target Mobile apps Experience from building non-commerce (non-PCI) micro-services in the cloud

Defining microservices Small, single function endpoints Loosely coupled services that collaborate to provide a product or experience Each service performs a specific thing Single domain Follows OO Single Responsibility Principle Often depends on supporting infrastructure Load balancers, proxies, databases, caching stores, CDN, configuration

Credit: https://cloud.google.com/solutions/scalable-and-resilient-apps Scalability Credit: https://cloud.google.com/solutions/scalable-and-resilient-apps

Real Life Adventure Story Mongo and the Monolith

Lessons from Treachery on Mongo Independent service deployments can save you from total failure External Dependencies: Monitor all access times Circuit break every remote call

Independent Deployments Failures are more isolated Independent scaling Capacity matches demand Auto-scaling independently CPU-based Network-based Memory-based Credit "Building Microservices" by Sam Newman

Monitor Access Times Database Caching layers Dependent services Options Log - ELK Emit metrics – Prometheus or Dropwizard Visualize Grafana Kibana Splunk

Circuit Breaker Pattern for preventing cascading errors from remote resources Wrap a call to a remote resource Define thresholds when circuit should be “open” Timeout Error percentages When circuit is open no calls are made to remote resource Configure circuit to be retried after a period of time Alerts can fire when circuits are open

Netflix Hystrix and Turbine Open source implementation of circuit-breaker pattern Java library for latency and fault-tolerant logic Wrap remote calls with Hystrix Fallback responses Configure timeouts and error thresholds Turbine: Web-base dashboard for displaying connection health and circuit status

Turbine Example Credit Netflix

Other Lessons Stateless services Asynchronous I/O Caching patterns Deploy frequently Protect uptime

Stateless Services Each request allocates and frees memory required to process the request Avoid problem of session affinity Allows any instance to respond to any request Java: avoid using HTTP Session object Solutions for session-based problems Time-expired key value stores (Memcached or Redis) Fast read access data stores (MongoDB)

Asynchronous I/O Traditional forms of system I/O calls are blocking Current thread waits for all data to be read or written before continuing More complex forms of I/O enable notification when operation has completed Enables non-blocking I/O More complex programming model Frees CPU to do more work while waiting for I/O to complete Asynchronous I/O enables higher throughput for I/O-bound processes Network calls Database access File access

Credit: https://www. toptal

Single Event Loop Model Main application thread processes events in a queue Inbound HTTP requests generate events If the handling of a request requires I/O use asynchronous I/O When I/O completes an event is placed on the queue with the result Offer better performance for I/O-bound services

Credit: https://medium

Asychronous I/O Server Platforms Node Javascript I/O performed on background threads Netty/Vertx/Ratpack JVM-based implementation Go Go language developed by Google Natively compiled Go-routines can be suspended based on activity Callback structure abstracted away from developer

Caching Time-to-live (TTL) Micro caching Grace period Soft Hard Micro caching Grace period Clearing considerations Composite keys Edge caching CDN – Akamai, Fastly Proxies – Varnish

Deploy Frequently Team of four engineers should deploy several times per week Deploy small things frequently Requires automated testing User acceptance testing still required Continuous Integration/Continuous Deployment Deployments are Automated and fast

Favoring Speed Though microservices support building with diverse platforms this hasn’t been strong feature for our teams Building on a well-understood tech stack (JVM) leverages efficiencies of deployment, scale and troubleshooting Ruthlessly pursue reducing time from code change to deploy

Common Cloud Deployment Models Virtual Machines Containers Mesos Docker/Kubernettes Serverless

Virtual Machines Each application server runs on its own dedicated VM Full OS instance Select UNIX or Windows flavors Supported natively via cloud provider Provision using cloud console or API Spinnaker Abstraction layer enabling cross-cloud provisioning and deployment Netflix

Containers App runs within a container which is managed by an environment Environment optimizes usage of infrastructure Free memory and processors will be allocated during scaling Disk access managed by environment Examples: Docker and Mesos

Serverless Cloud provider enables deployments of code into platform Ephemeral containers Function as a service API developer not concerned with provisioning of compute resources Typically limited platform options Common options: JavaScript, Python and Java Examples AWS Lambda Google Cloud Functions Microsoft Azure Functions

Protect Uptime Need to be available around the clock Measured in 9s (99.999% up) Monitoring and Alerting Key System metrics Health checks/Aliveness checks Response codes Response times Non functional requirements of all services must include monitoring

Microservices Advantages Supports technology diversity Can be developed with any language on any platform Experimentation Correct tool for the job Resilience Scaling Faster to develop and release Organizational alignment Composability Optimize for replace ability

Microservices Disadvantages/Challenges Complicated deployment and management Sharing of code is harder Technology/Architecture consistency is harder to obtain Decisions on how to partition system

More Details ”Building Microservices” by Sam Newman Martin Fowler Microservices https://martinfowler.com/articles/microservices.html 12 Factor Apps https://12factor.net/