1. Lessons Learned from Microservices at Scale
Mike Calvo
Target Corp

2. about this talk Speaker: Basis:
Director Engineering Target – target.com
MSSE Adjunct faculty
25 Years programming and leadership experience
Basis:
Lessons learned from building services to support target.com and Target Mobile apps
Experience from building non-commerce (non-PCI) micro-services in the cloud

3. Defining microservices
Small, single function endpoints
Loosely coupled services that collaborate to provide a product or experience
Each service performs a specific thing
Single domain
Follows OO Single Responsibility Principle
Often depends on supporting infrastructure
Load balancers, proxies, databases, caching stores, CDN, configuration

4. Credit: https://cloud.google.com/solutions/scalable-and-resilient-apps
Scalability
Credit:

5. Real Life Adventure Story
Mongo and the Monolith

6. Lessons from Treachery on Mongo
Independent service deployments can save you from total failure
External Dependencies:
Monitor all access times
Circuit break every remote call

7. Independent Deployments
Failures are more isolated
Independent scaling
Capacity matches demand
Auto-scaling independently
CPU-based
Network-based
Memory-based
Credit "Building Microservices" by Sam Newman

8. Monitor Access Times Database Caching layers Dependent services
Options
Log - ELK
Emit metrics – Prometheus or Dropwizard
Visualize
Grafana
Kibana
Splunk

9. Circuit Breaker
Pattern for preventing cascading errors from remote resources
Wrap a call to a remote resource
Define thresholds when circuit should be “open”
Timeout
Error percentages
When circuit is open no calls are made to remote resource
Configure circuit to be retried after a period of time
Alerts can fire when circuits are open

10. Netflix Hystrix and Turbine
Open source implementation of circuit-breaker pattern
Java library for latency and fault-tolerant logic
Wrap remote calls with Hystrix
Fallback responses
Configure timeouts and error thresholds
Turbine:
Web-base dashboard for displaying connection health and circuit status

11. Turbine Example
Credit Netflix

12. Other Lessons Stateless services Asynchronous I/O Caching patterns
Deploy frequently
Protect uptime

13. Stateless Services
Each request allocates and frees memory required to process the request
Avoid problem of session affinity
Allows any instance to respond to any request
Java: avoid using HTTP Session object
Solutions for session-based problems
Time-expired key value stores (Memcached or Redis)
Fast read access data stores (MongoDB)

14. Asynchronous I/O Traditional forms of system I/O calls are blocking
Current thread waits for all data to be read or written before continuing
More complex forms of I/O enable notification when operation has completed
Enables non-blocking I/O
More complex programming model
Frees CPU to do more work while waiting for I/O to complete
Asynchronous I/O enables higher throughput for I/O-bound processes
Network calls
Database access
File access

15. Credit: https://www. toptal

16. Single Event Loop Model
Main application thread processes events in a queue
Inbound HTTP requests generate events
If the handling of a request requires I/O use asynchronous I/O
When I/O completes an event is placed on the queue with the result
Offer better performance for I/O-bound services

17. Credit: https://medium

18. Asychronous I/O Server Platforms
Node
Javascript
I/O performed on background threads
Netty/Vertx/Ratpack
JVM-based implementation Go
Go language developed by Google
Natively compiled
Go-routines can be suspended based on activity
Callback structure abstracted away from developer

19. Caching Time-to-live (TTL) Micro caching Grace period
Soft
Hard
Micro caching
Grace period
Clearing considerations
Composite keys
Edge caching
CDN – Akamai, Fastly
Proxies – Varnish

20. Deploy Frequently
Team of four engineers should deploy several times per week
Deploy small things frequently
Requires automated testing
User acceptance testing still required
Continuous Integration/Continuous Deployment
Deployments are Automated and fast

21. Favoring Speed
Though microservices support building with diverse platforms this hasn’t been strong feature for our teams
Building on a well-understood tech stack (JVM) leverages efficiencies of deployment, scale and troubleshooting
Ruthlessly pursue reducing time from code change to deploy

22. Common Cloud Deployment Models
Virtual Machines
Containers
Mesos
Docker/Kubernettes
Serverless

23. Virtual Machines Each application server runs on its own dedicated VM
Full OS instance
Select UNIX or Windows flavors
Supported natively via cloud provider
Provision using cloud console or API
Spinnaker
Abstraction layer enabling cross-cloud provisioning and deployment
Netflix

24. Containers
App runs within a container which is managed by an environment
Environment optimizes usage of infrastructure
Free memory and processors will be allocated during scaling
Disk access managed by environment
Examples: Docker and Mesos

25. Serverless Cloud provider enables deployments of code into platform
Ephemeral containers
Function as a service
API developer not concerned with provisioning of compute resources
Typically limited platform options
Common options: JavaScript, Python and Java
Examples
AWS Lambda
Google Cloud Functions
Microsoft Azure Functions

26. Protect Uptime Need to be available around the clock
Measured in 9s (99.999% up)
Monitoring and Alerting Key
System metrics
Health checks/Aliveness checks
Response codes
Response times
Non functional requirements of all services must include monitoring

27. Microservices Advantages
Supports technology diversity
Can be developed with any language on any platform
Experimentation
Correct tool for the job
Resilience
Scaling
Faster to develop and release
Organizational alignment
Composability
Optimize for replace ability

28. Microservices Disadvantages/Challenges
Complicated deployment and management
Sharing of code is harder
Technology/Architecture consistency is harder to obtain
Decisions on how to partition system

29. More Details ”Building Microservices” by Sam Newman
Martin Fowler Microservices
12 Factor Apps