1. Next Generation Service
@Indeed using gRPC

2. Jaye Pitzeruse Senior Software Engineer 4.5 Years @ Indeed
Engineering Capabilities Service Architecture
Github: jpitz Pronouns: She / Her / Hers

3. Story of Migration
Before we’re able to talk about the next generation of services being built at Indeed, we must first understand the landscape of services.
<TODO>:: Add rough agenda

4. Some Concrete Numbers Boxcar Services ~160 in production
Proven to support high volume traffic
Runs out of box without any additional configuration
Client perceived latency is low
HTTP / RESTful Services
~20 in production
Proxy and Naive Round Robin based
Configuration for these services tend to be manual
Client perceived latency is high

5. Boxcar (<insert bitly>)
Proxy
Round Robin
For the purpose of today’s talk, you don’t need to know much about boxcar.
A few things to keep in mind:
Written on top of protocol buffers
It balances connections, not requests
One ongoing request per connection
The load balancing lies somewhere between a rr and proxy type solution
I included a link at the top of these slides in case you’re interested in learning more.

6. Library Implementation
Front-end Load Balancer
WebApp
WebApp
WebApp B B B
Service
Database
In it’s original implementation, Boxcar existed as a Java library that teams brought into their application for communicating with services. This common pattern is typically referred to as a “thick client” solution.
Each webapp embeds a small boxcar load balancer
There’s one load balancer per service (one for accountmanagementservice, one for candidatedataservice, etc)
The problem with this approach, was that as we started to support new languages, they were not able to communicate with the remote services.

7. Drawbacks to Library Implementation
Only a few languages with native implementations
Java and Go
Impossible for some languages to support boxcar
Lots of Development Toil
Implementing the library in languages like python and php would be extremely difficult, almost impossible.
Lots of Development Toil
Hard to test
Tribal knowledge requirement to get started
Slow to iterate on
When we (service architecture) wants to make some changes to the core framework, there is an extended roll out cycle to ensure that the wire protocol remains compatible
This means that iterating on the core implementation takes a lot of time (a lot more then we would like)

8. Library Implementation
Front-end Load Balancer
WebApp
WebApp
WebApp B B B
Service
Database
So looking back at the library problem, we can start by decoupling the boxcar implementation from the web application.
This will allow the web application to be written in any language
And so instead of implementing client libraries in every language, we decided to break it out of the application and run it as a small sidecar process.

9. Sidecar Implementation
Front-end Load Balancer
WebApp
WebApp
WebApp sc sc sc
Service
Docstore
As a sidecar, we’re now able to solve much of the development toil that we encountered in the library versions.
While we need to continue to maintain backwards compatibility of the wire protocol, we control the release cycle of the sidecars.
This ensures that we can have all sidecar process pick up a new version of the library by a certain date.
Because engineers are historically bad at naming things, we obviously named this co-process sidecar.

10. Sidecar Runs alongside parent process on same host
Terminates with parent process
Common pattern seen across many companies:
applications-and-services-258a5790a015

11. WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar

12. Things Sidecar Solved Only a few languages with native implementations
Java and Go
Impossible for some languages to support boxcar
Lots of Development Toil
By introducing sidecar, we were able to solve some of these problems
Lots of Development Toil
Hard to test
Wrote many tools that make it easier to test boxcar services
Tribal knowledge requirement to get started
Encapsulates complex load balancing logic
Client applications don’t need to be burdened with that responsibility
Slow to iterate on
Now that we control the deployable, we can iterate a bit quicker
Code maintenance
Pysidecar (some code generation)

13. The Next Generation

14. Innovation Rotation Improving REST Supporting gRPC / HTTP2
Service Mesh
Improving REST
Supporting gRPC / HTTP2
Tested and monitored the overhead of an HTTP2 Connection
Java process support it out of box
Service Mesh
Criteria Establishment
Evaluation of Solutions
Selection

15. Considered V2 Boxcar gRPC REST Sidecar Service Mesh
As with any iterative approach to development, you start to consider what the next generation might look like.

16. Long Term Boxcar gRPC REST Sidecar Service Mesh
And in the long term, deprecating and removing use of boxcar and it’s sidecar counterpart.

17. Boxcar gRPC REST Sidecar Service Mesh
After starting to work on this v2 system, I started to think about how we might migrate our existing services over to using the service mesh and .

18. How can we leverage Sidecar?
Improve performance
Remove toil for new language support
Intermediary for migration
Improve performance
Better use of tcp sockets
Remove toil for new language support
No custom libraries
No custom code generation
Intermediary for migration
Put us on a path where we want to go

19. WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
Recall back when we first talked about the communication pattern.

20. WebApp Service Sidecar WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
HTTP 2
Boxcar
WebApp
Service
Sidecar
A simple improvement that would improve performance of requests made to sidecar would be to add support for HTTP2 support.

21. WebApp Service Sidecar WebApp Service Sidecar WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
HTTP 2
Boxcar
WebApp
Service
Sidecar
gRPC
Boxcar
WebApp
Service
Sidecar
To remove the toil for new languages that want to speak to sidecar, we were able to add gRPC support.
Sidecar’s gRPC server supplies an UnknownServiceHandler that takes the gRPC request, parses out the service and method information, then routes the request to the corresponding boxcar server.

22. Generating gRPC Client Code
Languages Indeed needs to Support: Java, Golang, Python, PHP, NodeJS
Installing dependencies for each language was a hassle
Looked to simplify some of the requirements for the code generation

23. grpcgen usage: grpcgen [options] <protos> description:
Uses docker to generate the code for gRPC
options:
-l, --lang <lang> Specify the languages to generate.
-s, --src <src> Specify the source path.
-o, --out <out> Specify the output path.
-r, --registry <registry> Specify the registry to pull from.
-g, --group <group> Specify the group to pull from.
-h, --help Prints this message.

25. Service Mesh
“A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.”
William Morgan CEO, Buoyant
One of the things that I called out to sidecar also doing is acting as an intermediary for migration to our service mesh.
TODO: Included bulleted list of high notes

26. WebApp Service linkerd linkerd
HTTP 2
HTTP 2
linkerd
linkerd
HTTP 2
In our considered solution, we want our communication pattern to looks something like this.
In order for our boxcar service clients and servers to be able to migrate to the service mesh, they first must delegate load balancing to our existing sidecar process.

27. boxcar
WebApp
Service
In your traditional boxcar setup, client applications connect directly to the target boxcar server.

28. WebApp Service sidecar
boxcar
HTTP 2
sidecar
By delegating to sidecar, we can start to see our infrastructure start to match the desired end goal.
Our client application can now defer the logic of load balancing off to it’s sidecar process, while maintaining performance.

29. WebApp Service sidecar linkerd linkerd
boxcar
HTTP 2
HTTP 2
HTTP 2
sidecar
linkerd
HTTP 2
linkerd
From here, we can start to a/b test different clients / services with the new service mesh that we put into place

30. WebApp Service linkerd linkerd
HTTP 2
HTTP 2
linkerd
HTTP 2
linkerd
Ultimately removing the need for the boxcar protocol and sidecar all together.

31. WebApp Service linkerd linkerd
HTTP 2
HTTP 2
linkerd
linkerd
HTTP 2
Regardless of whether you’re writing RPC or RESTful services at Indeed, the final implementations will be:
Performant
Follow the same request path

32. Benefits to Utilizing a Service Mesh
Encapsulates:
Circuit breaking logic
Load balancing strategy
Service discovery
Consistent request path
Centralize visibility into request flows
Encapsulates:
No longer require language specific implementations for each of these.
Consistency is key
Consistent request path
When applications have different request paths, figuring out what went wrong can be rather difficult.
With a service mesh, RPC and RESTful requests follow the same pattern.
That means that when we need to investigate a service, we have a consistent pattern to follow for both types of services.
Centralize visibility into request flows
By utilizing libraries like opentracing and zipkin, we can better improve the visibility into your application request flow through a single UI.
This helps teams better understand where in the request flow things went wrong.

33. Easy Integration
All communication goes to localhost

34. Where are we today?

35. gRPC @Indeed Sidecar support Generated gRPC client libraries Golang
Python
NodeJS
Sidecar is effectively a gRPC to boxcar bridge
Generated clients

36. In Progress Service mesh support gRPC server support
mysql, redis, ...
gRPC server support
gRPC & Java: How do you handle proto3 and hadoop?
Service mesh support
We hit a few delays so we currently don’t have any services using the service mesh
gRPC server support
gRPC, Java, proto3 hadoop
Very popular question
Common solution is shading
Unanswered question is how to shade (shade in gRPC, shade in hadoop, etc)
In some early compatibility analysis work, I found that the serialization format between 2 and 3 is fairly compatible.
More extensive testing still needs to be done here

37. Thank You

38. Q/A Jaye Pitzeruse Senior Software Engineer 4.5 Years @ Indeed
Engineering Capabilities Service Architecture
Github: jpitz Pronouns: She / Her / Hers
Q/A
<TODO>
Add resources slide