1. Next Generation Service
@Indeed using gRPC

2. Senior Software Engineer
Jaye
Pitzeruse
Senior Software Engineer
4.5 Indeed
Engineering Capabilities Service Architecture Delivery Tools Pronouns: She / Her / Hers
Twitter
@_jpitz_
Github LinkedIn
jpitz

3. Story of Migration
Today I’m hear to tell a story about evolution and migration as Indeed starts to move it’s service from Boxcar to GRPC.
Historically, Indeed slowly evolved it’s architecture from a monolith to using a service oriented model.
In 2009, we developed Boxcar, Indeed’s proprietary distributed services framework.
In 2012, we announced our use of Boxcar and the improvements it made on our infrastructure.
In 2013, … slide transition … we gave an external tech talk on some of the finer details of the framework.

4. Boxcar (go.indeed.com/boxcar-youtube)
Proxy
Round Robin
In 2013, … slide transition … we gave an external tech talk on some of the finer details of the framework.
For those interested in the finer workings of the framework, please feel free to visit our talk on youtube.
For the purpose of today’s talk, you don’t need to know much about boxcar.
If you are interested in the finer details, please feel free to checkout our techtalk on it.
With that being said, a few things to keep in mind:
Written on top of protocol buffers
It balances connections between servers, not active requests being performed
One ongoing request per connection
The load balancing lies somewhere between a rr and proxy type solution

5. Some Concrete Numbers Boxcar Services ~160 in production
Proven to support high volume traffic
Runs out of box without any additional configuration
Client perceived latency is low
HTTP / RESTful Services
~20 in production
Proxy and Naive Round Robin based
Configuration for these services tend to be manual
Client perceived latency is high
Today, Boxcar still plays a major role in our architecture.
… run over contents of slides …

6. Library Implementation
Front-end Load Balancer
WebApp
WebApp
WebApp B B B
Service
Database
In it’s original implementation, Boxcar existed as a Java library that teams brought into their application for communicating with services.
This common pattern is typically referred to as a “thick client” solution.
Each webapp embeds a small boxcar load balancer
There’s one load balancer per service (one for accountmanagementservice, one for candidatedataservice, etc)
The problem with this approach, was that as we started grow, we also started to adopt new languages.

7. Drawbacks to Library Implementation
Only a few languages with native implementations
Java and Go
Impossible for some languages to support Boxcar
Lots of Development Toil
Implementing the library in languages like python and php would be extremely difficult, almost impossible.
Lots of Development Toil
Hard to test
Tribal knowledge requirement to get started
Slow to iterate on
When we (service architecture) want to make some changes to the core framework, there is an extended roll out cycle to ensure that the wire protocol remains compatible
This means that iterating on the core implementation takes a lot of time (a lot more then we would like)

8. Library Implementation
Front-end Load Balancer
WebApp
WebApp
WebApp B B B
Service
Database
As with any iterative solution, we look back at our original implementation to see what improvements we can make to the system.
One thing that we could do is decouple the boxcar implementation from the web application.
This will allow the web application to be written in any language
And so instead of implementing client libraries in every language, we decided to break it out of the application and run it as a small sidecar process.
… sidecar slide ...

9. Sidecar Runs alongside parent process on same host
Terminates with parent process
Common pattern seen across many companies:
applications-and-services-258a5790a015
This pattern of running another smaller application alongside the parent process has been gaining more and more traction over the last several years.
Canonically, these processes are referred to as sidecar processes.
At Indeed, we call these co-processes, but the community has been talking about this type of pattern for years.
Microsoft has this pattern available in Azure
Voxxed talked about this idea in 2015
Netfix has also talked about Prana

10. Sidecar Implementation
Front-end Load Balancer
WebApp
WebApp
WebApp sc sc sc
Service
Docstore
As a sidecar, we’re now able to solve much of the development toil that we encountered in the library versions.
While we need to continue to maintain backwards compatibility of the wire protocol, we control the release cycle of the sidecars.
This ensures that we can have all sidecar process pick up a new version of the library by a certain date.
… …. …

11. WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
Because engineers are historically bad at naming things, we obviously named this co-process sidecar.
In the original implementation:
A webapp would make an http request to the sidecar process
Sidecar would translate the request to Boxcar and perform communication with the remote service

12. Things Sidecar Solved Only a few languages with native implementations
Java and Go
Impossible for some languages to support Boxcar
Lots of Development Toil
By introducing sidecar, we were able to solve some of these problems.
Most languages have HTTP clients which removes the need to maintain client specific implementations of the load balancer.
Languages like Python and PHP were easily able to start communicating to boxcar service through this mechanism.
Lots of Development Toil
Hard to test
Wrote many tools that make it easier to test boxcar services
Tribal knowledge requirement to get started
Encapsulates complex load balancing logic
Client applications don’t need to be burdened with that responsibility
Now clients just need to know how to construct a request
Slow to iterate on
Now that we control the deployable, we can iterate a bit quicker
Slower than owning a centralized process, but faster than waiting for libraries to be rolled out.
Unfortunately, the introduction of sidecar in it’s original form resulted in additional development toil.
A library was introduced in python that encapsulated the logic for speaking with sidecar from python.
This library included a small code generation footprint so that concrete protobufs could be constructed and encoded in the appropriate manner.

13. The Next Generation
And so we started to really reconsider the landscape of Indeed.
Understanding that the company was growing and starting to adopt new technologies and practices, we wanted to be able to better support new use cases.
And so over the summer, a few colleagues and I went through an innovation rotation.
An innovation rotation is a small period of time (typically 3 months) where individuals can work on things that they find valuable to the company without needing to work directly on product driven requirements.

14. Innovation Rotation Improving REST Supporting gRPC / HTTP2
Service Mesh
During this rotation, we sought to accomplish a few things.
Improving REST
Supporting gRPC / HTTP2
Tested and monitored the overhead of an HTTP2 Connection
Java process support it out of box
Service Mesh
Criteria Establishment
Evaluation of Solutions
Selection

15. Considered V2 Boxcar gRPC REST Sidecar Service Mesh
As with any iterative approach to development, you start to consider what the next generation might look like.

16. Long Term Boxcar gRPC REST Sidecar Service Mesh
And in the long term, deprecating and removing use of boxcar and it’s sidecar counterpart.

17. Boxcar gRPC REST Sidecar Service Mesh
After starting to work on this v2 system, we started to consider how we might migrate our existing services over to using the service mesh and gRPC.
Sidecar posed an interesting position in this process.

18. How can we leverage Sidecar?
Improve performance
Remove toil for new language support
Intermediary for migration
Improve performance
Better use of tcp sockets
Use of a binary protocol more suited towards the protobuf compatability
Remove toil for new language support
No custom libraries
No custom code generation
Intermediary for migration
Put us on a path where we want to go

19. WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
Recall back when we first talked about the communication pattern.
When setting out to improve the performance of a call to Sidecar, a simple solution stood out.

20. WebApp Service Sidecar WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
HTTP 2
Boxcar
WebApp
Service
Sidecar
Using HTTP2 would keep the core server implementation backwards compatible.
All requests are made over a single tcp connection to sidecar instead of needing to spawn a new connection for every concurrent request.
This makes more efficient use of our network space.
It also facilitates the use of a binary protocol vs a text based protocol.

21. WebApp Service Sidecar WebApp Service Sidecar WebApp Service Sidecar
HTTP 1.1
Boxcar
WebApp
Service
Sidecar
HTTP 2
Boxcar
WebApp
Service
Sidecar
gRPC
Boxcar
WebApp
Service
Sidecar
From there, we sought to solve the client language support.
After understanding how Sidecar used the requests that were made to it, it wasn’t too hard to add support for gRPC.
gRPC is a well maintained opensource library that offers code generation across 11 languages.
In the go implementation of gRPC, we have the ability to implement an UnknownServiceHandler.
When a call is made to a server using gRPC, if the service does not exist, this handler is used to handle the request.
From the request, we can parse out the service and method information, then route to the corresponding service load balancer.

22. Generating gRPC Client Code
Installing dependencies for each language was troublesome
Indeed Supports: Java, Golang, Python, PHP, NodeJS
Looked to simplify some of the requirements for the code generation
After adding gRPC support to the server, I started to put together some client libraries.
In doing so, I found the process of installing the dependencies for every language we want to support to be tedious.
Currently, we support 5 languages: … list them …
So I started to look to simplify the process since it would only get worse.

23. grpcgen usage: grpcgen [options] <protos> description:
Uses docker to generate the code for gRPC
options:
-l, --lang <lang> Specify the languages to generate.
-s, --src <src> Specify the source path.
-o, --out <out> Specify the output path.
-r, --registry <registry> Specify the registry to pull from.
-g, --group <group> Specify the group to pull from.
-h, --help Prints this message.

25. Service Mesh
“A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.”
William Morgan CEO, Buoyant
One of the things that I called out to sidecar also doing is acting as an intermediary for migration to our service mesh.
Highnotes:
Centralizes communication to various systems
Responsible for reliable delivery of requests

26. WebApp Service linkerd linkerd
HTTP 2
HTTP 2
linkerd
linkerd
HTTP 2
In our considered solution, we want our communication pattern to looks something like this.
In order for our boxcar service clients and servers to be able to migrate to the service mesh, they first must delegate load balancing to our existing sidecar process.

27. boxcar
WebApp
Service
In the traditional / current boxcar setup, client applications connect directly to the target boxcar server.

28. WebApp Service sidecar
boxcar
HTTP 2
sidecar
By delegating to sidecar, we can start to see our infrastructure start to match the desired end goal.
Our client application can now defer the logic of load balancing and service discovery off to it’s sidecar process, while maintaining performance.

29. WebApp Service sidecar linkerd linkerd
boxcar
HTTP 2
HTTP 2
HTTP 2
sidecar
linkerd
HTTP 2
linkerd
From here, we can start to a/b test different clients / services with the new service mesh that we put into place

30. WebApp Service linkerd linkerd
HTTP 2
HTTP 2
linkerd
HTTP 2
linkerd
Ultimately removing the need for the boxcar protocol and sidecar all together.

31. WebApp Service linkerd linkerd
HTTP 2
HTTP 2
linkerd
linkerd
HTTP 2
Regardless of whether you’re writing RPC or RESTful services at Indeed, the final implementations will be:
Performant
Follow the same request path

32. Benefits to Utilizing a Service Mesh
Encapsulates:
Circuit breaking logic
Load balancing strategy
Service discovery
Consistent request path
Centralize visibility into request flows
Encapsulates:
No longer require language specific implementations for each of these.
Consistency is key
Consistent request path
When applications have different request paths, figuring out what went wrong can be rather difficult.
With a service mesh, RPC and RESTful requests follow the same pattern.
That means that when we need to investigate a service, we have a consistent pattern to follow for both types of services.
Centralize visibility into request flows
By utilizing libraries like opentracing and zipkin, we can better improve the visibility into your application request flow through a single UI.
This helps teams better understand where in the request flow things went wrong.

33. Easy Integration
All communication goes to localhost

34. Where are we today?

35. gRPC @Indeed Sidecar support Boxcar-gRPC Bridge Layer
Generated gRPC client libraries
Golang
Python
NodeJS
Sidecar support
Facilitates communication from non-Java processes
Easily add support for new languages at any time
Boxcar-gRPC Bridge Layer
Utilizes boxcar generated code, but performs communication using gRPC
Kicks the protoc compiler upgrade can down the road
Reduces code change required by applications seeking to migrate
@Indeed, we’re always seeking to make small, iterative changes
Generated gRPC client libraries
Golang
Python
NodeJS

36. In Progress Service mesh support
mysql, redis, ...
gRPC & Java: How do you handle proto3 and hadoop?
Service mesh support
We hit a few delays so we currently don’t have any services using the service mesh
gRPC Java support
gRPC, Java, proto3 hadoop
Very popular question
Common solution is shading
Unanswered question is how to shade (shade in gRPC, shade in hadoop, etc)
In some early compatibility analysis work, I found that the serialization format between 2 and 3 is fairly compatible.
More extensive testing still needs to be done here
Another solution that we’ve considered is using osgi to package deployables
Forces use of application classloader instead of the boot classloader

37. Let’s Recap

38. Inefficiencies in Boxcar
Reasons why we’ve started considering alternative solutions
Issues we’ve encountered as we have started to adopt newer languages

39. Evolving Sidecar using gRPC
Adding support for new languages is easy
Teams can get started at a fraction of the time it used to take them using a pre-built library

40. Leverage Sidecar for the Service Mesh

41. State of the World Using gRPC @Indeed
Finally, ...

42. Thank You

43. Q/A Jaye Pitzeruse Email pitz@indeed Twitter @_jpitz_ Github LinkedIn
Senior Software Engineer
4.5 Indeed
Engineering Capabilities Service Architecture Delivery Tools Pronouns: She / Her / Hers
Twitter
@_jpitz_
Github LinkedIn
jpitz