Next Generation Service @Indeed using gRPC

Senior Software Engineer Jaye Pitzeruse Senior Software Engineer 4.5 Years @ Indeed Engineering Capabilities Service Architecture Delivery Tools Pronouns: She / Her / Hers Email pitz@indeed Twitter @_jpitz_ Github LinkedIn jpitz

Story of Migration Today I’m hear to tell a story about evolution and migration as Indeed starts to move it’s service from Boxcar to GRPC. Historically, Indeed slowly evolved it’s architecture from a monolith to using a service oriented model. In 2009, we developed Boxcar, Indeed’s proprietary distributed services framework. In 2012, we announced our use of Boxcar and the improvements it made on our infrastructure. In 2013, … slide transition … we gave an external tech talk on some of the finer details of the framework.

Boxcar (go.indeed.com/boxcar-youtube) Proxy Round Robin In 2013, … slide transition … we gave an external tech talk on some of the finer details of the framework. For those interested in the finer workings of the framework, please feel free to visit our talk on youtube. For the purpose of today’s talk, you don’t need to know much about boxcar. If you are interested in the finer details, please feel free to checkout our techtalk on it. With that being said, a few things to keep in mind: Written on top of protocol buffers It balances connections between servers, not active requests being performed One ongoing request per connection The load balancing lies somewhere between a rr and proxy type solution

Some Concrete Numbers Boxcar Services ~160 in production Proven to support high volume traffic Runs out of box without any additional configuration Client perceived latency is low HTTP / RESTful Services ~20 in production Proxy and Naive Round Robin based Configuration for these services tend to be manual Client perceived latency is high Today, Boxcar still plays a major role in our architecture. … run over contents of slides …

Library Implementation Front-end Load Balancer WebApp WebApp WebApp B B B Service Database In it’s original implementation, Boxcar existed as a Java library that teams brought into their application for communicating with services. This common pattern is typically referred to as a “thick client” solution. Each webapp embeds a small boxcar load balancer There’s one load balancer per service (one for accountmanagementservice, one for candidatedataservice, etc) The problem with this approach, was that as we started grow, we also started to adopt new languages.

Drawbacks to Library Implementation Only a few languages with native implementations Java and Go Impossible for some languages to support Boxcar Lots of Development Toil Implementing the library in languages like python and php would be extremely difficult, almost impossible. Lots of Development Toil Hard to test Tribal knowledge requirement to get started Slow to iterate on When we (service architecture) want to make some changes to the core framework, there is an extended roll out cycle to ensure that the wire protocol remains compatible This means that iterating on the core implementation takes a lot of time (a lot more then we would like)

Library Implementation Front-end Load Balancer WebApp WebApp WebApp B B B Service Database As with any iterative solution, we look back at our original implementation to see what improvements we can make to the system. One thing that we could do is decouple the boxcar implementation from the web application. This will allow the web application to be written in any language And so instead of implementing client libraries in every language, we decided to break it out of the application and run it as a small sidecar process. … sidecar slide ...

Sidecar Runs alongside parent process on same host Terminates with parent process Common pattern seen across many companies: https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar https://www.voxxed.com/2015/01/use-container-sidecar-microservices/ https://medium.com/netflix-techblog/prana-a-sidecar-for-your-netflix-paas-based- applications-and-services-258a5790a015 This pattern of running another smaller application alongside the parent process has been gaining more and more traction over the last several years. Canonically, these processes are referred to as sidecar processes. At Indeed, we call these co-processes, but the community has been talking about this type of pattern for years. Microsoft has this pattern available in Azure Voxxed talked about this idea in 2015 Netfix has also talked about Prana

Sidecar Implementation Front-end Load Balancer WebApp WebApp WebApp sc sc sc Service Docstore As a sidecar, we’re now able to solve much of the development toil that we encountered in the library versions. While we need to continue to maintain backwards compatibility of the wire protocol, we control the release cycle of the sidecars. This ensures that we can have all sidecar process pick up a new version of the library by a certain date. … …. …

WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar Because engineers are historically bad at naming things, we obviously named this co-process sidecar. In the original implementation: A webapp would make an http 1.1 request to the sidecar process Sidecar would translate the request to Boxcar and perform communication with the remote service

Things Sidecar Solved Only a few languages with native implementations Java and Go Impossible for some languages to support Boxcar Lots of Development Toil By introducing sidecar, we were able to solve some of these problems. Most languages have HTTP clients which removes the need to maintain client specific implementations of the load balancer. Languages like Python and PHP were easily able to start communicating to boxcar service through this mechanism. Lots of Development Toil Hard to test Wrote many tools that make it easier to test boxcar services Tribal knowledge requirement to get started Encapsulates complex load balancing logic Client applications don’t need to be burdened with that responsibility Now clients just need to know how to construct a request Slow to iterate on Now that we control the deployable, we can iterate a bit quicker Slower than owning a centralized process, but faster than waiting for libraries to be rolled out. Unfortunately, the introduction of sidecar in it’s original form resulted in additional development toil. A library was introduced in python that encapsulated the logic for speaking with sidecar from python. This library included a small code generation footprint so that concrete protobufs could be constructed and encoded in the appropriate manner.

The Next Generation And so we started to really reconsider the landscape of Indeed. Understanding that the company was growing and starting to adopt new technologies and practices, we wanted to be able to better support new use cases. And so over the summer, a few colleagues and I went through an innovation rotation. An innovation rotation is a small period of time (typically 3 months) where individuals can work on things that they find valuable to the company without needing to work directly on product driven requirements.

Innovation Rotation Improving REST Supporting gRPC / HTTP2 Service Mesh During this rotation, we sought to accomplish a few things. Improving REST Supporting gRPC / HTTP2 Tested and monitored the overhead of an HTTP2 Connection Java process support it out of box Service Mesh Criteria Establishment Evaluation of Solutions Selection

Considered V2 Boxcar gRPC REST Sidecar Service Mesh As with any iterative approach to development, you start to consider what the next generation might look like.

Long Term Boxcar gRPC REST Sidecar Service Mesh And in the long term, deprecating and removing use of boxcar and it’s sidecar counterpart.

Boxcar gRPC REST Sidecar Service Mesh After starting to work on this v2 system, we started to consider how we might migrate our existing services over to using the service mesh and gRPC. Sidecar posed an interesting position in this process.

How can we leverage Sidecar? Improve performance Remove toil for new language support Intermediary for migration Improve performance Better use of tcp sockets Use of a binary protocol more suited towards the protobuf compatability Remove toil for new language support No custom libraries No custom code generation Intermediary for migration Put us on a path where we want to go

WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar Recall back when we first talked about the communication pattern. When setting out to improve the performance of a call to Sidecar, a simple solution stood out.

WebApp Service Sidecar WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar HTTP 2 Boxcar WebApp Service Sidecar Using HTTP2 would keep the core server implementation backwards compatible. All requests are made over a single tcp connection to sidecar instead of needing to spawn a new connection for every concurrent request. This makes more efficient use of our network space. It also facilitates the use of a binary protocol vs a text based protocol.

WebApp Service Sidecar WebApp Service Sidecar WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar HTTP 2 Boxcar WebApp Service Sidecar gRPC Boxcar WebApp Service Sidecar From there, we sought to solve the client language support. After understanding how Sidecar used the requests that were made to it, it wasn’t too hard to add support for gRPC. gRPC is a well maintained opensource library that offers code generation across 11 languages. In the go implementation of gRPC, we have the ability to implement an UnknownServiceHandler. When a call is made to a server using gRPC, if the service does not exist, this handler is used to handle the request. From the request, we can parse out the service and method information, then route to the corresponding service load balancer.

Generating gRPC Client Code Installing dependencies for each language was troublesome Indeed Supports: Java, Golang, Python, PHP, NodeJS Looked to simplify some of the requirements for the code generation https://github.com/indeedeng-alpha/grpcgen-docker After adding gRPC support to the server, I started to put together some client libraries. In doing so, I found the process of installing the dependencies for every language we want to support to be tedious. Currently, we support 5 languages: … list them … So I started to look to simplify the process since it would only get worse.

grpcgen usage: grpcgen [options] <protos> description: Uses docker to generate the code for gRPC options: -l, --lang <lang> Specify the languages to generate. -s, --src <src> Specify the source path. -o, --out <out> Specify the output path. -r, --registry <registry> Specify the registry to pull from. -g, --group <group> Specify the group to pull from. -h, --help Prints this message.

Service Mesh “A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.” William Morgan CEO, Buoyant http://philcalcado.com/2017/08/03/pattern_service_mesh.html One of the things that I called out to sidecar also doing is acting as an intermediary for migration to our service mesh. Highnotes: Centralizes communication to various systems Responsible for reliable delivery of requests

WebApp Service linkerd linkerd HTTP 2 HTTP 2 linkerd linkerd HTTP 2 In our considered solution, we want our communication pattern to looks something like this. In order for our boxcar service clients and servers to be able to migrate to the service mesh, they first must delegate load balancing to our existing sidecar process.

boxcar WebApp Service In the traditional / current boxcar setup, client applications connect directly to the target boxcar server.

WebApp Service sidecar boxcar HTTP 2 sidecar By delegating to sidecar, we can start to see our infrastructure start to match the desired end goal. Our client application can now defer the logic of load balancing and service discovery off to it’s sidecar process, while maintaining performance.

WebApp Service sidecar linkerd linkerd boxcar HTTP 2 HTTP 2 HTTP 2 sidecar linkerd HTTP 2 linkerd From here, we can start to a/b test different clients / services with the new service mesh that we put into place

WebApp Service linkerd linkerd HTTP 2 HTTP 2 linkerd HTTP 2 linkerd Ultimately removing the need for the boxcar protocol and sidecar all together.

WebApp Service linkerd linkerd HTTP 2 HTTP 2 linkerd linkerd HTTP 2 Regardless of whether you’re writing RPC or RESTful services at Indeed, the final implementations will be: Performant Follow the same request path

Benefits to Utilizing a Service Mesh Encapsulates: Circuit breaking logic Load balancing strategy Service discovery Consistent request path Centralize visibility into request flows Encapsulates: No longer require language specific implementations for each of these. Consistency is key Consistent request path When applications have different request paths, figuring out what went wrong can be rather difficult. With a service mesh, RPC and RESTful requests follow the same pattern. That means that when we need to investigate a service, we have a consistent pattern to follow for both types of services. Centralize visibility into request flows By utilizing libraries like opentracing and zipkin, we can better improve the visibility into your application request flow through a single UI. This helps teams better understand where in the request flow things went wrong.

Easy Integration All communication goes to localhost

Where are we today?

gRPC @Indeed Sidecar support Boxcar-gRPC Bridge Layer Generated gRPC client libraries Golang Python NodeJS Sidecar support Facilitates communication from non-Java processes Easily add support for new languages at any time Boxcar-gRPC Bridge Layer Utilizes boxcar generated code, but performs communication using gRPC Kicks the protoc compiler upgrade can down the road Reduces code change required by applications seeking to migrate @Indeed, we’re always seeking to make small, iterative changes Generated gRPC client libraries Golang Python NodeJS

In Progress Service mesh support mysql, redis, ... gRPC & Java: How do you handle proto3 and hadoop? https://groups.google.com/forum/#!topic/grpc-io/ABwMhW9bU34 https://github.com/jpitz/proto2-3 Service mesh support We hit a few delays so we currently don’t have any services using the service mesh gRPC Java support gRPC, Java, proto3 hadoop Very popular question Common solution is shading Unanswered question is how to shade (shade in gRPC, shade in hadoop, etc) In some early compatibility analysis work, I found that the serialization format between 2 and 3 is fairly compatible. More extensive testing still needs to be done here Another solution that we’ve considered is using osgi to package deployables Forces use of application classloader instead of the boot classloader

Let’s Recap

Inefficiencies in Boxcar Reasons why we’ve started considering alternative solutions Issues we’ve encountered as we have started to adopt newer languages

Evolving Sidecar using gRPC Adding support for new languages is easy Teams can get started at a fraction of the time it used to take them using a pre-built library

Leverage Sidecar for the Service Mesh

State of the World Using gRPC @Indeed Finally, ...

Thank You

Q/A Jaye Pitzeruse Email pitz@indeed Twitter @_jpitz_ Github LinkedIn Senior Software Engineer 4.5 Years @ Indeed Engineering Capabilities Service Architecture Delivery Tools Pronouns: She / Her / Hers Email pitz@indeed Twitter @_jpitz_ Github LinkedIn jpitz https://go.indeed.com/boxcar-youtube https://github.com/indeedeng-alpha/grpcgen-docker http://philcalcado.com/2017/08/03/pattern_service_mesh.html https://groups.google.com/forum/#!topic/grpc-io/ABwMhW9bU34 https://github.com/jpitz/proto2-3