Next Generation Service @Indeed using gRPC

Jaye Pitzeruse Senior Software Engineer 4.5 Years @ Indeed Engineering Capabilities Service Architecture pitz@indeed.com Twitter: @_jpitz_ Github: jpitz Pronouns: She / Her / Hers

Story of Migration Before we’re able to talk about the next generation of services being built at Indeed, we must first understand the landscape of services. <TODO>:: Add rough agenda

Some Concrete Numbers Boxcar Services ~160 in production Proven to support high volume traffic Runs out of box without any additional configuration Client perceived latency is low HTTP / RESTful Services ~20 in production Proxy and Naive Round Robin based Configuration for these services tend to be manual Client perceived latency is high

Boxcar (<insert bitly>) Proxy Round Robin For the purpose of today’s talk, you don’t need to know much about boxcar. A few things to keep in mind: Written on top of protocol buffers It balances connections, not requests One ongoing request per connection The load balancing lies somewhere between a rr and proxy type solution I included a link at the top of these slides in case you’re interested in learning more.

Library Implementation Front-end Load Balancer WebApp WebApp WebApp B B B Service Database In it’s original implementation, Boxcar existed as a Java library that teams brought into their application for communicating with services. This common pattern is typically referred to as a “thick client” solution. Each webapp embeds a small boxcar load balancer There’s one load balancer per service (one for accountmanagementservice, one for candidatedataservice, etc) The problem with this approach, was that as we started to support new languages, they were not able to communicate with the remote services.

Drawbacks to Library Implementation Only a few languages with native implementations Java and Go Impossible for some languages to support boxcar Lots of Development Toil Implementing the library in languages like python and php would be extremely difficult, almost impossible. Lots of Development Toil Hard to test Tribal knowledge requirement to get started Slow to iterate on When we (service architecture) wants to make some changes to the core framework, there is an extended roll out cycle to ensure that the wire protocol remains compatible This means that iterating on the core implementation takes a lot of time (a lot more then we would like)

Library Implementation Front-end Load Balancer WebApp WebApp WebApp B B B Service Database So looking back at the library problem, we can start by decoupling the boxcar implementation from the web application. This will allow the web application to be written in any language And so instead of implementing client libraries in every language, we decided to break it out of the application and run it as a small sidecar process.

Sidecar Implementation Front-end Load Balancer WebApp WebApp WebApp sc sc sc Service Docstore As a sidecar, we’re now able to solve much of the development toil that we encountered in the library versions. While we need to continue to maintain backwards compatibility of the wire protocol, we control the release cycle of the sidecars. This ensures that we can have all sidecar process pick up a new version of the library by a certain date. Because engineers are historically bad at naming things, we obviously named this co-process sidecar.

Sidecar Runs alongside parent process on same host Terminates with parent process Common pattern seen across many companies: https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar https://www.voxxed.com/2015/01/use-container-sidecar-microservices/ https://medium.com/netflix-techblog/prana-a-sidecar-for-your-netflix-paas-based- applications-and-services-258a5790a015

WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar

Things Sidecar Solved Only a few languages with native implementations Java and Go Impossible for some languages to support boxcar Lots of Development Toil By introducing sidecar, we were able to solve some of these problems Lots of Development Toil Hard to test Wrote many tools that make it easier to test boxcar services Tribal knowledge requirement to get started Encapsulates complex load balancing logic Client applications don’t need to be burdened with that responsibility Slow to iterate on Now that we control the deployable, we can iterate a bit quicker Code maintenance Pysidecar (some code generation)

The Next Generation

Innovation Rotation Improving REST Supporting gRPC / HTTP2 Service Mesh Improving REST Supporting gRPC / HTTP2 Tested and monitored the overhead of an HTTP2 Connection Java process support it out of box Service Mesh Criteria Establishment Evaluation of Solutions Selection

Considered V2 Boxcar gRPC REST Sidecar Service Mesh As with any iterative approach to development, you start to consider what the next generation might look like.

Long Term Boxcar gRPC REST Sidecar Service Mesh And in the long term, deprecating and removing use of boxcar and it’s sidecar counterpart.

Boxcar gRPC REST Sidecar Service Mesh After starting to work on this v2 system, I started to think about how we might migrate our existing services over to using the service mesh and .

How can we leverage Sidecar? Improve performance Remove toil for new language support Intermediary for migration Improve performance Better use of tcp sockets Remove toil for new language support No custom libraries No custom code generation Intermediary for migration Put us on a path where we want to go

WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar Recall back when we first talked about the communication pattern.

WebApp Service Sidecar WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar HTTP 2 Boxcar WebApp Service Sidecar A simple improvement that would improve performance of requests made to sidecar would be to add support for HTTP2 support.

WebApp Service Sidecar WebApp Service Sidecar WebApp Service Sidecar HTTP 1.1 Boxcar WebApp Service Sidecar HTTP 2 Boxcar WebApp Service Sidecar gRPC Boxcar WebApp Service Sidecar To remove the toil for new languages that want to speak to sidecar, we were able to add gRPC support. Sidecar’s gRPC server supplies an UnknownServiceHandler that takes the gRPC request, parses out the service and method information, then routes the request to the corresponding boxcar server.

Generating gRPC Client Code Languages Indeed needs to Support: Java, Golang, Python, PHP, NodeJS Installing dependencies for each language was a hassle Looked to simplify some of the requirements for the code generation https://github.com/indeedeng-alpha/grpcgen-docker

grpcgen usage: grpcgen [options] <protos> description: Uses docker to generate the code for gRPC options: -l, --lang <lang> Specify the languages to generate. -s, --src <src> Specify the source path. -o, --out <out> Specify the output path. -r, --registry <registry> Specify the registry to pull from. -g, --group <group> Specify the group to pull from. -h, --help Prints this message.

Service Mesh “A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.” William Morgan CEO, Buoyant http://philcalcado.com/2017/08/03/pattern_service_mesh.html One of the things that I called out to sidecar also doing is acting as an intermediary for migration to our service mesh. TODO: Included bulleted list of high notes

WebApp Service linkerd linkerd HTTP 2 HTTP 2 linkerd linkerd HTTP 2 In our considered solution, we want our communication pattern to looks something like this. In order for our boxcar service clients and servers to be able to migrate to the service mesh, they first must delegate load balancing to our existing sidecar process.

boxcar WebApp Service In your traditional boxcar setup, client applications connect directly to the target boxcar server.

WebApp Service sidecar boxcar HTTP 2 sidecar By delegating to sidecar, we can start to see our infrastructure start to match the desired end goal. Our client application can now defer the logic of load balancing off to it’s sidecar process, while maintaining performance.

WebApp Service sidecar linkerd linkerd boxcar HTTP 2 HTTP 2 HTTP 2 sidecar linkerd HTTP 2 linkerd From here, we can start to a/b test different clients / services with the new service mesh that we put into place

WebApp Service linkerd linkerd HTTP 2 HTTP 2 linkerd HTTP 2 linkerd Ultimately removing the need for the boxcar protocol and sidecar all together.

WebApp Service linkerd linkerd HTTP 2 HTTP 2 linkerd linkerd HTTP 2 Regardless of whether you’re writing RPC or RESTful services at Indeed, the final implementations will be: Performant Follow the same request path

Benefits to Utilizing a Service Mesh Encapsulates: Circuit breaking logic Load balancing strategy Service discovery Consistent request path Centralize visibility into request flows Encapsulates: No longer require language specific implementations for each of these. Consistency is key Consistent request path When applications have different request paths, figuring out what went wrong can be rather difficult. With a service mesh, RPC and RESTful requests follow the same pattern. That means that when we need to investigate a service, we have a consistent pattern to follow for both types of services. Centralize visibility into request flows By utilizing libraries like opentracing and zipkin, we can better improve the visibility into your application request flow through a single UI. This helps teams better understand where in the request flow things went wrong.

Easy Integration All communication goes to localhost

Where are we today?

gRPC @Indeed Sidecar support Generated gRPC client libraries Golang Python NodeJS Sidecar is effectively a gRPC to boxcar bridge Generated clients

In Progress Service mesh support gRPC server support mysql, redis, ... gRPC server support gRPC & Java: How do you handle proto3 and hadoop? https://groups.google.com/forum/#!topic/grpc-io/ABwMhW9bU34 https://github.com/jpitz/proto2-3 Service mesh support We hit a few delays so we currently don’t have any services using the service mesh gRPC server support gRPC, Java, proto3 hadoop Very popular question Common solution is shading Unanswered question is how to shade (shade in gRPC, shade in hadoop, etc) In some early compatibility analysis work, I found that the serialization format between 2 and 3 is fairly compatible. More extensive testing still needs to be done here

Thank You

Q/A Jaye Pitzeruse Senior Software Engineer 4.5 Years @ Indeed Engineering Capabilities Service Architecture pitz@indeed.com Twitter: @_jpitz_ Github: jpitz Pronouns: She / Her / Hers Q/A <TODO> Add resources slide