The adoption of Kubernetes in the cloud-native world, along with the proliferation of service mesh architectures, has popularized the sidecar pattern.
This pattern is named sidecar because it resembles a sidecar attached to a motorcycle. In the pattern, the sidecar is a separate container image that runs alongside the main container image (in the same Kubernetes Pod) and provides supporting features for the application. The sidecar also shares the same lifecycle as the main container application, being created and retired alongside the main application container.
The sidecar model has a number of strengths that make it appealing. It works very well in a polyglot world because you can use the same sidecar for services regardless of what language the services (or the sidecar) are written in. It also serves to decouple common infrastructure requirements from application code. This allows service teams to focus on their application logic, while infrastructure teams provide reusable blocks that run alongside the application container to serve their needs.
At Salesforce, many internal infrastructure teams have written sidecars to provide glue services such as logging, metrics, authorization, and certificate rotation for Kubernetes workloads. (Our team explored this a bit in our Kubecon 2019, San Diego talk.)
When Kubernetes announced the support for mutating admission controllers in 1.9 release, the sidecar pattern really became a “first-class citizen” of Kubernetes. Many of the infrastructure teams at Salesforce independently chose the Kubernetes mutating admission controller webhook model to dynamically inject sidecars in Kubernetes workloads. This worked well until we realized that each of the teams was:
- writing the exact same code
- writing the same set of unit tests and integration tests and producing the same Docker image
- writing the same Helm charts
- debugging the same problems in Kubernetes clusters
At that point we realized we should take a step back, and see if we could derive a common pattern. We discovered that each team was using an annotation on newly created pods to trigger the injection of one or more sidecars (some kind of initcontainers, containers, possibly volumes). Each team had their own annotation namespace (sometimes more than one) and an annotation trigger. If the annotation was present on the pod, that meant it was a target for injection.
For example:
rsyslog.k8s-integration.sfdc.com/inject: true
We looked around in the open source world to see if someone had already solved this generically, such as by providing a generic framework for sidecar injection. We did find at least one open source project, but it didn’t fit our situation particularly well, or meet all of our needs. At that point, we wrote a spec for what an ideal sidecar injector would look like, dropped some code into a new repo, and the generic sidecar injector was born.
Sidecar Configurations and Mutations
At a high level, the generic sidecar injector framework divides the configuration of the mutating webhook admission controller into two parts:
- What needs to be injected ( sidecar configurations)
- What triggers those injections (the mutation configurations)
Separating out these configurations allows teams to specify multiple sidecars and multiple mutations and independently choose which mutation injects which sidecars. This loose coupling supports different teams’ structures, such as if one team is supporting multiple sidecars or each team is supporting just one.
The sidecar configuration format:
initContainers:
- name: rsyslog-init
image: blah
containers:
- name: rsyslog-sidecar
volumes:
- name: rsyslog-spool-vol
The mutation configuration format:
mutationConfigs:
- name: "rsyslog-file-tailer"
annotationNamespace: "rsyslog.k8s-integration.sfdc.com"
annotationTrigger: "inject"
initContainers: ["rsyslog-init"]
containers: ["rsyslog-sidecar"]
volumes: ["rsyslog-spool-vol"]
This design was immediately so useful that multiple teams are already using the framework to inject sidecars, with many more in the process of migration.
Passing Configurations to Sidecars
It’s great that we now we have the ability to inject sidecars. But what if a team that owns a sidecar also wants to dynamically pass a configuration into the injected sidecar which is only available at the time of injection? It turns out we can do this via annotations on the pod. The injected sidecar can define an environment variable whose value is sourced from an annotation in the pod being created. This allows the sidecar team to enable service owners to pass more elaborate configuration via pod annotations:
env:
- name: LOG_TYPES_JSON
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.annotations['rsyslog.k8s-integration.sfdc.com/log-configs']
This not only allows large configurations to be passed to the sidecar, but it also allows the environment variable in an injected sidecar to source values from any part of the pod being created.
Templating of Sidecar Configurations
The environment variable approach works well, but it only supports configuring the environment variables of injected sidecars using different parts of the pod. What if we wanted to configure any part of the injected sidecar (not just environment variables) using annotations passed via pod? An internal team had exactly this requirement, and together we created the solution of templating the sidecar configurations.
Here’s an example. This injected sidecar has a secret whose value needs to be derived from the service account of the pod being created:
volumes:
- name: foo
secret:
### This templated field will come from the pod manifest passed to the mutating webhook controller
secretName: aws-iam-{% .Spec.ServiceAccountName %}
We’ve found this generic mutating webhook framework extremely useful within Salesforce. Not only are multiple teams collaborating on a single codebase, they are helping each other get better, reviewing new changes together, and collaborating on a common platform. And, as promised at Kubecon 2019 San Diego, today I’m very excited to announce that we’re open sourcing this framework at https://github.com/salesforce/generic-sidecar-injector ,so others in the community can use and benefit from it. We’re also excited to see what features or contributions the community will add and the ways in which they will explore its use.
This is just the start. There are many new and interesting problems to be solved with the mutating admission controller model. Some of them include upgrading sidecars, supporting canaries, in-place updates, lifecycle management, a common Helm chart, and more. Stay tuned for more updates in this area.
EDIT: KubeCon EU 2021 talk on Operationalizing Sidecars at Salesforce
This is the result of the combined work of many internal teams who have collaborated outside of their regular work to make this a success. I’d like to specially thank the following people for their contributions to this framework: Robert Wang, Gabriel Intrator, Hayk Baluyan, Quy Lan Anh Ngo, Mike Grass, Matthew Klosak and Prabh Simran Singh.
