Project Agumbe: Share Objects Across Namespaces in Kubernetes

At Salesforce, we use Kubernetes to orchestrate our services layer and recently ran into a use case where we wanted to apply and manage certain common objects across Kubernetes namespaces. Since there’s no native solution to share objects across namespaces or the concept of a global object, we used Kubernetes’ extensibility to solve the problem. In this post, I’ll shed light on how we accomplished this.

Namespaced objects in Kubernetes aren’t designed to be shared across logical boundaries. If you want to share a certain object across multiple namespaces, you must scope the object at the cluster level. The inability to scope objects at a higher level has created a gap for certain use cases that we (at Salesforce) have while running Kubernetes in production.

When multiple namespaces (where a namespace may be mapped to a service, tenant, etc) need a reference to a common object such as a Secret or ConfigMap, there isn’t an easy way to make these objects available to desired namespaces other than having to create them in each namespace. This repetitive process is time-consuming, error-prone, and boring.

Let’s take a look at an example. Enterprises rely on a private registry to store container images and application artifacts. When an application pod is deployed, the kubelet uses the sensitive information stored in a Kubernetes Secret object to authenticate against the private registry and pull a private image on behalf of the pod.

We not only have to inject these secrets during namespace provisioning, but we must also be compliant and rotate the secrets at regular intervals. When you have hundreds and thousands of namespaces to manage, the process becomes complicated and cumbersome introducing significant operational overhead such as:

• Rotate Secrets and Certificates
• Update ConfigMaps, PodDisruptionBudgets, Quotas, NetworkPolicy, and role-based access control (RBAC) resources
• Detect config drifts, manual changes, etc

Kubernetes is designed to be highly configurable and extensible. So we decided to make use of the operator and controller patterns to design a solution. We built a custom controller called Agumbe, named after a small coastal town in Karnataka, India, with the primary motive being, “to replicate whatever is thrown at it!!”

---

Architecture

Let’s take a look at the concepts on which Agumbe is built.

The operator framework was first introduced by CoreOS and the Kubernetes community in mid-2018. Operators are a software extension that follows the principle of a control loop. A control loop uses the current cluster state to make intelligent decisions. Operators are clients of the Kubernetes API and act as controllers for a custom resource.

• A custom resource is an extension of the Kubernetes API. It represents a customization of a Kubernetes installation, making it more modular.
• A controller’s responsibility is to move the current state of the cluster closer to the desired state. For example, a replication controller ensures that the desired number of pods are running at all intervals of time.

The most common way to deploy an operator is to add the Custom Resource Definition (CRD) which defines the structure or schema of the custom resource and its associated controller to your cluster. The CRD controller normally runs outside of the control plane, much as you would run any containerized application. The client, who might be a cluster administrator or an application developer interacts with the custom resource to accomplish a certain objective.

Agumbe is a first-class citizen of the Kubernetes APIs and provides system administrators an abstraction of a cluster scoped object.The Agumbe controller introduces a custom resource objectcalled GlobalObject which has a reference to the object that needs to be replicated. When this custom resource is created, updated, or deleted, Agumbe simply replicates the state of the object into the desired namespaces.

The preceding diagram shows how Agumbe works. The control loop constantly watches for any change on the GlobalObject custom resource. When an event occurs, the controller takes the corresponding action of replicating the secret (depicted as Parent Secret in the diagram) from the admin namespace to the other namespaces: proxy, app, and database.

---

Components

Let’s take a look at the components constituting the Agumbe controller.

1. Deployment

• Since we must transform or replicate sensitive data, it’s important to run Agumbe in a namespace following the least permissive model. We used an admin namespace that can only be accessed by cluster administrators.
• In production clusters, we run multiple replicas of the controller and elect a leader to avoid running into a race condition. The election process is triggered during pod:Init and the pod with the lowest priority wins the election and becomes the leader.
• Agumbe is designed to be modular and pluggable, and hence it’s possible to replicate any namespace scoped object through a simple config file change.

# MODULAR AND PLUGGABLE CONFIG

resources.yaml: |-
  ---
  core:
  - name: namespace
    api: CoreV1Api
  - name: sanitize
    api: ApiClient
  scoped:
  - name: secret
    api: CoreV1Api
    convention: secret
  - name: configmap
    api: CoreV1Api
    convention: config_map

2. Role-Based Access Control (RBAC)

Agumbe needs access permissions to monitor the GlobalObject custom resource object changes and to create, read, update, delete objects in the namespace. Hence the controller needs ClusterRole privileges.

The following ClusterRole example has a set of rules, with each rule defining what level of permissions one would like to grant on a specific resource. In the below example, we give Agumbe the permission to replicate Kubernetes Secrets and ConfigMaps to the target namespace (but is not limited to these two resources only).

# CLUSTER-ROLE MANIFEST

rules:
- apiGroups: [""]
  resources: ["secrets", "configmaps"]
  verbs: ["get", "create", "update", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["infra.einstein.ai"]
  resources: ["globalobjects"]
  verbs: ["get", "watch"]
- apiGroups: ["events.k8s.io"]
  resources: ["events"]
  verbs: ["create"]

3. Custom Resource Definition (CRD)

The CRD for globalsecrets.einstein.ai defines the schema for the custom resources that will be created by Kubernetes clients.

# CUSTOM-RESOURCE-DEFINITION MANIFEST

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: globalobjects.infra.einstein.ai
  labels:
    app: agumbe
spec:
  scope: Namespaced
  group: infra.einstein.ai
  version: v1beta
  names:
    kind: GlobalObject
    singular: globalobject
    plural: globalobjects
    shortNames:
    - go
  additionalPrinterColumns:
    - name: Object Type
      type: string
      priority: 0
      JSONPath: .spec.type
      description: Type of the object to replicate into namespace(s)
    - name: Object Name
      type: string
      priority: 0
      JSONPath: .spec.name
      description: Name of the object to replicate into namespace(s)
    - name: Object Target Name
      type: string
      priority: 0
      JSONPath: .spec.targetName
      description: Name of the object in the target namespace(s)
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            type:
              type: string
              description: "Type of the object to replicate"
            name:
              type: string
              description: "Name of the object to replicate"
            targetName:
              type: string
              description: "Name of the object at the target"
            targetNamespaces:
              type: array
              description: "Target namespace(s)"
            matchLabels:
              type: array
              description: "Namespace(s) labels to match"
          required: ["type", "name"]

4. Parent Objects

These are the objects that are replicated to the namespaces. The GlobalObject custom resource specification has a reference to Kubernetes native objects such as a Secret or ConfigMap. Therefore, it’s required that the parent object exist before you create the GlobalObject custom resource.

5. GlobalObject Custom Resource

A custom resource is an extension of the Kubernetes API. After the custom resource is installed, users can create and access its objects using the kubectl client.

---

In Action

Let’s see how the object replication occurs using Agumbe. Before the GlobalObject custom resource is created, let’s verify the labels on the target namespaces and ensure there are no objects of the type we want to replicate.

We shall see why labels on a namespace are important in the subsequent steps.

# GET LABELS ON NAMESPACES

$ for namespace in proxy app database logging metrics; do kubectl get ns $namespace -o=custom-columns=NAMESPACE:.metadata.name,LABELS:.metadata.labels; done
NAMESPACE   LABELS
proxy       map[app=proxy]
NAMESPACE   LABELS
app         map[app=app]
NAMESPACE   LABELS
database    map[app=database]
NAMESPACE   LABELS
logging     map[infra.einstein.ai/namespace:monitoring]
NAMESPACE   LABELS
metrics     map[infra.einstein.ai/namespace:monitoring]


# GET SECRETS IN NAMESPACES

$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace; done
NAME                      NAMESPACE
NAME                      NAMESPACE
NAME                      NAMESPACE
NAME                      NAMESPACE
NAME                      NAMESPACE

From the above results, the namespaces proxy, app, and database have a label key equal to “app” and the label value equal to their function; namespaces logging and metrics have a label key, value pair of “infra.einstein.ai/namespace” and “monitoring” respectively.

Now let’s create a Secret object that we want to replicate in the admin namespace. Let’s name it secret-sep-01-2020 with the data key “hello” and the value set to the base64 encoded result of the string “world”.

# PARENT SECRETS MANIFEST

---
apiVersion: v1
kind: Secret
metadata:
  name: secret-sep-01-2020
  namespace: admin
type: Opaque
data:
  hello: d29ybGQ=

On creating the secret, verify that it exists in the admin namespace.

# GET SECRET IN ADMIN NAMESPACE

$ kubectl get secret -n admin -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace
NAME                                          NAMESPACE
secret-sep-01-2020                            admin

In this step, let’s create the GlobalObject custom resource with the object reference as the Secret created in the previous step (secret-sep-01-2020). To scope the target namespaces, we use both specifications that are available to us:

• .spec.matchLabels: Match all namespaces that have a matching label (key/value pair)
• .spec.targetNamespaces: Explicitly specify the namespaces

The GlobalObject spec shown below replicates the secret called secret-sep-01-2020 as my-secret into,

• namespaces proxy, app, and database since they are described under .spec.targetNamespaces
• namespaces logging and metrics , since they match the key/value pair specified under .spec.matchLabels

# CUSTOM-RESOURCE MANIFEST

---
apiVersion: infra.einstein.ai/v1beta
kind: GlobalObject
metadata:
  name: global-secret
  namespace: admin
spec:
  type: Secret
  name: secret-sep-01-2020
  matchLabels:
  - key: infra.einstein.ai/namespace
    value: monitoring
  targetName: my-secret
  targetNamespaces:
  - proxy
  - app
  - database

After we create the GlobalObject custom resource, verify that the secret has been replicated to the target namespaces. This can be accomplished by making a GET operation call on the object and using output filters to display the desired fields.

# GET GLOBALOBJECT CUSTOM RESOURCE IN ADMIN NAMESPACE

$ kubectl get globalsecret -n admin
NAME              SECRET                 TARGET SECRET NAME
global-secret     secret-sep-01-2020     my-secret


# GET SECRETS IN NAMESPACES

$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace; done
NAME                    NAMESPACE
my-secret               proxy
NAME                    NAMESPACE
my-secret               app
NAME                    NAMESPACE
my-secret               database
NAME                    NAMESPACE
my-secret               logging
NAME                    NAMESPACE
my-secret               metrics


# GET SECRETS (KEY, VALUE) IN NAMESPACES

$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,VALUE:.data; done
NAME                  NAMESPACE   VALUE
my-secret             proxy       map[hello:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             app         map[hello:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             database    map[hello:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             logging     map[hello:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             metrics     map[hello:d29ybGQ=]

We can verify the object replication by observing the controller logs. The command displays the application logs from the Agumbe controller running in admin namespace.

$ kubectl logs deployment/agumbe -n admin -c logger
[2020-09-14 22:21:59,912] [INFO    ] Controller priority 0, setting as elected leader
[2020-09-14 22:23:14,242] [INFO    ] [admin/global-secret] CREATE: GlobalObject "admin/Secret/global-secret" created
[2020-09-14 22:23:14,300] [INFO    ] [admin/global-secret] CREATE: Namespaces ['logging', 'metrics'] matched for label "infra.einstein.ai/namespace=monitoring"
[2020-09-14 22:23:14,345] [INFO    ] [admin/global-secret] CREATE: Secret admin/secret-sep-01-2020 duped to proxy/my-secret
[2020-09-14 22:23:14,351] [INFO    ] [admin/global-secret] CREATE: Secret admin/secret-sep-01-2020 duped to app/my-secret
[2020-09-14 22:23:14,358] [INFO    ] [admin/global-secret] CREATE: Secret admin/secret-sep-01-2020 duped to database/my-secret
[2020-09-14 22:23:14,363] [INFO    ] [admin/global-secret] CREATE: Secret admin/secret-sep-01-2020 duped to logging/my-secret
[2020-09-14 22:23:14,368] [INFO    ] [admin/global-secret] CREATE: Secret admin/secret-sep-01-2020 duped to metrics/my-secret

Next, let’s rotate/update the data stored in the Secret. Create a new Secret called secret-oct-01-2020. Let’s change the data key from “hello” to “world” and leave everything else the same.

# PARENT SECRETS MANIFEST (UPDATED)

---
apiVersion: v1
kind: Secret
metadata:
  name: secret-oct-01-2020
  namespace: admin
type: Opaque
data:
  world: d29ybGQ=

After you verify that the new secret is created, edit the GlobalObject custom resource and change the object reference from secret-sep-01-2020 to secret-oct-01-2020.

# CUSTOM-RESOURCE MANIFEST (UPDATED)

-  name: secret-sep-01-2020
+  name: secret-oct-01-2020

After you apply the change, notice how the Secret data key got updated from “hello” to “world”. There are no other changes to either the data value or the target object name. This confirms that the new Secret value was replicated into the namespaces through a PUT operation.

# GET GLOBALOBJECT CUSTOM RESOURCE IN ADMIN NAMESPACE

$ kubectl get globalsecret -n admin
NAME              SECRET                 TARGET SECRET NAME
global-secret     secret-oct-01-2020     my-secret


# GET SECRETS IN NAMESPACES

$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace; done
NAME                    NAMESPACE
my-secret               proxy
NAME                    NAMESPACE
my-secret               app
NAME                    NAMESPACE
my-secret               database
NAME                    NAMESPACE
my-secret               logging
NAME                    NAMESPACE
my-secret               metrics


# GET SECRETS (KEY, VALUE) IN NAMESPACES

$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,VALUE:.data; done
NAME                  NAMESPACE   VALUE
my-secret             proxy       map[world:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             app         map[world:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             database    map[world:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             logging     map[world:d29ybGQ=]
NAME                  NAMESPACE   VALUE
my-secret             metrics     map[world:d29ybGQ=]

Verify that the change was replicated into the target namespaces by tailing the controller logs. Since we are modifying an existing object, notice that it’s an UPDATE event and not a CREATE event.

$ kubectl logs deploy/agumbe-57b98984bb-kc7qn -n admin -c logger
[2020-09-14 22:49:06,372] [INFO    ] [admin/global-secret] UPDATE: GlobalObject "admin/Secret/global-secret" updated
[2020-09-14 22:49:06,379] [INFO    ] [admin/global-secret] UPDATE: Namespaces ['logging', 'metrics'] matched for label "infra.einstein.ai/namespace=monitoring"
[2020-09-14 22:49:06,394] [INFO    ] [admin/global-secret] UPDATE: Secret admin/secret-sep-01-2020 duped to proxy/my-secret
[2020-09-14 22:49:06,407] [INFO    ] [admin/global-secret] UPDATE: Secret admin/secret-sep-01-2020 duped to app/my-secret
[2020-09-14 22:49:06,418] [INFO    ] [admin/global-secret] UPDATE: Secret admin/secret-sep-01-2020 duped to database/my-secret
[2020-09-14 22:49:06,430] [INFO    ] [admin/global-secret] UPDATE: Secret admin/secret-sep-01-2020 duped to logging/my-secret
[2020-09-14 22:49:06,438] [INFO    ] [admin/global-secret] UPDATE: Secret admin/secret-sep-01-2020 duped to metrics/my-secret

Conclusion

To summarize, Agumbe is a custom Kubernetes controller that provides cluster administrators an abstracted global view of objects scoped to the namespace. The controller enables us to share certain common objects across logical boundaries while being modular and cloud-agnostic at the same time.

---

Acknowledgments

Agumbe was inspired by the Kopf project. A huge shoutout to the amazingly passionate people behind this open-source project. I would also like to thank Arpeet Kale for his valuable feedback during the design and implementation phase, Dianne Siebold for making this document look pretty, and the entire Einstein AI engineering and leadership team (Daniel Tisher, Ivo Mihov, and Indira Iyer) for the constant support and encouragement that we’ve received from day one.