Notary: A Certificate Lifecycle Management Controller for Kubernetes

Introduction

All services in the Einstein Vision and Language Platform use TLS/SSL certificates to encrypt communication between microservices. The certificates are generated in AWS Certificate Manager (ACM) and stored in the AWS Secrets Manager in the form of keystores and truststores (private and public keys). Certificate creation can be a manual process, especially if the permission model dictates that a certificate can only be provisioned or de-provisioned by an AWS account admin.

Certificate management includes provisioning certificates and monitoring their expiration and renewal. AWS ACM provides the ability to auto-renew only public certs attached to AWS native resources (such as ELB and RDS). Einstein Vision and Language Platform services run on an EKS cluster with service exposure scoped to the cluster network, which means that the ACM certificates aren’t attached to AWS native resources. Hence, the certificate auto-renewal feature provided by AWS ACM can’t be used.

To automate certificate provisioning and management and to provide a self-service model for carrying out these activities, we built a custom Kubernetes controller by introducing Kubernetes native resources: certificate, keystore, and truststore. Additionally, we built automation for checking certificate expiration and alert on expiration of these certificates. We call our solution Notary.

Architecture

Notary is built on the core Kubernetes concepts:

• Controllers
• Custom Resource Definition (CRD)

What is a Kubernetes controller?

In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state. A controller tracks at least one Kubernetes resource type. These objects have a spec field that represents the desired state. The controller for that resource is responsible for performing actions to achieve the desired state. For example, the job controller, which is a built-in Kubernetes controller, helps to execute tasks in Kubernetes cluster by deploying pods as dictated by the pod spec.

The Kubernetes API can be extended to create application-specific controllers to create, configure, and manage instances of complex, stateful applications on behalf of a Kubernetes user. The custom Kubernetes controller builds upon the Kubernetes resource and controller concepts, and includes domain or application-specific knowledge to automate common tasks.

What is a CRD?

CRD stands for Custom Resource Definition.

A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.

A custom resource is an extension of the Kubernetes API that provides functionality that isn’t available in a default Kubernetes installation. A custom resource represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.

A custom resource can appear and disappear in a running cluster through dynamic registration. And cluster admins can update a custom resource independently of the cluster itself. After a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like pods.

Notary

Notary is an extension to the Kubernetes API to satisfy an application-specific use case — encryption of traffic between services running on Kubernetes cluster — by managing the certificates and their lifecycle.

Notary is a custom Kubernetes controller that watches on these custom resources: certificate, keystore, and truststore. The controller watches for create and update events, and takes defined action based on those events.

Notary interacts with AWS Certificate Manager to request and download certificates to create keystores and truststores. After it creates keystores and truststores, Notary initiates a session with AWS Secrets Manager to upload them.

The architecture diagram above depicts the functionality of Notary. The Notary controller constantly watches the custom resources — certificate, keystore, and truststore—for a create, update, or delete event.

• Certificate: When the create event for the certificate resource occurs, the Notary controller requests a new certificate from AWS Certificate Manager.
• Keystore: When the create event for the keystore resource occurs, the Notary controller downloads the service’s certificate bundle from AWS Certificate Manager, creates a keystore, and then uploads the keystore to AWS Secrets Manager.
• Truststore: When the create event for the truststore resource occurs, the Notary controller downloads the service’s certificate bundle along with upstream and downstream services’ certificates from AWS Certificate Manager. It then builds a truststore which includes the service certificate chain along with upstream and downstream service certificates, and uploads the truststore to AWS Secrets Manager.

Components

Notary is built using Kubernetes objects. Let’s take a look at the components of Notary.

Deployment

Since we’re dealing with sensitive data — certificate, keystore, and truststore—for services, it’s important to secure the Notary pods. To address that, Notary is deployed in its own namespace where the namespace resources can only be accessed by cluster administrators.

RBAC

RBAC, role-based access control, is a way to manage access to resources based on roles. Notary needs access to the CRD objects certificate, keystore, and truststore which can be deployed in any service-specific namespace. Since Notary needs to watch all namespaces it needs ClusterRole privileges.

These examples use ClusterRole and ClusterRoleBinding to give Notary access to specific resources and assign a certain level of access to resources.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    operator: notary
  name: notary
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
- apiGroups:
  - infra.einstein.ai
  resources:
  - certificates
  - keystores
  - truststores
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - events.k8s.io
  - ""
  resources:
  - events
  verbs:
  - createapiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: notary
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: notary
subjects:
- kind: ServiceAccount
  name: notary
  namespace: notary

IAM Roles

Notary needs access to AWS Certificate Manager to request new certificates and access to AWS Secrets Manager to upload keystores and truststores. We use AWS IAM (Identity and Access Management) roles to grant access to AWS services required for Notary. The following JSON is an example of an IAM role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "acm:RequestCertificate",
                "acm:GetCertificate",
                "acm:ListCertificates",
                "acm:ExportCertificate",
                "acm:UpdateCertificateOptions",
                "acm:AddTagsToCertificate",
                "acm:ListTagsForCertificate",
                "acm:DescribeCertificate",
                "acm:ResendValidationEmail",
                "acm:RemoveTagsFromCertificate",
                "acm-pca:GetCertificate",
                "acm-pca:IssueCertificate",
                "acm-pca:ListCertificateAuthorities"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "secretsmanager:UntagResource",
                "secretsmanager:DescribeSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:GetSecretValue",
                "secretsmanager:CreateSecret",
                "secretsmanager:DeleteSecret",
                "secretsmanager:ListSecretVersionIds",
                "secretsmanager:GetRandomPassword",
                "secretsmanager:GetSecretValue",
                "secretsmanager:RestoreSecret",
                "secretsmanager:RotateSecret",
                "secretsmanager:CancelRotateSecret",
                "secretsmanager:UpdateSecret",
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:UpdateSecretVersionStage",
                "secretsmanager:ListSecrets",
                "secretsmanager:TagResource"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

CRD

Certificate CRD

This custom resource defines the metadata required to request a certificate from AWS Certificate Manager. The metadata includes the certificate FQDN and type (public or private) of certificate.

# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: certificates.infra.einstein.ai
  labels:
    operator: {{ .Values.app.name }}
spec:
  scope: Namespaced
  group: infra.einstein.ai
  version: v1alpha1
  names:
    kind: Certificate
    singular: certificate
    plural: certificates
    shortNames:
    - cert
  additionalPrinterColumns:
    - name: Type
      type: string
      JSONPath: .spec.type
      description: Type of certificate
    - name: FQDN
      type: string
      priority: 0
      JSONPath: .spec.fqdn
      description: FQDN of the certificate
    - name: AltName
      type: string
      priority: 0
      JSONPath: .spec.alt
      description: Alternate names of the certificate
    - name: ARN
      type: string
      priority: 0
      JSONPath: .status.arn
      description: ARN of the certificate
    - name: Tags
      type: string
      priority: 0
      JSONPath: .status.tags
      description: Tags on the certificate
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            type:
              type: string
              description: "AWS Certificate Manager PEM certificate issued by either a Private of Public Root CA."
              enum: ["Public", "Private"]
            fqdn:
              type: string
              description: "FQDN of the certificate"
            alt:
              type: array
              description: "Alternate name(s) for the certifcate"
          required: ["type", "fqdn"]

When the certificate CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager for a new certificate.

Required parameters

• type: Type of certificate, either public or private
• fqdn: Fully qualified domain name of the service

Keystore CRD

This custom resource defines the keystore resource. The resource spec accepts metadata to generate a new keystore for a particular service and then uploads the keystore to AWS Secrets Manager.

# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: keystores.infra.einstein.ai
  labels:
    operator: {{ .Values.app.name }}
spec:
  scope: Namespaced
  group: infra.einstein.ai
  version: v1alpha1
  names:
    kind: Keystore
    singular: keystore
    plural: keystores
    shortNames:
    - ks
  additionalPrinterColumns:
    - name: FQDN
      type: string
      priority: 0
      JSONPath: .spec.fqdn
      description: FQDN of the certificate
    - name: Certificate Name
      type: string
      priority: 0
      JSONPath: .spec.certName
      description: Name of the certificate that need to be included in the keystore
    - name: ARN
      type: string
      priority: 0
      JSONPath: .status.arn
      description: ARN of the keystore created by Notary which is uploaded as secret in AWS secrets manager
    - name: Tags
      type: string
      priority: 0
      JSONPath: .status.tags
      description: Tags on the certificate
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            certName:
              type: string
              description: "Name of the certificate that will be included in the keystore"
            fqdn:
              type: string
              description: "FQDN of the service of which certificate is created"
          required: ["fqdn", "certName"]

When the keystore CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager to download the service certificate, builds the keystore, and uploads the keystore to AWS Secrets Manager.

Required parameters

• certName: Name tag of the newly requested certificate
• fqdn: Fully qualified domain name of the service

Truststore CRD

This custom resource defines the truststore resource. The resource spec accepts service metadata along with the service FQDN and certificate names of the upstream and downstream services.

# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: truststores.infra.einstein.ai
  labels:
    operator: {{ .Values.app.name }}
spec:
  scope: Namespaced
  group: infra.einstein.ai
  version: v1alpha1
  names:
    kind: Truststore
    singular: truststore
    plural: truststores
    shortNames:
    - ts
  additionalPrinterColumns:
    - name: FQDN
      type: string
      priority: 0
      JSONPath: .spec.fqdn
      description: FQDN of the certificate
    - name: ARN
      type: string
      priority: 0
      JSONPath: .status.arn
      description: ARN of the truststore created by Notary which is uploaded as secret in AWS secrets manager
    - name: Tags
      type: string
      priority: 0
      JSONPath: .status.tags
      description: Tags on the certificate
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            upstream:
              type: array
              description: "Tags of the upstream service(s) certificate that talk to this service"
            downstream:
              type: array
              description: "Tags of the downstream service(s) certificate that this service talks to"
            fqdn:
              type: string
              description: "FQDN of the service of which certificate is created"
            certName:
              type: string
              description: "Name  of the new certificate that need to be included in the truststore"
          required: ["fqdn", "upstream", "downstream", "certName"]

Required parameters

• upstream: Tags of the upstream service(s) certificate that talks to this service
• downstream: Tags of the downstream service(s) certificate that this service talks to
• fqdn: Fully qualified domain name of this service
• certName: Name tag of the newly-requested certificate

Notary In Action

Now let’s take a look at Notary in action. Consider a sample application, with a service named test-service. The test-service service talks to a proxy service and a database service. To have encrypted service-to-service communication, we must have certificates for each service and the resulting keystore and truststore.

Pre-requisites

As mentioned earlier, in addition to creating certificates, Notary also creates keystores and truststores. Both keystores and truststores are sensitive data, so they need to be protected. Notary password protects these artifacts at the time of creation so in case of unintended/unauthorized access the keystore and truststore can’t be read without their corresponding password.

This implies there has to be a way to share the keystore and truststore credentials with the service that uses them to encrypt/decrypt data. Currently, we handle this by creating a metadata secret for every service. This is a one-time admin-triggered automation (more on that in future blog post), part of service onboarding on our infrastructure.

This metadata secret is an encoded JSON secret that follows the format shown here.

{
  "tlsStoreRootPath": "tls",
  "tlsKeyStoreName": "test-key-store",
  "tlsTrustStoreName": "test-trust-store",
  "tlsKeyStorePassword": "test-key-store-password",
  "tlsTrustStorePassword": "test-trust-store-password"
}

After the service is onboarded and the metadata secret is created, Notary is ready to manage certificates, keystores, and truststores for that service.

Step 1: Create the Certificate

To create certificates for the test service, proxy service, and database service, we must create certificate objects for all three services. Following is a Certificate custom resource YAML spec for test-service. Similarly, certificates can be created for the proxy service and database service.

apiVersion: infra.einstein.ai/v1alpha1
kind: Certificate
metadata:
  generation: 1
  labels:
    env: DEV
  name: test-service
  namespace: test-service
spec:
  alt:
  - test-service.localhost
  fqdn: test-service.test-service.svc.cluster.local
  type: Private

The above spec can be saved as test-service.yaml file. Let’s use this file to create the test-service certificate object in the EKS cluster using the following command.

kubectl apply -f test-service.yaml

After you apply the object in the cluster, Notary requests AWS Certificate Manager to create a private certificate with the name test-service and an FQDN test-service.test-service.svc.cluster.local.

We can also verify in the controller logs that the private certificate was created.

[2021-03-03 21:40:45,709] kopf.objects         [INFO    ] [test-service/test-service] Updated new certificate passphrase for service test-service
[2021-03-03 21:40:45,713] kopf.objects         [INFO    ] [test-service/test-service] Handler 'processCertificate' succeeded.
[2021-03-03 21:40:45,713] kopf.objects         [INFO    ] [test-service/test-service] Creation event is processed: 1 succeeded; 0 failed.

After we create certificates for all three services, we can also list the Kubernetes Certificate objects using this command.

$ kubectl get cert --all-namespaces
NAME               TYPE      FQDN                                                   
db-new             Private   db-service.db-service.svc.cluster.local         
proxy-new          Private   proxy-service.proxy-service.svc.cluster.local  
test-service-new   Private   test-service.test-service.svc.cluster.local     

Step 2: Create the Keystore

The next step is to create the keystore for test-service. To create the keystore, apply the following spec in the cluster for test-service in test-service namespace.

---
apiVersion: infra.einstein.ai/v1alpha1
kind: Keystore
metadata:
  name: test-service-key-store
  namespace: test-service
spec:
  fqdn: test-service.test-service.svc.cluster.local
  certName: test-service-new

Now let’s create the test-service keystore object in the EKS cluster using this command.

kubectl apply -f test-service-keystore.yaml

After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local and creates the keystore for the private key. The keystore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application/service during deployment using its own IAM role.

We can verify the creation of the keystore from the controller logs as shown here.

[2020-10-27 23:44:43,596] kopf.objects         [INFO    ] [test-service/test-service-key-store] Downloaded Certificate for service test-service-new
[2020-10-27 23:44:43,598] kopf.objects         [INFO    ] [test-service/test-service-key-store] Downloaded CertificateChain for service test-service-new
[2020-10-27 23:44:43,599] kopf.objects         [INFO    ] [test-service/test-service-key-store] Downloaded PrivateKey for service test-service-new
[2020-10-27 23:44:43,612] kopf.objects         [INFO    ] [test-service/test-service-key-store] Created keystore successfully for test-service
[2020-10-27 23:44:43,694] kopf.objects         [INFO    ] [test-service/test-service-key-store] Updated secret: test-service-key-store
[2020-10-27 23:44:43,696] kopf.objects         [INFO    ] [test-service/test-service-key-store] Secret ARN: arn:aws:secretsmanager:us-west-2:<account-id>:secret:test-service-key-store-ZimHEX
[2020-10-27 23:44:43,701] kopf.objects         [INFO    ] [test-service/test-service-key-store] Handler 'processKeystore' succeeded.
[2020-10-27 23:44:43,702] kopf.objects         [INFO    ] [test-service/test-service-key-store] Creation event is processed: 1 succeeded; 0 failed.

After we create the keystore, we can list the Kubernetes keystore objects using this command.

$ kubectl get ks -n test-service
NAME                     FQDN                                      
test-service-key-store   test-service.test-service.svc.cluster.local   

Step 3: Create the Truststore

The next step is to create the truststore for test-service. To create the truststore, apply the following spec in the cluster in the test-service namespace. When you create the truststore, you must specify the upstream and downstream services in the spec. In our sample application, the test-service has proxy-service as an upstream service and database-service as a downstream service.

---
apiVersion: infra.einstein.ai/v1alpha1
kind: Truststore
metadata:
  name: test-service-trust-store
  namespace: test-service
spec:
  fqdn: test-service.test-service.svc.cluster.local
  certName: test-service-new
  upstream:
    - tag: proxy-new
      fqdn: proxy-service.proxy-service.svc.cluster.local
  downstream:
    - tag: db-new
      fqdn: db-service.db-service.svc.cluster.local

Let’s create the test-service truststore object in the EKS cluster using this command.

kubectl apply -f test-service-truststore.yaml

After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local. Notary then downloads the certificates for the upstream and downstream services and creates the truststore including all downloaded certificates. The truststore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application/service during deployment using its own IAM role.

We verify the creation of the truststore from the controller logs as shown below.

[2020-10-28 00:06:32,972] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Downloaded Certificate for service test-service-new
[2020-10-28 00:06:32,975] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Downloaded CertificateChain for service test-service-new
[2020-10-28 00:06:32,976] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Downloaded PrivateKey for service test-service-new
[2020-10-28 00:06:33,577] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Added service proxy-new certificates to Trust store
[2020-10-28 00:06:34,397] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Added service db-new certificates to Trust store
[2020-10-28 00:06:35,191] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Added service root-ca certificates to Trust store
[2020-10-28 00:06:35,742] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Added service test-service-New certificates to Trust store
[2020-10-28 00:06:35,867] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Updated secret: test-service-trust-store
[2020-10-28 00:06:35,869] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Secret ARN: arn:aws:secretsmanager:us-west-2:<account-id>:secret:test-service-trust-store-mBJAIO
[2020-10-28 00:06:35,875] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Handler 'processTruststore' succeeded.
[2020-10-28 00:06:35,876] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Creation event is processed: 1 succeeded; 0 failed.

After we create the truststore, we can list the Kubernetes truststore objects using the following command.

$ kubectl get ts -n test-service
NAME                     FQDN                                      
test-service-trust-store   test-service.test-service.svc.cluster.local   

Conclusion

Notary is a Kubernetes controller that bridges the gap between AWS services and Kubernetes native resources to facilitate encryption of traffic between services running on a Kubernetes cluster using AWS-generated certificates. It provides cluster administrators an easy way to create and manage certificates using AWS Certificate Manager and store them in AWS Secrets Manager for the services to use them securely.

References

• Kubernetes — Custom Resources
• Kubernetes — Controllers

Acknowledgments

We would like to thank the entire Einstein Vision and Language Platform Infra team for valuable feedback during the design and implementation phase of Notary and the leadership team — Daniel Tisher, Ivo Mihov, and Indira Iyer — for their support and encouragement. We would also like to thank Dianne Siebold for reviewing the blog post and structuring the content. Notary was built using the Kopf project.