Skip to main content

Notary: A Certificate Lifecycle Management Controller for Kubernetes

Vaishnavi Galgali
Vaishnavi Galgali
Mar 04 - 12 min read

Introduction

All services in the Einstein Vision and Language Platform use TLS/SSL certificates to encrypt communication between microservices. The certificates are generated in AWS Certificate Manager (ACM) and stored in the AWS Secrets Manager in the form of keystores and truststores (private and public keys). Certificate creation can be a manual process, especially if the permission model dictates that a certificate can only be provisioned or de-provisioned by an AWS account admin.

Certificate management includes provisioning certificates and monitoring their expiration and renewal. AWS ACM provides the ability to auto-renew only public certs attached to AWS native resources (such as ELB and RDS). Einstein Vision and Language Platform services run on an EKS cluster with service exposure scoped to the cluster network, which means that the ACM certificates aren’t attached to AWS native resources. Hence, the certificate auto-renewal feature provided by AWS ACM can’t be used.

To automate certificate provisioning and management and to provide a self-service model for carrying out these activities, we built a custom Kubernetes controller by introducing Kubernetes native resources: certificatekeystore, and truststore. Additionally, we built automation for checking certificate expiration and alert on expiration of these certificates. We call our solution Notary.

Architecture

Notary is built on the core Kubernetes concepts:

  • Controllers
  • Custom Resource Definition (CRD)

What is a Kubernetes controller?

In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state. A controller tracks at least one Kubernetes resource type. These objects have a spec field that represents the desired state. The controller for that resource is responsible for performing actions to achieve the desired state. For example, the job controller, which is a built-in Kubernetes controller, helps to execute tasks in Kubernetes cluster by deploying pods as dictated by the pod spec.

The Kubernetes API can be extended to create application-specific controllers to create, configure, and manage instances of complex, stateful applications on behalf of a Kubernetes user. The custom Kubernetes controller builds upon the Kubernetes resource and controller concepts, and includes domain or application-specific knowledge to automate common tasks.

What is a CRD?

CRD stands for Custom Resource Definition.

A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.

A custom resource is an extension of the Kubernetes API that provides functionality that isn’t available in a default Kubernetes installation. A custom resource represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.

A custom resource can appear and disappear in a running cluster through dynamic registration. And cluster admins can update a custom resource independently of the cluster itself. After a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like pods.

Notary

Notary is an extension to the Kubernetes API to satisfy an application-specific use case — encryption of traffic between services running on Kubernetes cluster — by managing the certificates and their lifecycle.

Notary is a custom Kubernetes controller that watches on these custom resources: certificatekeystore, and truststore. The controller watches for create and update events, and takes defined action based on those events.

Notary interacts with AWS Certificate Manager to request and download certificates to create keystores and truststores. After it creates keystores and truststores, Notary initiates a session with AWS Secrets Manager to upload them.

Notary Architecture

The architecture diagram above depicts the functionality of Notary. The Notary controller constantly watches the custom resources — certificatekeystore, and truststore—for a create, update, or delete event.

  • Certificate: When the create event for the certificate resource occurs, the Notary controller requests a new certificate from AWS Certificate Manager.
  • Keystore: When the create event for the keystore resource occurs, the Notary controller downloads the service’s certificate bundle from AWS Certificate Manager, creates a keystore, and then uploads the keystore to AWS Secrets Manager.
  • Truststore: When the create event for the truststore resource occurs, the Notary controller downloads the service’s certificate bundle along with upstream and downstream services’ certificates from AWS Certificate Manager. It then builds a truststore which includes the service certificate chain along with upstream and downstream service certificates, and uploads the truststore to AWS Secrets Manager.

Components

Notary is built using Kubernetes objects. Let’s take a look at the components of Notary.

Deployment

Since we’re dealing with sensitive data — certificatekeystore, and truststore—for services, it’s important to secure the Notary pods. To address that, Notary is deployed in its own namespace where the namespace resources can only be accessed by cluster administrators.

RBAC

RBAC, role-based access control, is a way to manage access to resources based on roles. Notary needs access to the CRD objects certificatekeystore, and truststore which can be deployed in any service-specific namespace. Since Notary needs to watch all namespaces it needs ClusterRole privileges.

These examples use ClusterRole and ClusterRoleBinding to give Notary access to specific resources and assign a certain level of access to resources.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
operator: notary
name: notary
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- apiGroups:
- infra.einstein.ai
resources:
- certificates
- keystores
- truststores
verbs:
- get
- list
- watch
- apiGroups:
- events.k8s.io
- ""
resources:
- events
verbs:
- createapiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: notary
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: notary
subjects:
- kind: ServiceAccount
name: notary
namespace: notary

IAM Roles

Notary needs access to AWS Certificate Manager to request new certificates and access to AWS Secrets Manager to upload keystores and truststores. We use AWS IAM (Identity and Access Management) roles to grant access to AWS services required for Notary. The following JSON is an example of an IAM role.

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"acm:RequestCertificate",
"acm:GetCertificate",
"acm:ListCertificates",
"acm:ExportCertificate",
"acm:UpdateCertificateOptions",
"acm:AddTagsToCertificate",
"acm:ListTagsForCertificate",
"acm:DescribeCertificate",
"acm:ResendValidationEmail",
"acm:RemoveTagsFromCertificate",
"acm-pca:GetCertificate",
"acm-pca:IssueCertificate",
"acm-pca:ListCertificateAuthorities"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"secretsmanager:UntagResource",
"secretsmanager:DescribeSecret",
"secretsmanager:PutSecretValue",
"secretsmanager:GetSecretValue",
"secretsmanager:CreateSecret",
"secretsmanager:DeleteSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:GetRandomPassword",
"secretsmanager:GetSecretValue",
"secretsmanager:RestoreSecret",
"secretsmanager:RotateSecret",
"secretsmanager:CancelRotateSecret",
"secretsmanager:UpdateSecret",
"secretsmanager:GetResourcePolicy",
"secretsmanager:UpdateSecretVersionStage",
"secretsmanager:ListSecrets",
"secretsmanager:TagResource"
],
"Resource": "*",
"Effect": "Allow"
}
]
}

CRD

Certificate CRD

This custom resource defines the metadata required to request a certificate from AWS Certificate Manager. The metadata includes the certificate FQDN and type (public or private) of certificate.

# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: certificates.infra.einstein.ai
labels:
operator: {{ .Values.app.name }}
spec:
scope: Namespaced
group: infra.einstein.ai
version: v1alpha1
names:
kind: Certificate
singular: certificate
plural: certificates
shortNames:
- cert
additionalPrinterColumns:
- name: Type
type: string
JSONPath: .spec.type
description: Type of certificate
- name: FQDN
type: string
priority: 0
JSONPath: .spec.fqdn
description: FQDN of the certificate
- name: AltName
type: string
priority: 0
JSONPath: .spec.alt
description: Alternate names of the certificate
- name: ARN
type: string
priority: 0
JSONPath: .status.arn
description: ARN of the certificate
- name: Tags
type: string
priority: 0
JSONPath: .status.tags
description: Tags on the certificate
validation:
openAPIV3Schema:
properties:
spec:
properties:
type:
type: string
description: "AWS Certificate Manager PEM certificate issued by either a Private of Public Root CA."
enum: ["Public", "Private"]
fqdn:
type: string
description: "FQDN of the certificate"
alt:
type: array
description: "Alternate name(s) for the certifcate"
required: ["type", "fqdn"]

When the certificate CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager for a new certificate.

Required parameters

  • type: Type of certificate, either public or private
  • fqdn: Fully qualified domain name of the service

Keystore CRD

This custom resource defines the keystore resource. The resource spec accepts metadata to generate a new keystore for a particular service and then uploads the keystore to AWS Secrets Manager.

# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: keystores.infra.einstein.ai
labels:
operator: {{ .Values.app.name }}
spec:
scope: Namespaced
group: infra.einstein.ai
version: v1alpha1
names:
kind: Keystore
singular: keystore
plural: keystores
shortNames:
- ks
additionalPrinterColumns:
- name: FQDN
type: string
priority: 0
JSONPath: .spec.fqdn
description: FQDN of the certificate
- name: Certificate Name
type: string
priority: 0
JSONPath: .spec.certName
description: Name of the certificate that need to be included in the keystore
- name: ARN
type: string
priority: 0
JSONPath: .status.arn
description: ARN of the keystore created by Notary which is uploaded as secret in AWS secrets manager
- name: Tags
type: string
priority: 0
JSONPath: .status.tags
description: Tags on the certificate
validation:
openAPIV3Schema:
properties:
spec:
properties:
certName:
type: string
description: "Name of the certificate that will be included in the keystore"
fqdn:
type: string
description: "FQDN of the service of which certificate is created"
required: ["fqdn", "certName"]

When the keystore CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager to download the service certificate, builds the keystore, and uploads the keystore to AWS Secrets Manager.

Required parameters

  • certName: Name tag of the newly requested certificate
  • fqdn: Fully qualified domain name of the service

Truststore CRD

This custom resource defines the truststore resource. The resource spec accepts service metadata along with the service FQDN and certificate names of the upstream and downstream services.

# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: truststores.infra.einstein.ai
labels:
operator: {{ .Values.app.name }}
spec:
scope: Namespaced
group: infra.einstein.ai
version: v1alpha1
names:
kind: Truststore
singular: truststore
plural: truststores
shortNames:
- ts
additionalPrinterColumns:
- name: FQDN
type: string
priority: 0
JSONPath: .spec.fqdn
description: FQDN of the certificate
- name: ARN
type: string
priority: 0
JSONPath: .status.arn
description: ARN of the truststore created by Notary which is uploaded as secret in AWS secrets manager
- name: Tags
type: string
priority: 0
JSONPath: .status.tags
description: Tags on the certificate
validation:
openAPIV3Schema:
properties:
spec:
properties:
upstream:
type: array
description: "Tags of the upstream service(s) certificate that talk to this service"
downstream:
type: array
description: "Tags of the downstream service(s) certificate that this service talks to"
fqdn:
type: string
description: "FQDN of the service of which certificate is created"
certName:
type: string
description: "Name of the new certificate that need to be included in the truststore"
required: ["fqdn", "upstream", "downstream", "certName"]

Required parameters

  • upstream: Tags of the upstream service(s) certificate that talks to this service
  • downstream: Tags of the downstream service(s) certificate that this service talks to
  • fqdn: Fully qualified domain name of this service
  • certName: Name tag of the newly-requested certificate

Notary In Action

Now let’s take a look at Notary in action. Consider a sample application, with a service named test-service. The test-service service talks to a proxy service and a database service. To have encrypted service-to-service communication, we must have certificates for each service and the resulting keystore and truststore.

Pre-requisites

As mentioned earlier, in addition to creating certificates, Notary also creates keystores and truststores. Both keystores and truststores are sensitive data, so they need to be protected. Notary password protects these artifacts at the time of creation so in case of unintended/unauthorized access the keystore and truststore can’t be read without their corresponding password.

This implies there has to be a way to share the keystore and truststore credentials with the service that uses them to encrypt/decrypt data. Currently, we handle this by creating a metadata secret for every service. This is a one-time admin-triggered automation (more on that in future blog post), part of service onboarding on our infrastructure.

This metadata secret is an encoded JSON secret that follows the format shown here.

{
"tlsStoreRootPath": "tls",
"tlsKeyStoreName": "test-key-store",
"tlsTrustStoreName": "test-trust-store",
"tlsKeyStorePassword": "test-key-store-password",
"tlsTrustStorePassword": "test-trust-store-password"
}

After the service is onboarded and the metadata secret is created, Notary is ready to manage certificateskeystores, and truststores for that service.

Step 1: Create the Certificate

To create certificates for the test service, proxy service, and database service, we must create certificate objects for all three services. Following is a Certificate custom resource YAML spec for test-service. Similarly, certificates can be created for the proxy service and database service.

apiVersion: infra.einstein.ai/v1alpha1
kind: Certificate
metadata:
generation: 1
labels:
env: DEV
name: test-service
namespace: test-service
spec:
alt:
- test-service.localhost
fqdn: test-service.test-service.svc.cluster.local
type: Private

The above spec can be saved as test-service.yaml file. Let’s use this file to create the test-service certificate object in the EKS cluster using the following command.

kubectl apply -f test-service.yaml

After you apply the object in the cluster, Notary requests AWS Certificate Manager to create a private certificate with the name test-service and an FQDN test-service.test-service.svc.cluster.local.

AWS Certificate Manager console displaying new certificate

We can also verify in the controller logs that the private certificate was created.

[2021-03-03 21:40:45,709] kopf.objects         [INFO    ] [test-service/test-service] Updated new certificate passphrase for service test-service
[2021-03-03 21:40:45,713] kopf.objects [INFO ] [test-service/test-service] Handler 'processCertificate' succeeded.
[2021-03-03 21:40:45,713] kopf.objects [INFO ] [test-service/test-service] Creation event is processed: 1 succeeded; 0 failed.

After we create certificates for all three services, we can also list the Kubernetes Certificate objects using this command.

$ kubectl get cert --all-namespaces
NAME TYPE FQDN
db-new Private db-service.db-service.svc.cluster.local
proxy-new Private proxy-service.proxy-service.svc.cluster.local
test-service-new Private test-service.test-service.svc.cluster.local

Step 2: Create the Keystore

The next step is to create the keystore for test-service. To create the keystore, apply the following spec in the cluster for test-service in test-service namespace.

---
apiVersion: infra.einstein.ai/v1alpha1
kind: Keystore
metadata:
name: test-service-key-store
namespace: test-service
spec:
fqdn: test-service.test-service.svc.cluster.local
certName: test-service-new

Now let’s create the test-service keystore object in the EKS cluster using this command.

kubectl apply -f test-service-keystore.yaml

After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local and creates the keystore for the private key. The keystore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application/service during deployment using its own IAM role.

AWS Secret Manager Console displaying newly created secret

We can verify the creation of the keystore from the controller logs as shown here.

[2020-10-27 23:44:43,596] kopf.objects         [INFO    ] [test-service/test-service-key-store] Downloaded Certificate for service test-service-new
[2020-10-27 23:44:43,598] kopf.objects [INFO ] [test-service/test-service-key-store] Downloaded CertificateChain for service test-service-new
[2020-10-27 23:44:43,599] kopf.objects [INFO ] [test-service/test-service-key-store] Downloaded PrivateKey for service test-service-new
[2020-10-27 23:44:43,612] kopf.objects [INFO ] [test-service/test-service-key-store] Created keystore successfully for test-service
[2020-10-27 23:44:43,694] kopf.objects [INFO ] [test-service/test-service-key-store] Updated secret: test-service-key-store
[2020-10-27 23:44:43,696] kopf.objects [INFO ] [test-service/test-service-key-store] Secret ARN: arn:aws:secretsmanager:us-west-2:<account-id>:secret:test-service-key-store-ZimHEX
[2020-10-27 23:44:43,701] kopf.objects [INFO ] [test-service/test-service-key-store] Handler 'processKeystore' succeeded.
[2020-10-27 23:44:43,702] kopf.objects [INFO ] [test-service/test-service-key-store] Creation event is processed: 1 succeeded; 0 failed.

After we create the keystore, we can list the Kubernetes keystore objects using this command.

$ kubectl get ks -n test-service
NAME FQDN
test-service-key-store test-service.test-service.svc.cluster.local

Step 3: Create the Truststore

The next step is to create the truststore for test-service. To create the truststore, apply the following spec in the cluster in the test-service namespace. When you create the truststore, you must specify the upstream and downstream services in the spec. In our sample application, the test-service has proxy-service as an upstream service and database-service as a downstream service.

---
apiVersion: infra.einstein.ai/v1alpha1
kind: Truststore
metadata:
name: test-service-trust-store
namespace: test-service
spec:
fqdn: test-service.test-service.svc.cluster.local
certName: test-service-new
upstream:
- tag: proxy-new
fqdn: proxy-service.proxy-service.svc.cluster.local
downstream:
- tag: db-new
fqdn: db-service.db-service.svc.cluster.local

Let’s create the test-service truststore object in the EKS cluster using this command.

kubectl apply -f test-service-truststore.yaml

After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local. Notary then downloads the certificates for the upstream and downstream services and creates the truststore including all downloaded certificates. The truststore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application/service during deployment using its own IAM role.

AWS Secrets Manager displaying newly created secret

We verify the creation of the truststore from the controller logs as shown below.

[2020-10-28 00:06:32,972] kopf.objects         [INFO    ] [test-service/test-service-trust-store] Downloaded Certificate for service test-service-new
[2020-10-28 00:06:32,975] kopf.objects [INFO ] [test-service/test-service-trust-store] Downloaded CertificateChain for service test-service-new
[2020-10-28 00:06:32,976] kopf.objects [INFO ] [test-service/test-service-trust-store] Downloaded PrivateKey for service test-service-new
[2020-10-28 00:06:33,577] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service proxy-new certificates to Trust store
[2020-10-28 00:06:34,397] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service db-new certificates to Trust store
[2020-10-28 00:06:35,191] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service root-ca certificates to Trust store
[2020-10-28 00:06:35,742] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service test-service-New certificates to Trust store
[2020-10-28 00:06:35,867] kopf.objects [INFO ] [test-service/test-service-trust-store] Updated secret: test-service-trust-store
[2020-10-28 00:06:35,869] kopf.objects [INFO ] [test-service/test-service-trust-store] Secret ARN: arn:aws:secretsmanager:us-west-2:<account-id>:secret:test-service-trust-store-mBJAIO
[2020-10-28 00:06:35,875] kopf.objects [INFO ] [test-service/test-service-trust-store] Handler 'processTruststore' succeeded.
[2020-10-28 00:06:35,876] kopf.objects [INFO ] [test-service/test-service-trust-store] Creation event is processed: 1 succeeded; 0 failed.

After we create the truststore, we can list the Kubernetes truststore objects using the following command.

$ kubectl get ts -n test-service
NAME FQDN
test-service-trust-store test-service.test-service.svc.cluster.local

Conclusion

Notary is a Kubernetes controller that bridges the gap between AWS services and Kubernetes native resources to facilitate encryption of traffic between services running on a Kubernetes cluster using AWS-generated certificates. It provides cluster administrators an easy way to create and manage certificates using AWS Certificate Manager and store them in AWS Secrets Manager for the services to use them securely.

References

Acknowledgments

We would like to thank the entire Einstein Vision and Language Platform Infra team for valuable feedback during the design and implementation phase of Notary and the leadership team — Daniel Tisher, Ivo Mihov, and Indira Iyer — for their support and encouragement. We would also like to thank Dianne Siebold for reviewing the blog post and structuring the content. Notary was built using the Kopf project.

Related Architecture Articles

View all