Rotate TLS Certificates

How-to rotate TLS certificates.

Certificates can go out of date, or the private keys of the server certificate and the signing CA can become compromised. The Operator supports Kubernetes certificate rotation in order to enable the replacement of these expired certificates or compromised keys.

The following are some examples of certificate rotation:

Replacement of the certificate chain and key pair (server secret)

Use when:
- Certificates expire
- Server or intermediate CA keys have been compromised
Replacement of the whole PKI (both Operator and server secrets)

Use when:
- The root CA has been compromised
- The root CA has expired

The relevant errors will be shown when a certificate is found to be invalid or compromised.

Expired certificate

certificate cannot be verified: x509: certificate has expired or is not yet valid

Compromised certificate

certificate cannot be verified: x509: certificate signed by unknown authority

Certificate rotation is fully supported when using client certificate authentication. All certificate rotation occurs over TLS, however due to technical reasons, the rotation of expired certificates must occur over plain text. However when rotating either expired or non-expired certificates, the private keys are never exposed by the Couchbase Operator over the wire since private keys are securely mounted on Couchbase Server pods by Kubernetes.

Replacing Server Certificates

In the event of server certificate expiry or compromise, a new certificate and key pair generated by the same CA can be used to create a new server TLS secret to be loaded onto Couchbase Server pods, replacing the existing secret. If a certificate chain is being used, it’s possible to just generate a new leaf certificate from the intermediate CA.

First, get the existing secret and direct it into a new YAML file (resource.yaml in this example):

$ kubectl get secret couchbase-server-tls -o yaml > resource.yaml

The contents of the new file will look similar to the following:

apiVersion: v1
kind: Secret
metadata:
  name: couchbase-server-tls
type: Opaque
data:
  tls.crt: Q2VydGlmaWXRhO...
  tls.key: LS0tLS1CRUdJTi...

Next, replace the relevant data fields in the YAML file. To do this, start by base64-encoding the PKI files in question.

Certificate chain:

$ base64 -i tls.crt

Private key:

$ base64 -i tls.key

Then replace the relevant fields under data with the new base64 values:

apiVersion: v1
kind: Secret
metadata:
  name: couchbase-server-tls
type: Opaque
data:
  tls.crt: NhdGU6CiAgICBEY...
  tls.key: LSUlFcFFJQkFBS0N...

Finish the certificate rotation by pushing the new secret to Kubernetes:

$ kubectl replace -f resource.yaml

You can then check that the new secret has been applied:

$ kubectl get secret couchbase-server-tls -o yaml

apiVersion: v1
data:
  tls.crt: NhdGU6CiAgICBEY...
  tls.key: LSUlFcFFJQkFBS0N...
kind: Secret
metadata:
  creationTimestamp: "2019-01-09T09:42:35Z"
  name: couchbase-server-tls
  namespace: default
  resourceVersion: "116380"
  selfLink: /api/v1/namespaces/default/secrets/couchbase-server-tls
  uid: e5111d54-13f2-11e9-baeb-0ac70bf4cf44
type: Opaque

Each pod will have a copy of the new certificate and key in a mounted inbox on each node.

What it Looks Like

Once triggered, the server TLS rotation operation should be monitored for successful completion:

$ kubectl describe couchbasecluster
...
Events:
  Type    Reason              Age   From  Message
  ----    ------              ----  ----  -------
  Normal  TLSUpdated          8s          TLS certificate was updated for member test-couchbase-v92lg-0000 (1)
  Normal  TLSUpdated          7s          TLS certificate was updated for member test-couchbase-v92lg-0001
  Normal  TLSUpdated          5s          TLS certificate was updated for member test-couchbase-v92lg-0002

1	A successful TLS rotation will update each Couchbase Server pod in the cluster and raise the `TLSUpdated` event.

TLS keys and certificates are securely mounted in the Pod as a Kubernetes Secret, so they are never exposed over the network. When the Secret is updated, it will take a short period of time for kubelet to be notified of the update, and update to be made to the file system in the Pod. As such, there will be a delay — in the order of about a minute — between triggering the rotation and the Operator reporting progress.

Replacing the entire PKI

If the root CA is compromised, a full rotation and replacement of the PKI will be required. When you replace the PKI, you’ll also need to replace the CA certificate in the server secret.

What it Looks Like

Once triggered, the PKI rotation operation should be monitored for successful completion:

$ kubectl describe couchbasecluster
...
Events:
  Type    Reason              Age   From  Message
  ----    ------              ----  ----  -------
  Normal  TLSUpdated          12s         TLS certificate was updated for member test-couchbase-1i25n-0000 (1)
  Normal  TLSUpdated          7s          TLS certificate was updated for member test-couchbase-1i25n-0001
  Normal  TLSUpdated          5s          TLS certificate was updated for member test-couchbase-1i25n-0002
  Normal  ClientTLSUpdated    4s          CA certificate was updated (2)

1	A successful TLS rotation will update each Couchbase Server pod in the cluster and raise the `TLSUpdated` event.
2	Once all the Server instances are updated, the Operator needs to start using the new CA certificate in order to trust the new server certificates. This is indicated by the `ClientTLSUpdated` event.

A full PKI rotation must not be interrupted once it has begun. Interruption risks severing communication between the Operator and the Couchbase cluster, as some certificates cannot be validated against the active CA. In the unlikely event something does go wrong, don’t restart anything, it may make the problem worse, and seek immediate guidance from Couchbase support.

Replacing expired Certificates

In the event of server certificate or root CA expiry, TLS rotation and replacement can be performed according to the previous sections in order to restore secure connectivity to Couchbase Server Pods. When the Couchbase Operator has detected the expiration of Couchbase Server certificates or any of the root CAs, the Couchbase cluster resource will raise the ClientTLSInvalid event:

$ kubectl describe couchbasecluster
...
Events:
  Type    Reason              Age   From                Message
  ----    ------              ----  ----                -------
  Normal  ClientTLSInvalid    9s (x48 over 8m2s)        Failed to validate TLS certificate chain
  Normal  ClientTLSUpdated    81s                       Client certificate was updated

The ClientTLSInvalid event is raised until valid certificates are provided to replace the expired certificates.

In order to replace expired server or root CA certificates you need to ensure that allowPlainTextCertReload is enabled. This value can be set within the Couchbase Cluster resource:

apiVersion: v2
kind: CouchbaseCluster
metadata:
  name: ""
spec:
  networking:
    tls:
      allowPlainTextCertReload: true

Enabling allowPlainTextCertReload will cause user credentials to be momentarily exposed while the Operator makes plain text http requests to Couchbase Server. Plain text requests are required because the Operator needs to request Couchbase Server to reload its certificates without rejecting invalid TLS client requests. Due to this behavior it is more effective to proactively monitor and rotate certificates prior to rotation in order to ensure security of your cluster.

The following calls are made in plain text to the Couchbase when recovering a cluster with only expired server certs:

Reload node certificate

The following calls are made in plain text to the Couchbase when recovering a cluster with expired root CA certificates:

When only the client certificates have expired then the Operator can securely rotate the certificates without the need to perform plain text requests to Couchbase Server.

Expired certificates cannot be rotated if Node encryption is enabled due to the security risk caused by the amount of insecure traffic that will be generated when disabling this feature.