How to Encrypt Kubernetes Secret at Rest

Introduction

Kubernetes (K8s) is a powerful orchestration tool for containerized applications, but securing sensitive data such as secrets is crucial. By default, Kubernetes stores secrets unencrypted in etcd, which can be a security risk. This guide will walk you through encrypting Kubernetes secrets at rest.

Why Encrypt Secrets at Rest?

Encrypting secrets at rest ensures that sensitive data is not easily accessible if the storage medium is compromised. It adds an extra layer of security to protect your data against unauthorized access.

Accessing secret before Encryption

First create the secret :kubectl create secret generic my-secret --from-literal=key1=anish --from-literal=key2=bista
Check the secret Yaml :kubectl get secret my-secret -oyaml

        apiVersion: v1
        data:
          key1: YW5pc2g=
          key2: YmlzdGE=
        kind: Secret
        metadata:
          creationTimestamp: "2024-06-01T06:50:13Z"
          name: my-secret
          namespace: default
          resourceVersion: "2485"
          uid: de231c98-9eb7-484f-a3e0-0122f3578d49
        type: Opaque

Check the key1 and key2 field the value is base64 encoded and it can be easily decoded.
Decode the values of the respective key

echo "YW5pc2g=" | base64 --decode

The output of key1 will be anish

echo "YmlzdGE=" | base64 --decode

The output of key2 will be bista
This way anyone can easily know your secret and which is why it is not good approach to store your secret.

Read the Secret using ETCDCTL

ETCDCTL_API=3 etcdctl \
   --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
   --cert=/etc/kubernetes/pki/etcd/server.crt \
   --key=/etc/kubernetes/pki/etcd/server.key  \
   get /registry/secrets/default/my-secret | hexdump -C

Now , In the output you are easily able to see your secret .

That's why we need Encryption.

Step-by-Step Guide to Encrypting K8s Secrets at Rest

Step 1: Prepare Your Kubernetes Environment

Ensure you have a working Kubernetes cluster and administrative access to it. You'll need to modify the kube-apiserver configuration.

Step 2: Choose Your Encryption Provider

Kubernetes supports several encryption providers. The most commonly used is the AES-CBC provider.

Step 3: Generate an Encryption Key

Generate a base64-encoded 32-byte key for AES-256 encryption.

head -c 32 /dev/urandom | base64

The output is :

RMmA5Zi7w63DUl/iVt2eYY8hD1HDcm54YPrBd6jX4wk=

Step 4: Configure Encryption at Rest

Create the Encryption Configuration File:

vi enc.yaml :

---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: RMmA5Zi7w63DUl/iVt2eYY8hD1HDcm54YPrBd6jX4wk=
      - identity: {}

Replace the value of the secret with the value that you have generated with head -c 32 /dev/urandom | base64 in your case.

Create the Directory mkdir /etc/kubernetes/enc
Copy theenc.yamlto above directorycp enc.yaml /etc/kubernetes/enc
Update the Kube-apiserver Configuration:
Edit the kube-apiserver manifest file, typically located at vi /etc/kubernetes/manifests/kube-apiserver.yaml, to include the encryption configuration file:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.30.1.2:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml  # add this line
    volumeMounts:
    ...
    - name: enc                           # add this line
      mountPath: /etc/kubernetes/enc      # add this line
      readOnly: true                      # add this line
    ...
  volumes:
  ...
  - name: enc                             # add this line
    hostPath:                             # add this line
      path: /etc/kubernetes/enc           # add this line
      type: DirectoryOrCreate             # add this line

Step 5: Apply the Configuration

After updating the kube-apiserver configuration, the kubelet will automatically restart the kube-apiserver container to apply the changes.Once the api-server starts working now create another secret kubectl create secret generic my-secret-2 --from-literal=key1=heyhaha

Check the Encryption whether it is working or not

        ETCDCTL_API=3 etcdctl \
           --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
           --cert=/etc/kubernetes/pki/etcd/server.crt \
           --key=/etc/kubernetes/pki/etcd/server.key  \
           get /registry/secrets/default/my-secret-2 | hexdump -C

You won't see any secret in output cause it is encrypted .

But, what happened to my-secret that we first created. Does it also encrypted ? The answer is no. This encryption doesn't works for existing secret for that you have to Re-encrypt Existing secret.

Step 6: Re-encrypt Existing Secrets

Existing secrets need to be re-encrypted to apply the new encryption settings. So, Run this command to re-encrypt existing changes.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Now, check for existing secret my-secret

        ETCDCTL_API=3 etcdctl \
        >    --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
        >    --cert=/etc/kubernetes/pki/etcd/server.crt \
        >    --key=/etc/kubernetes/pki/etcd/server.key  \
        >    get /registry/secrets/default/my-secret | hexdump -C

The output you will get is encypted.

Conclusion

Encrypting Kubernetes secrets at rest is an essential security measure to protect sensitive information in your clusters. By following these steps, you can ensure that your secrets are securely stored and protected from unauthorized access. Regularly review and rotate your encryption keys as part of your security best practices.