• May 26, 2024
• in homelab, secrets, kubernetes
• 3 min read

Keep your Secrets Secure

This will demonstrate how I changed from creating kubernetes secrets directly, to using sealed secrets to create a sealed secret and let the controller create the kuernetes secret for us.

Current Secrets Management Workflow

One thing I don't like from my workflow is to create and update secrets inside Kubernetes. It will be in something like this:

• Base64 encode a value
• Create the secret in a yaml file
• Deploy the secret then remove the yaml file

But when I want to rotate or update the secret:

• View the kubernetes secret, see if it needs changing
• Base64 encode the new value
• Store it into a new yaml file
• Deploy the new updated secret

I find this workflow tedious, so I switched to sealed-secrets

What is Sealed Secrets

Sealed Secrets enables you to encrypt your secrets using the kubeseal utility which uses asymmetric crypto to encrypt secrets that only the controller can decrypt, which makes it possible to store the sealed value of your secret inside a git public repository as only the controller can decrypt the encrypted string.

So in practice we would generate the secret and pass it to kubeseal and then dump the sealedsecret to stdout which we can then store as a sealedsecret resource, which we can safely store in a public git repository as example.

Install Dependencies

We will need to install kubeseal, and for MacOS using homebrew its:

brew install kubeseal

For other operating systems, please see:

• https://github.com/bitnami-labs/sealed-secrets?tab=readme-ov-file#kubeseal

Deploy the Sealed Secrets Controller

For ArgoCD inside my apps/kube-system/sealed-secrets/Chart.yaml I have:

---
apiVersion: v2
name: sealed-secrets
description: Sealed Secrets Helm Chart
type: application
version: 0.26.2
dependencies:
- name: sealed-secrets
  version: 2.15.3
  repository: https://bitnami-labs.github.io/sealed-secrets/

And inside my apps/kube-system/sealed-secrets/values.yaml I have:

sealed-secrets:
  fullnameOverride: sealed-secrets-controller
  createController: true
  secretName: "sealed-secrets-key"

  metrics:
    serviceMonitor:
      enabled: true
      namespace: "monitoring"
      labels:
        release: kube-prometheus-stack
    dashboards:
      create: true
      labels:
        grafana_dashboard: "1"
      annotations:
        grafana_folder: "Sektorlab"
      namespace: "monitoring"

Inside the apps/kube-system/sealed-secrets directory I executed:

helm dependency update

And then I pushed the changes up to main and then I saw the sealed-secrets controller pod started in the kube-system namespace.

If you are using Helm to deploy, you can remove the top key of sealed-secrets: from the values and deploy.

How to Create a Sealed Secret

We first need to create a kubernetes secret, then pass it to kubeseal and then save the output to yaml, for example, creating a secret with admin-user: admin:

kubectl create secret generic db-secrets \
  --from-literal=admin-user=admin
  --namespace default --dry-run=client \
  --output yaml | \
  kubeseal --format yaml \
  --namespace default \
  --scope=namespace-wide > sealedsecret.yaml

When we look at sealedsecret.yaml we will see something like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  name: db-secrets
  namespace: default
spec:
  encryptedData:
    admin-user: AgBY3WYt+Vqyz6jl57gwfPYUUK505waNU2MMCQSLJbwklOh4CXEW8ZSp9Ze1b5QwFgionV4Ch7Al9gKdGvjupYe+n/Og5Il/Nd7FzAzvg69g8xFKMDwy7YCSAJypgUevZ7Ff1WKcyV0T6P0TO1+aUquLuMb1aqFSQHvfWc7xhhbzO+U+/f7t+bFiuHDXUMpR5qzOGCejfFoF26a8BSguY20P1BjqOaW402Y/4sVnK8Zm+rDweq0Ddx19tB09c21hBoau2cGOSz7auRK2Rw+QT9AZW2QlJZD36AHK+mW5gJsA8It1AGbsZAyzAcnWA/PCmOI9KypnWxXHNZrDutb18pwWtsIrMWDbWbg2jUV2Ag7ZjSTcRqOKHQaqevBeJAk3i2RzSdJAALNKODjfCnaho9ijUgUwovgqD+djVDVoF0Hh8YfNzxR94JtVKDA8kns/SQklucoIXwek0lO5O1Yy3sEtvO9NdIn1aTfZqj2qnxdRnldIr+sSMTyF6oa8xQGQhoS0Q1vMWH1Kg/vAcoHdGwTwB3uO3A3w5a63OX8FYRjhEE5D/X4b3UBm0LE32nvBGVKTSODZFF/GPoC04tCr9rGWRxl3hVZOuM+SshCGCc7F/Lr+W+lEkBkcWt7y4YvggPymog/tBx7KFi0at+6W85+jJ9h0/YHVaexa0gSRxKQaZhZnQ8BxEctKVN7NvvR8zj3Gvz/fcw==
  template:
    metadata:
      annotations:
        sealedsecrets.bitnami.com/namespace-wide: "true"
      name: db-secrets
      namespace: default

We can then apply the yaml with:

kubectl apply -f sealedsecret.yaml

We can then view the events from the namespace to see if the secret was successfully unsealed:

kubectl get events -n default
LAST SEEN   TYPE     REASON     OBJECT                    MESSAGE
21s         Normal   Unsealed   sealedsecret/db-secrets   SealedSecret unsealed successfully

And then we can inspect the kubernetes secret which was created from the sealedsecret:

kubectl get secrets/db-secrets -o yaml
apiVersion: v1
kind: Secret
type: Opaque
data:
  admin-user: YWRtaW4=
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  name: db-secrets
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: db-secrets
    uid: a2495f22-b40d-4afb-ae6f-b036474b2b6d

New Workflow

So the new workflow enables me to do the following:

• Run the kubectl create secret and output the yaml to git
• Refence the secret it will deploy in my workload
• Once ArgoCD deploys the sealedsecret, a secret is generated and the workload can pickup the secret

Resources

• https://github.com/bitnami-labs/sealed-secrets