How I Manage Secrets in Kubernetes Without Losing Sleep

Kubernetes Secret objects are base64-encoded, not encrypted. If you’re storing sensitive values directly in your cluster and committing their manifests to Git, you’re one breach away from a very bad day.

Here’s the secret management setup that lets me sleep at night.

The problem with native Kubernetes Secrets

kubectl get secret my-app-secret -o jsonpath='{.data.password}' | base64 -d
# → mysupersecretpassword

Anyone with read access to your cluster — or to your Git repo if you’re storing manifests — can trivially decode your secrets. Base64 is encoding, not encryption.

The solution: External Secrets Operator

External Secrets Operator (ESO) syncs secrets from external providers (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, etc.) into Kubernetes Secret objects. Your manifests never contain actual secret values.

Architecture

AWS Secrets Manager ──► ESO (controller) ──► Kubernetes Secret ──► Pod
        ↑
    Source of truth
    (encrypted, versioned, audited)

Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

Configure the SecretStore

# secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

Define an ExternalSecret

# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secret
  namespace: my-app
spec:
  refreshInterval: 1h          # re-sync every hour
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: my-app-secret        # K8s Secret name to create
    creationPolicy: Owner
  data:
    - secretKey: database-password
      remoteRef:
        key: prod/my-app/db
        property: password
    - secretKey: api-key
      remoteRef:
        key: prod/my-app/external-api
        property: key

This creates a proper Kubernetes Secret from values stored in AWS — and if the AWS secret rotates, ESO will sync the update automatically within the refreshInterval.

Bonus: Secret rotation without restarts

Pair ESO with Reloader to automatically restart pods when secrets change:

# Add this annotation to your Deployment
annotations:
  reloader.stakater.com/auto: "true"

Now when AWS rotates a credential and ESO syncs it, Reloader detects the Secret change and rolls your pods — zero manual intervention.

What goes in Git (safely)

✅ ExternalSecret manifests       — no sensitive values
✅ SecretStore configs            — just references
✅ Helm values (non-sensitive)    — fine
❌ Actual secret values           — never
❌ .env files with real passwords — never

This setup gives you: encryption at rest (AWS-managed), audit logs, rotation support, and GitOps-compatible manifests. That’s a setup worth sleeping over.