Kubernetes Secrets: An Overview

The Problem with Sensitive Data in Deployment Manifests

When deploying applications like MySQL, we often need to pass sensitive information such as database passwords or root credentials. Including these directly in a Kubernetes deployment file is risky:

To address this, Kubernetes provides the Secret resource. Secrets allow you to store sensitive data separately and refer to them in your deployment manifests.

Creating and Using a Kubernetes Secret

1. Creating a Secret

Secrets store data in base64-encoded format. Here’s how to create a Kubernetes Secret for a MySQL root password:

Encode the password:

echo "redhat" | base64

Create a manifest (mysql-secret.yaml):

apiVersion: v1  
kind: Secret  
metadata:  
  name: mysql-root  
  namespace: default  
type: Opaque  
data:  
  MYSQL_ROOT_PASSWORD: cmVkaGF0Cg==

2. Referring to the Secret in a Deployment

env:  
  - name: MYSQL_ROOT_PASSWORD  
    valueFrom:  
      secretKeyRef:  
        name: mysql-root  
        key: MYSQL_ROOT_PASSWORD

The Limitation of Kubernetes Secrets in GitOps

While Kubernetes Secrets hide sensitive data from deployment manifests, the Secret manifest itself (e.g., mysql-secret.yaml) contains the password in a base64-encoded format. This is not secure for GitOps workflows, as anyone with access to the Git repository can decode the password:

echo -n 'cmVkaGF0Cg==' | base64 -d  
# Output: redhat

This creates a significant risk, as GitOps principles require storing all configuration, including Secrets, in Git.

Introducing Sealed Secrets: Secure Secrets for GitOps

Sealed Secrets, an operator developed by Bitnami, solves this problem by encrypting sensitive data using an asymmetric encryption mechanism. Let’s understand how this works.

Symmetric vs. Asymmetric Encryption

In Sealed Secrets:

Public Key: Used to encrypt the sensitive data. This key is shared and safe to use.
Private Key: Used to decrypt the data. This key is securely stored in the Sealed Secrets controller within the Kubernetes cluster.

How Sealed Secrets Work

Sensitive data is encrypted using the public key.
The encrypted data is stored in a Sealed Secret manifest.
Only the Sealed Secrets controller in the Kubernetes cluster can decrypt the data using its private key.
When applied, the Sealed Secret controller generates a standard Kubernetes Secret in the cluster for the application to use.

Using Sealed Secrets in Practice

1. Installing the Sealed Secrets Controller

Add the repo of bitnami’s sealed-secrets.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

Install the helm chart for deploying sealed secret.

 helm install sealed-secrets -n kube-system sealed-secrets/sealed-secrets

2. Installing Kubeseal command:

The kubeseal CLI tool is used to encrypt sensitive data. Install it from https://github.com/bitnami-labs/sealed-secrets/releases

Extract the file and set the environment variable.

3. Create a Kubernetes Secret Locally

Prepare a Secret manifest:

apiVersion: v1
kind: Secret
metadata:
  name: mysql-root
  namespace: default
type: opaque
data:
  MYSQL_ROOT_PASSWORD: cmVkaGF0Cg==

4. Encrypt the Secret with kubeseal

Encrypt the Secret manifest to create a Sealed Secret:

kubeseal --controller-name=sealed-secrets --controller-namespace=kube-system < mysql-secret.yaml > mysql-sealed-secret.yaml

This command takes the input from the k8s secret manifest and convert this secret into sealed secret.

Now there is no need of mysql-secret manifest, delete regular mysql-secret file.
The encrypted mysql-sealed-secret.yaml is safe to commit to Git.

5. Create ArgoCD Application:

Now to finally deploy the application into our kubernetes cluster, we need to configure ArgoCD to sync with Git Repository.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: mysql
  namespace: argocd
spec:
  project: default
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: default
  source:
    repoURL: 'https://github.com/harsh2478/GitOps-Training.git'
    path: sealed-secret
    targetRevision: HEAD

Apply the changes.

Now sync the application with your git repository.

argocd app sync argocd/mysql

The Sealed Secrets controller will decrypt the data and create a standard Kubernetes Secret resource for the end-application to use as it only understand plain text.

Template keyword in sealed secret describe the standard kubernetes secrets.

You can notice the Secret is in base64-encoded format, which is safe because it never leaves the Kubernetes cluster.
The process for using the Secret in a deployment remains unchanged:

env:  
  - name: MYSQL_ROOT_PASSWORD  
    valueFrom:  
      secretKeyRef:  
        name: mysql-root  
        key: MYSQL_ROOT_PASSWORD

Why Does GitOps Need Sealed Secrets?

GitOps is built on the principle of treating Git as the single source of truth for application configuration and infrastructure. However, this creates challenges when dealing with sensitive information, like database passwords, API keys, or certificates, for several reasons:

1. Security Risks of Storing Plaintext Secrets in Git

Git repositories are inherently designed to store code and configuration in a readable format. If sensitive data like passwords or private keys are stored in plaintext or base64-encoded form:

Data Breaches: Unauthorized access to the repository exposes critical information.
Insider Threats: Even authorized users might accidentally or maliciously access or share sensitive data.

2. GitOps Principles Require Storing All Configuration in Git

In GitOps, the entire application and infrastructure state, including Secrets, must be version-controlled in Git. This creates a paradox:

Configuration must be transparent and traceable in Git.
Secrets must remain confidential and inaccessible to unauthorized users.

3. Kubernetes Secrets Are Not Enough

While Kubernetes Secrets are a dedicated resource for storing sensitive data, they are not encrypted by default:

Base64 Encoding: Kubernetes Secrets use base64 encoding, which only obfuscates the data, making it trivially easy to decode with a simple command

Sealed Secrets Bridge the Gap

Sealed Secrets address these challenges by encrypting sensitive data before it is stored in Git. Here’s why they are crucial for GitOps:

Sealed Secrets use asymmetric encryption, where only the Sealed Secrets controller in the Kubernetes cluster holds the private key.
The encrypted Sealed Secret manifest can be safely committed to Git, as it cannot be decrypted outside the cluster.
Sealed Secrets retain compatibility with GitOps principles by allowing Secrets to be versioned, reviewed, and audited like any other configuration file.
ArgoCD can seamlessly process Sealed Secrets as part of the deployment pipeline.
The decrypted Secrets are created within the Kubernetes cluster, ensuring the sensitive data never leaves the secure environment.

ArgoCD: A Deep Dive into Kubernetes Secrets and Sealed Secrets in ArgoCD