𝛑
𝛑
Posts List
  1. Intro
  2. Some Background Knowledge
    1. Kubernetes & Secret
      1. Architecture & Workflow
      2. Access
    2. Peripheral Knowledge
    3. Vault
    4. cert-manager
    5. AWS Secret Manager
    6. Kubeseal
  3. Secret In Actions
    1. cert-manager
    2. Vault
    3. Kubeseal with Kubernetes
  4. Conclusion
  5. Resources

Talking About K8S Secret Manager

2020.02.11 Felix Zhao 中文 1.4k words ~8 min read

Intro

Over the past two weeks, I’ve been digging into how to store secrets securely. As you probably know, Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications — a core part of the CNCF ecosystem. Security design is definitely something worth putting real effort into here. Truth is, Kubernetes’ built-in security isn’t great. For instance, it only base64-encodes Secrets (that’s encoding, not encryption), and in the early days nobody really cared about securing etcd either.

So I went and researched Kubernetes secret management solutions. After going through the options, I narrowed it down to four:

  1. Vault on Kubernetes (already tested)
  2. Vault with cert-manager on Kubernetes (already tested)
  3. AWS Secret Manager on Kubernetes (not tested)
  4. Kubeseal on Kubernetes (already tested)

Kubernetes cluster on Alibaba Cloud: 3 masters, 4 workers.

Some Background Knowledge

Kubernetes & Secret

I’ll keep this brief since this post is focused on secret management solutions, not a Kubernetes deep-dive.

Architecture & Workflow

First, here’s an overview of the Kubernetes architecture.

Screenshot from 2020-02-11 14-51-48

The classic Kubernetes setup is Master/Worker mode. Each node can run different pods, but a pod can’t span multiple nodes. Every container runs a single process. On each node you’ve got three components: Kube-Proxy, Kubelet, and a Container Runtime — most of the time that’s Docker.

Kubernetes is entirely controlled via REST API.

As for Secrets — they can be tokens, DB passwords, HTTPS certs, and so on. A proper secret manager needs at least these capabilities:

  • Key Store
  • Key Rotation
  • Key Sharing
  • Seal/Unseal
  • Authentication/Authorization

When building an application on Kubernetes, configuration should be split into two parts: plaintext stuff goes into ConfigMap, sensitive stuff goes into Secret. Here’s the workflow:

Screenshot from 2020-02-11 14-56-23

The Controller watches the state of all pods. Every change flows through these parts:

  • Master Node: Deployment Controller → ReplicaSet Controller → Scheduler assigns pod to node
  • Worker Node: tells Docker to run the container

Access

1. External Users

Kubernetes supports three ways to expose services externally (ExternalName is a special case):

  • ClusterIP — the cloud (IAAS layer) provides an external or elastic IP
  • LoadBalancer — the cloud provides a load balancer from its product offerings
  • NodePort — no dedicated external IP needed; access via IP:PORT directly

You’ll also want to know about routing maps, APM, log collection, etc.

2. Ops

For internal access, use kubectl proxy and its subcommands.

  • kubectl proxy lets you view the Kubernetes dashboard. When you hit http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login, you’ll need a token — get it with: kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

  • kubectl port-forward 8080:8080 — lets you access an internal service from your local machine. Pay attention to namespace (is it default? what’s the current context?) and services. Example: kubectl port-forward vault-xxxxxx-xxxxxx 8200 to access the Vault dashboard.

Peripheral Knowledge

  • helm usage (helm3 removed Tiller)
  • oh-my-zsh usage (enabling the kubectl & helm plugins is really helpful)
  • istio — helps you control Kubernetes resources, basically an extra control plane

Vault

Vault is made by HashiCorp — a company that’s made a huge impact on DevOps with tools like Vagrant, Terraform, and Packer. Big fan. Let’s talk about Vault.

image (image from Vault docs)

In short: it handles everything. PKI certificates, SSH certificates, cross-region, cross-cloud, cross-datacenter — all of it.

cert-manager

cert-manager is a native Kubernetes certificate management controller. It mainly works through Issuers and ClusterIssuers. Here’s the architecture:

image

Different issuers provide different seal/unseal mechanisms. Except for self-signed, all others need to be configured with an external service. Supported types:

  • SelfSigned
  • CA
  • Vault
  • Venafi
  • External
  • ACME

You can use ACME mode, but it’s not required here.

AWS Secret Manager

image

Didn’t have enough resources to test this one, and compared to Vault it’s less feature-rich, so I skipped the experiment. That said, there’s real-world usage out there — GoDaddy uses it.

Kubeseal

Kubeseal is designed to encrypt your Secret into a SealedSecret, which is safe to store — even in a public repository. It has two parts: a client side and a server side. After installation, you encrypt locally with the client, and the server (which is a controller inside Kubernetes) handles decryption.

Here’s how it looks:

image

Secret In Actions

cert-manager

Step 1: Install

  • With kubectl:
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml
  • With helm:
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install --name cert-manager --namespace cert-manager --version v0.13.0 jetstack/cert-manager

Note: the official demo is outdated.

Step 2: Issuers with SelfSigned

If you want to use a different issuer, make sure it’s already installed. For example, if you’re using Vault as the issuer, you need to install vault-helm (the Vault agent server) first.

Vault

vault-k8s-auth-workflow

Step 1: Install Vault with helm

  • vault-helm (Vault agent on Kubernetes):
git clone https://github.com/hashicorp/vault-helm && cd vault-helm
helm install ./vault-helm

Check out this tutorial.

  • kubernetes-vault (Kubernetes Vault controller) — there are two approaches:

image

I followed this Quick Start to learn it:

Screenshot from 2020-02-02 14-00-46

Kubeseal with Kubernetes

Check the background section above for the workflow.

Step 1: Install

  • Client:
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.7/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
  • Server:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.7/controller.yaml

Step 2: Usage

i➜  kubeseal-guides  ᐅ  echo -n bar | kubectl create secret generic mysecret --dry-run --from-file=foo=/dev/stdin -o json >mysecret.json

i➜  kubeseal-guides  ᐅ  kubeseal < mysecret.json >mysealedsecret.json
i➜  kubeseal-guides  ᐅ

i➜  kubeseal-guides  ᐅ  kubectl apply -f mysealedsecret.json
sealedsecret.bitnami.com/mysecret created
i➜  kubeseal-guides  ᐅ  kubectl get secrets mysecret.json
Error from server (NotFound): secrets "mysecret.json" not found
i➜  kubeseal-guides  ᐅ  kubectl get secrets mysecret
Error from server (NotFound): secrets "mysecret" not found
i➜  kubeseal-guides  ᐅ  kubectl get secrets mysecret -n kube-system
NAME       TYPE     DATA   AGE
mysecret   Opaque   1      21s

Before encryption — mysecret.json:

{
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
        "name": "mysecret",
        "creationTimestamp": null
    },
    "data": {
        "foo": "YmFy"
    }
}

After encryption:

{
  "kind": "SealedSecret",
  "apiVersion": "bitnami.com/v1alpha1",
  "metadata": {
    "name": "mysecret",
    "namespace": "default",
    "creationTimestamp": null
  },
  "spec": {
    "template": {
      "metadata": {
        "name": "mysecret",
        "namespace": "default",
        "creationTimestamp": null
      }
    },
    "encryptedData": {
      "foo": "AgAao2yYWSK7bN/Ll6NlsyESPhJ3ZnPLkikGtd3+y9oJ+p5PuJaPSWAclxsdLjX5nxucdLoEWa53IktzH0PbeWyyyyyyyyyyyyyyyyyyyyyyU0AA5txJX5QjVkCNA9vxIL7XeqLVyi/eno7oEEdA2BXySAK5a6Q3k3oTJ0uTiPJZOYFvsFeWpz2D4qNuKH9h0LqF3vqJVSmZF4QWdYEA1GndEJRAVzxP8V8HT0unss81w3yPt/bAmeunN4AyyyyyyyyyyyyyyyyyyyyyyyyyWadQ5h0LogC+vbBLKxuJzTXFzVRAzYbg6hbGJTZWQu0isSmLJZrwVKiyF54UIPWh4EnTbim/PLrU08CnuLhgGToeA24uwm/5dmmDnC2BvvQyeFi77fj4uLnJMx5LYw5wPYft0nCkowRJmhuu2cqUviUQ8FArAHc6xQOLKIjt5tojc2BNiIY7aKLzz9VSWVvcID7XfWRkdonYQbfBbGShZKdKCxxxxxxxxxxxxxxxxxxxxxxxxxx="
    }
  },
  "status": {
    
  }
}

One thing to watch out for: if you can’t fetch the certificate, you may need to expose the service with kubectl expose service -n kube-system sealed-secrets-controller --type=ClusterIP.

Conclusion

Whichever solution you pick, you need to wire it into your deployment or patch it in. Whether it’s Cloud Security or Cloud Native Security, security-by-default and zero-trust principles are non-negotiable. Due to some constraints, I couldn’t include screenshots of every experiment — but I’d encourage you to run through them yourself.

Leaving off with a Chinese poem I love, and its translation:

有人住高楼，有人在深沟，有人光万丈，有人一身锈，世人万千种，浮云莫去求，斯人若彩虹，遇上方知有。——《怦然心动》

Some of us get dipped in flat, some in satin, some in gloss. But every once in a while you find someone who’s iridescent, and when you do, nothing will ever compare.

Resources