How does Kubernetes handle secrets securely and what are the limitations?
Understand how Kubernetes Secrets store sensitive data, why base64 is not encryption, and how RBAC, etcd encryption, and external managers close the gaps.
Expected Interview Answer
A Kubernetes Secret is an API object for storing sensitive data like passwords, tokens, and keys, kept separate from pod specs and mounted into pods as environment variables or files; by default it is only base64-encoded in etcd, not encrypted, so extra hardening is required.
Secrets are stored in etcd and delivered to pods via volume mounts (tmpfs) or environment variables. Security depends on enabling encryption at rest for etcd, restricting access with RBAC, and limiting who can read Secrets. Key limitations are that base64 is encoding, not encryption; anyone with etcd or namespace read access can retrieve values; environment-variable secrets can leak via logs or child processes; and Secrets are namespaced with size limits (roughly 1MiB). Many teams use external managers (Vault, cloud KMS) with the Secrets Store CSI driver for stronger control.
- Decouples sensitive data from application code and images
- Delivered as tmpfs files or env vars without hardcoding
- RBAC can restrict who reads or edits Secrets
- Supports encryption at rest for etcd via an EncryptionConfiguration
- Integrates with external stores through the Secrets Store CSI driver
AI Mentor Explanation
A Secret is like the sealed team-strategy envelope handed only to players who need it before a match. On its own the envelope is just folded paper — easily opened by anyone who grabs it — so the real security is the locked dressing room (encryption at rest) and the doorman checking passes (RBAC). Without those guards, calling it 'sealed' gives a false sense of safety.
Step-by-Step Explanation
Step 1
Create the Secret
Define a Secret (Opaque, tls, or dockerconfigjson) with kubectl create secret or a manifest; values are base64-encoded, not encrypted.
Step 2
Enable encryption at rest
Configure an EncryptionConfiguration on the API server (aescbc/kms) so Secrets are encrypted before being written to etcd.
Step 3
Restrict access with RBAC
Grant get/list on Secrets only to the specific service accounts and users that need them, scoped per namespace.
Step 4
Consume in pods
Mount the Secret as a tmpfs volume (preferred) or inject as env vars, and pull images with imagePullSecrets where needed.
Step 5
Integrate external managers
For stronger control, use the Secrets Store CSI driver with Vault or a cloud KMS and rotate credentials regularly.
What Interviewer Expects
- That base64 is encoding, not encryption
- Knowledge of encryption at rest for etcd
- Role of RBAC in limiting Secret access
- Difference between env-var and volume-mounted secrets
- Awareness of external secret managers and rotation
Common Mistakes
- Claiming Secrets are encrypted by default
- Committing Secret manifests with base64 values to Git thinking they are safe
- Using env-var secrets that can leak into logs or crash dumps
- Ignoring RBAC so any namespace user can read Secrets
- Forgetting to enable etcd encryption at rest
Best Answer (HR Friendly)
“Kubernetes Secrets keep passwords and keys out of application code and deliver them to apps safely. However, by default they are only lightly encoded, so teams must turn on encryption, restrict who can read them, and often use dedicated secret tools for real protection.”
Code Example
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
namespace: app
type: Opaque
data:
# base64-encoded values (NOT encrypted at rest by default)
username: YWRtaW4=
password: c3VwZXJzZWNyZXQ=
---
# API server EncryptionConfiguration to encrypt Secrets in etcd
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <32-byte-base64-key>
- identity: {}apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: myapp:1.0
volumeMounts:
- name: creds
mountPath: /etc/secrets
readOnly: true
volumes:
- name: creds
secret:
secretName: db-credentialsFollow-up Questions
- Why is a volume-mounted secret often safer than an env-var secret?
- How does the Secrets Store CSI driver improve on native Secrets?
- What does enabling KMS encryption for etcd change?
- How do you rotate a Secret without downtime?
- How can RBAC and audit logging reduce secret exposure?
MCQ Practice
1. By default, how are Kubernetes Secrets stored in etcd?
Without an EncryptionConfiguration, Secrets are only base64-encoded in etcd, which is encoding, not encryption.
2. Which approach reduces the risk of secrets leaking into logs?
Env vars can be printed by crash dumps or child processes; volume-mounted files avoid that exposure.
3. What primarily controls who can read a Secret?
RBAC governs get/list/watch access to Secret objects per namespace.
Flash Cards
Is base64 encryption? — No. Base64 is reversible encoding; Secrets need etcd encryption at rest for real protection.
Encryption at rest — Configured via an EncryptionConfiguration (aescbc or KMS) on the API server to encrypt Secrets in etcd.
Env var vs volume secret — Volume (tmpfs) mounts are safer; env vars can leak through logs, crash dumps, and child processes.
RBAC and Secrets — Restrict get/list/watch on Secrets to only the service accounts and users that require them.
External managers — Vault or cloud KMS via the Secrets Store CSI driver add rotation and finer-grained control.