Containers are designed to be ephemeral. But your data isn’t. Here’s how Kubernetes bridges that gap — and how to run stateful workloads without losing sleep.
Kubernetes Zero to Hero · Episode 06 ·
In Episode 05, we externalised configuration and secrets from our container images. Our applications are now cleanly separated from their environment. But there’s one more gap to close before we can run real production workloads.
Data.
Everything we’ve built so far is stateless — web servers, APIs, proxies. They compute something and return a response. Nothing written to disk needs to survive a restart. But the moment you introduce a database, a message queue, a file upload service, or any workload that needs to remember something across pod restarts — you have a stateful problem.
And Kubernetes, designed around ephemeral containers, needs to handle it carefully.
This episode covers the complete Kubernetes storage model: how volumes work, how to request persistent storage, how the storage lifecycle is managed, and how StatefulSets handle workloads that can’t be treated as interchangeable cattle.
Why Container Storage Is Complicated
When a container starts, it gets a fresh filesystem from its image. Anything written during the container’s lifetime lives in a writable layer on top — and when the container is deleted, that layer is gone with it.
# Prove it to yourself
$ kubectl run ephemeral-test --image=nginx:1.25
$ kubectl exec -it ephemeral-test -- bash -c "echo 'important data' > /tmp/myfile.txt"
$ kubectl exec -it ephemeral-test -- cat /tmp/myfile.txt
important data
# Delete and recreate the pod
$ kubectl delete pod ephemeral-test
$ kubectl run ephemeral-test --image=nginx:1.25
$ kubectl exec -it ephemeral-test -- cat /tmp/myfile.txt
cat: /tmp/myfile.txt: No such file or directory
Gone. This is by design — ephemeral containers are predictable, reproducible, and easy to reason about. But it means storage requires explicit, deliberate configuration.
The Kubernetes Storage Model: Three Layers
Kubernetes storage is built on three abstractions that separate the what from the how and the who:
Developer asks for storage
↓
PersistentVolumeClaim (PVC) ← "I need 10Gi of fast storage"
↓
PersistentVolume (PV) ← actual storage resource provisioned
↓
StorageClass ← defines HOW storage is provisioned
↓
Physical Storage (cloud disk, NFS, local SSD...)
StorageClass — defines the type of storage available and the provisioner that creates it. Cloud providers ship default StorageClasses (AWS EBS, GCP Persistent Disk, Azure Disk). You can define custom ones.
PersistentVolume (PV) — a piece of storage in the cluster. It can be provisioned manually by an admin or dynamically by a StorageClass. It exists independently of any pod.
PersistentVolumeClaim (PVC) — a request for storage by a user or pod. “I need 10Gi of ReadWriteOnce storage.” Kubernetes finds a matching PV and binds them together.
The beauty of this model: developers write PVCs (what they need), cluster admins configure StorageClasses (how it’s provisioned), and neither needs to know the details of the other’s work.
StorageClasses: Defining Your Storage Tiers
# See available StorageClasses in your cluster
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE AGE
standard (default) docker.io/hostpath Delete Immediate 5d
# custom-storageclass.yaml — example for AWS EBS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com # AWS EBS CSI driver
parameters:
type: gp3 # GP3 SSD
iops: "3000"
throughput: "125"
encrypted: "true"
reclaimPolicy: Retain # keep the volume when PVC is deleted
volumeBindingMode: WaitForFirstConsumer # don't provision until a pod needs it
allowVolumeExpansion: true # allow resizing the volume later
Reclaim policies:
Policy → What happens when PVC is deleted Delete → PV and underlying storage are deleted. Default for cloud providers. Retain → PV remains. Must be manually reclaimed. Use for important data. Recycle → Deprecated. Scrubs the volume and makes it available again.
💡 Use
Retainfor anything you care about.Deleteis convenient but unforgiving. One accidentalkubectl delete pvcand your data is gone. In production databases,Retainis non-negotiable.
PersistentVolumes: The Actual Storage
You rarely create PVs manually in a cloud environment — StorageClasses provision them dynamically. But understanding the object is essential.
# pv-manual.yaml — manual PV (on-premises, local storage)
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce # only one node can mount this volume at a time
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /mnt/data # path on the node
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node-1 # only available on node-1
Access Modes
Mode → Abbreviation → Meaning ReadWriteOnce → RWO → One node can mount read-write. Most common. ReadOnlyMany → ROX → Many nodes can mount read-only. ReadWriteMany → RWX → Many nodes can mount read-write. Requires NFS or similar. ReadWriteOncePod → RWOP → Only one pod (not just node) can mount read-write. K8s 1.22+.
⚠️ RWO means one NODE, not one pod. Multiple pods on the same node can mount an RWO volume simultaneously. If you truly need one-pod exclusivity, use RWOP.
PersistentVolumeClaims: Requesting Storage
A PVC is how a pod asks for storage. The developer doesn’t need to know if it’s AWS EBS, GCP PD, or local NFS — they just describe what they need.
# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: database-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd # which StorageClass to use
resources:
requests:
storage: 20Gi # how much storage
$ kubectl apply -f pvc.yaml
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
database-pvc Bound pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890 20Gi RWO fast-ssd 5s
# ↑ Bound = a PV was found/created and linked
PVC Lifecycle States
Status → Meaning Pending → No matching PV found yet. Waiting for dynamic provisioning. Bound → Matched to a PV. Ready to use. Lost → The bound PV has disappeared. Data may be at risk.
Using a PVC in a Pod
# pod-with-storage.yaml
apiVersion: v1
kind: Pod
metadata:
name: database-pod
spec:
containers:
- name: postgres
image: postgres:15
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data # where postgres stores data
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: database-pvc # reference the PVC by name
$ kubectl apply -f pod-with-storage.yaml
# Write some data
$ kubectl exec -it database-pod -- psql -U postgres -c "CREATE TABLE test (id serial, name text);"
$ kubectl exec -it database-pod -- psql -U postgres -c "INSERT INTO test (name) VALUES ('hello');"# Delete the pod — data survives
$ kubectl delete pod database-pod# Recreate — data is still there
$ kubectl apply -f pod-with-storage.yaml
$ kubectl exec -it database-pod -- psql -U postgres -c "SELECT * FROM test;"
id | name
----+-------
1 | hello
This is the fundamental proof: the data outlives the pod because it lives on the PV, not in the container.
Dynamic Provisioning: The Cloud-Native Way
In cloud environments, you almost never create PVs manually. StorageClasses provision them automatically when a PVC is created:
# All you need in a cloud cluster — no PV manifest required
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-storage
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard # uses the default StorageClass
resources:
requests:
storage: 50Gi
$ kubectl apply -f pvc.yaml
# Within seconds, a cloud disk is provisioned automatically
$ kubectl get pv
NAME CAPACITY STATUS CLAIM
pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890 50Gi Bound default/app-storage
The cloud provider’s CSI (Container Storage Interface) driver handles the actual provisioning — creating an EBS volume on AWS, a Persistent Disk on GCP, or a Managed Disk on Azure.
StatefulSets: When Pods Can’t Be Interchangeable
Here’s the problem with using a Deployment for a database.
Deployments treat all pods as identical and interchangeable. They can be started in any order, stopped in any order, and assigned random names like postgres-7d4b9c-x2k9f. For a web server, this is perfect. For a database cluster, it's catastrophic.
Databases have requirements that Deployments can’t meet:
- Stable network identity — other nodes in the cluster need to address each member by a predictable name
- Stable storage — each instance needs its own dedicated, persistent volume that follows it across restarts
- Ordered startup and shutdown — a Postgres primary must start before its replicas
- Ordered rolling updates — update replicas before primary, not all at once
StatefulSets were built for exactly this.
# postgres-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: "postgres" # must match a Headless Service (below)
replicas: 3
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:15
ports:
- containerPort: 5432
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates: # each pod gets its OWN PVC
- metadata:
name: postgres-data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
$ kubectl apply -f postgres-statefulset.yaml
# Pods get predictable, stable names — not random hashes
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
postgres-0 1/1 Running 0 30s ← always starts first
postgres-1 1/1 Running 0 20s ← starts after postgres-0 is Ready
postgres-2 1/1 Running 0 10s ← starts after postgres-1 is Ready# Each pod gets its own PVC — named predictably
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY
postgres-data-postgres-0 Bound pvc-aaa... 20Gi
postgres-data-postgres-1 Bound pvc-bbb... 20Gi
postgres-data-postgres-2 Bound pvc-ccc... 20Gi
The Headless Service: Stable DNS for StatefulSet Pods
StatefulSets require a Headless Service — a Service with clusterIP: None. This doesn't create a virtual IP or load balance traffic. Instead it creates individual DNS records for each pod.
# headless-service.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres # must match StatefulSet's serviceName
spec:
clusterIP: None # this is what makes it headless
selector:
app: postgres
ports:
- port: 5432
targetPort: 5432
# Each pod now has its own stable DNS name:
# postgres-0.postgres.default.svc.cluster.local
# postgres-1.postgres.default.svc.cluster.local
# postgres-2.postgres.default.svc.cluster.local
# Other pods can address specific StatefulSet members directly
$ kubectl exec -it some-other-pod -- psql -h postgres-0.postgres -U postgres
This is what makes database replication possible inside Kubernetes — the primary and replicas can always find each other by name, even after restarts.
StatefulSet vs Deployment: When to Use Which
Characteristic → Deployment → StatefulSet
Pod names → Random (app-7d4b9c-x2k9f) → Stable (app-0, app-1, app-2)
Pod DNS → Via Service (shared) → Individual DNS per pod
Storage → Shared PVC or no PVC → Dedicated PVC per pod
Startup order → Parallel, any order → Sequential (0, 1, 2...)
Shutdown order → Parallel, any order → Reverse sequential (2, 1, 0)
Rolling updates → All pods at once → One at a time, reverse order
Use for → Stateless apps → Databases, queues, distributed systems
Volume Types Beyond PVCs
PVCs and PVs are the right tool for persistent data. But Kubernetes has other volume types for specific needs:
emptyDir — Temporary Shared Storage
Created when a pod starts, deleted when the pod is deleted. Perfect for sharing data between containers in the same pod, or for scratch space.
spec:
containers:
- name: app
volumeMounts:
- name: cache
mountPath: /tmp/cache
- name: sidecar
volumeMounts:
- name: cache
mountPath: /tmp/cache # same volume — both containers share it
volumes:
- name: cache
emptyDir: {} # created fresh for each pod
hostPath — Mounting a Node Directory
Mounts a directory from the host node into the pod. Powerful but dangerous — tightly couples your pod to a specific node.
volumes:
- name: host-logs
hostPath:
path: /var/log
type: Directory
⚠️ Avoid
hostPathin production. It breaks pod portability, creates security risks (pods can read host filesystem), and makes your cluster harder to reason about. Acceptable only for DaemonSets that genuinely need node-level access (log collectors, monitoring agents).
projected — Combining Multiple Sources
Mount multiple sources (ConfigMap, Secret, ServiceAccount token) into a single directory.
volumes:
- name: combined-config
projected:
sources:
- configMap:
name: app-config
- secret:
name: app-secrets
- serviceAccountToken:
path: token
expirationSeconds: 3600
Volume Snapshots: Point-in-Time Backups
Modern Kubernetes supports VolumeSnapshots — point-in-time copies of a PVC. Supported by most cloud CSI drivers.
# Take a snapshot
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-snapshot-2024-01
spec:
volumeSnapshotClassName: csi-aws-vsc
source:
persistentVolumeClaimName: postgres-data-postgres-0
---
# Restore from snapshot into a new PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-restored
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
dataSource:
name: postgres-snapshot-2024-01
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
Debugging Storage Issues
# PVC stuck in Pending
$ kubectl describe pvc <pvc-name>
# Common causes:
# "no persistent volumes available for this claim" → no matching PV, check StorageClass
# "waiting for a volume to be created" → dynamic provisioner is working (wait)
# "storageclass not found" → wrong storageClassName in PVC
# Check StorageClass exists
$ kubectl get storageclass# Check if CSI driver is running
$ kubectl get pods -n kube-system | grep csi# Pod can't mount volume
$ kubectl describe pod <pod-name>
# Look for: "Unable to attach or mount volumes"
# Common cause: volume is stuck on a different node (RWO limitation)# Check PV binding
$ kubectl get pv
$ kubectl describe pv <pv-name># Check volume is actually mounted inside the pod
$ kubectl exec -it <pod-name> -- df -h
$ kubectl exec -it <pod-name> -- mount | grep <mount-path>
Hands-On Lab
# 1. Create a PVC (on minikube, use the default StorageClass)
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-pvc
spec:
accessModes: [ReadWriteOnce]
storageClassName: standard
resources:
requests:
storage: 1Gi
EOF
# 2. Check it's bound
$ kubectl get pvc lab-pvc# 3. Create a pod that uses it
$ kubectl run storage-test \
--image=nginx:1.25 \
--overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"lab-pvc"}}],"containers":[{"name":"storage-test","image":"nginx:1.25","volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}'# 4. Write data
$ kubectl exec -it storage-test -- bash -c "echo 'survived a restart' > /data/proof.txt"
$ kubectl exec -it storage-test -- cat /data/proof.txt# 5. Delete the pod
$ kubectl delete pod storage-test# 6. Recreate and verify data survived
$ kubectl run storage-test \
--image=nginx:1.25 \
--overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"lab-pvc"}}],"containers":[{"name":"storage-test","image":"nginx:1.25","volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}'
$ kubectl exec -it storage-test -- cat /data/proof.txt
survived a restart ← the money moment# 7. Deploy a 3-replica StatefulSet and observe ordered startup
$ kubectl get pods --watch # in a second terminal
$ kubectl apply -f postgres-statefulset.yaml# 8. Verify each pod has its own PVC
$ kubectl get pvc# 9. Clean up
$ kubectl delete pod storage-test
$ kubectl delete pvc lab-pvc
Step 6 is the moment this episode is built around. The data survived. That’s the whole point.
What’s Next
In Episode 07, we tackle Ingress Controllers and Load Balancing — the production-grade way to expose multiple services to the internet through a single entry point, with path-based routing, host-based routing, TLS termination, and more. No more one-LoadBalancer-per-service sprawl.
Episode Topic
EP 01 Introduction to Kubernetes
EP 02 Kubernetes Architecture & Components
EP 03 Pods, Deployments & ReplicaSets
EP 04 Services & Networking
EP 05 ConfigMaps, Secrets & Environment Config
EP 06 ← You are here Storage: PersistentVolumes & StatefulSets
EP 07 Ingress Controllers & Load Balancing
EP 08 RBAC & Security Fundamentals
EP 09 Monitoring with Prometheus & Grafana
EP 10 Production Hardening & Helm Charts
Before the Next Article
Make sure you can explain — without notes — the difference between these three scenarios:
- A pod restarts. What happens to data written to the container’s filesystem?
- A pod restarts. What happens to data written to a mounted PVC?
- A PVC is deleted with
reclaimPolicy: Delete. What happens to the underlying data?
If all three are clear, you understand the Kubernetes storage model.
Stateful workloads are where a lot of engineers hit a wall with Kubernetes. You’ve just cleared it. The cluster that runs databases confidently is a fundamentally different beast from the one that only runs stateless APIs. See you in Episode 07.