You’ve learned every major concept in this series. Now we tie it all together — packaging your application with Helm, applying the complete production hardening checklist, and building the operational discipline that lets you sleep through the night.
Kubernetes Zero to Hero · Episode 10 ·
Ten episodes. Four bonus deep dives. One complete journey from “what is Kubernetes?” to running production workloads with full observability, access control, network isolation, and persistent storage.
This is the finale.
We’re going to do two things in this article. First, we’ll learn Helm — Kubernetes’ package manager — and package our application into a reusable, configurable chart. Second, we’ll compile the complete production hardening checklist — every security, reliability, and operational practice from across this series synthesised into one actionable document you can apply to any cluster.
By the end, you’ll have both the tooling and the standards to run Kubernetes in production with confidence.
Part One: Helm Charts
What Helm Is and Why It Exists
In the Kustomize episode (Bonus), we managed our own application configs with overlays and patches. Kustomize is the right tool for that. Helm solves a different problem: packaging and distributing Kubernetes applications.
When you install Prometheus, cert-manager, or NGINX Ingress, you’re not writing those manifests from scratch — you install a Helm chart. A chart is a bundled, parameterisable package of Kubernetes manifests with a standard structure, versioning, dependency management, and a values system for configuration.
You also write your own charts — for your own applications. A Helm chart for your service means:
- One command to deploy to any environment: helm install my-app ./chart --values prod-values.yaml
- Versioned releases with history: helm history my-app
- Instant rollback: helm rollback my-app 2
- Dependency management: your chart can declare it needs PostgreSQL and cert-manager installed first
Helm’s Three Core Concepts
Chart — the package. A directory of templates, a Chart.yaml describing it, and a values.yaml with defaults.
Release — an installation of a chart into a cluster. Installing the same chart twice creates two releases with different names. Each has independent state and history.
Repository — a collection of published charts. helm repo add adds a repo. helm install pulls from it.
Anatomy of a Helm Chart
# Create a new chart scaffold
helm create web-app
web-app/
├── Chart.yaml # chart metadata: name, version, description
├── values.yaml # default configuration values
├── charts/ # dependencies (sub-charts)
└── templates/ # Kubernetes manifest templates
├── deployment.yaml
├── service.yaml
├── ingress.yaml
├── serviceaccount.yaml
├── hpa.yaml
├── _helpers.tpl # reusable template snippets
├── NOTES.txt # post-install instructions shown to user
└── tests/
└── test-connection.yaml
Chart.yaml: Chart Metadata
# Chart.yaml
apiVersion: v2
name: web-app
description: A production-ready web application chart
type: application
version: 1.3.0 # chart version — bump when the chart changes
appVersion: "2.1.4" # version of the app being packaged
maintainers:
- name: Platform Team
email: platform@company.com
dependencies:
- name: postgresql # this chart requires PostgreSQL
version: "12.x.x"
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled # only install if enabled in values
values.yaml: The Configuration Contract
The values.yaml is the public interface of your chart. Every configurable aspect of your application should be a value with a sensible default.
# values.yaml
replicaCount: 1
image:
repository: my-registry/web-app
pullPolicy: IfNotPresent
tag: "" # defaults to chart appVersion if empty
serviceAccount:
create: true
automount: false # secure default
name: ""
service:
type: ClusterIP
port: 80
ingress:
enabled: false # disabled by default
className: nginx
annotations: {}
hosts:
- host: chart-example.local
paths:
- path: /
pathType: Prefix
tls: []
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 70
config:
logLevel: "info"
cacheTtl: "3600"
maxConnections: "10"
postgresql:
enabled: false # opt-in dependency
auth:
postgresPassword: ""
database: myapp
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
nodeSelector: {}
tolerations: []
affinity: {}Templates: Turning Values Into Manifests
Templates are Kubernetes manifests with Go template syntax injecting values. The key is keeping templates generic — every environment-specific value comes from values.yaml or --set.
# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "web-app.fullname" . }}
labels:
{{- include "web-app.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "web-app.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "web-app.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "web-app.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 10 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 8080
protocol: TCP
env:
- name: LOG_LEVEL
value: {{ .Values.config.logLevel | quote }}
- name: CACHE_TTL
value: {{ .Values.config.cacheTtl | quote }}
- name: MAX_CONNECTIONS
value: {{ .Values.config.maxConnections | quote }}
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 10 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 10 }}
resources:
{{- toYaml .Values.resources | nindent 10 }}
# templates/ingress.yaml
{{- if .Values.ingress.enabled -}} {{/* only rendered if enabled */}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "web-app.fullname" . }}
labels:
{{- include "web-app.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- toYaml .Values.ingress.tls | nindent 4 }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
pathType: {{ .pathType }}
backend:
service:
name: {{ include "web-app.fullname" $ }}
port:
number: {{ $.Values.service.port }}
{{- end }}
{{- end }}
{{- end }}
# templates/_helpers.tpl — reusable snippets
{{- define "web-app.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- define "web-app.labels" -}}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
{{ include "web-app.selectorLabels" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}{{- define "web-app.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}Environment-Specific Values Files
# values-production.yaml
replicaCount: 5
image:
tag: "v2.1.4"
ingress:
enabled: true
hosts:
- host: app.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: web-app-tls
hosts:
- app.example.com
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/ssl-redirect: "true"
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 20
targetCPUUtilizationPercentage: 70
config:
logLevel: "warning"
cacheTtl: "7200"
maxConnections: "50"
postgresql:
enabled: true
auth:
postgresPassword: "changeme"
database: myapp_production
# Lint the chart — catch template errors before deploying
helm lint ./web-app
# Render templates locally — review the full YAML output
helm template my-app ./web-app --values values-production.yaml
# Dry run — sends to API Server for validation but doesn't apply
helm install my-app ./web-app \
--values values-production.yaml \
--namespace production \
--dry-run
# Install
helm install my-app ./web-app \
--values values-production.yaml \
--namespace production \
--create-namespace
# Check the release
helm list -n production
helm status my-app -n production
# Upgrade (rolling update)
helm upgrade my-app ./web-app \
--values values-production.yaml \
--namespace production
# See release history
helm history my-app -n production
REVISION STATUS CHART APP VERSION DESCRIPTION
1 superseded web-app-1.2.0 2.1.3 Install complete
2 superseded web-app-1.2.0 2.1.3 Upgrade complete
3 deployed web-app-1.3.0 2.1.4 Upgrade complete
# Rollback to previous revision
helm rollback my-app 2 -n production
# Uninstall
helm uninstall my-app -n production
Part Two: The Production Hardening Checklist
This is the other half of this episode — and arguably the more important half. Every topic we’ve covered in this series contributes to this checklist. Use it as a pre-launch gate for any production cluster or application.
✅ Cluster Foundation
□ Kubernetes version is within 2 minor versions of latest stable
□ Control plane runs across 3 nodes (HA etcd, no single point of failure)
□ etcd is backed up automatically on a schedule (test restores quarterly)
□ Worker nodes are distributed across 3+ availability zones
□ Node OS is hardened — minimal packages, automatic security patches enabled
□ kubeadm-managed clusters: kubeadm upgrade plan run before every upgrade
□ Managed clusters (EKS/GKE/AKS): upgrade policy set, not left on old versions
✅ Namespace and Organisational Hygiene
□ No production workloads run in the default namespace
□ Every namespace has a ResourceQuota and LimitRange
□ Every namespace has meaningful labels (environment, team, purpose)
□ Pod Security Admission enforced: restricted profile on all non-system namespaces
□ Namespace ownership is documented (owner annotation on namespace object)
✅ Workload Configuration
□ Every container has resource requests AND limits
□ Every container has liveness AND readiness probes
□ runAsNonRoot: true on all pod security contexts
□ readOnlyRootFilesystem: true on all containers that support it
□ allowPrivilegeEscalation: false on all containers
□ capabilities.drop: [ALL] on all containers — re-add only what's needed
□ No containers run as root (UID 0)
□ No privileged containers outside of system namespaces
□ automountServiceAccountToken: false on all pods that don't call the K8s API
□ Image tags are pinned — never using :latest in production
□ Images are pulled from a private registry, not Docker Hub directly
□ Container images are scanned for vulnerabilities before deployment
□ PodDisruptionBudgets defined for all critical workloads (minAvailable: 1+)
✅ Networking
□ Default deny NetworkPolicy in every production namespace
□ DNS egress rule included (UDP/TCP 53 to kube-system)
□ Kubelet probe IPs allowed on health check ports
□ Only explicitly needed inter-service communication is allowed
□ AWS metadata endpoint (169.254.169.254) blocked from application pods
□ Ingress Controller installed — not one LoadBalancer Service per application
□ All external traffic is HTTPS — HTTP redirects to HTTPS
□ TLS certificates managed by cert-manager (automatic renewal)
□ Services use ClusterIP internally — NodePort/LoadBalancer only where necessary
✅ Access Control (RBAC)
□ cluster-admin bound only to named human admins — never to groups or ServiceAccounts
□ Application ServiceAccounts have minimum required permissions only
□ No wildcard permissions (* verbs or * resources) in application Roles
□ Developers have edit (not admin) in their namespaces
□ Production access requires elevated permissions, not the same as staging
□ kubectl access to production requires explicit authentication (not default kubeconfig)
□ All cluster access is logged and audited
□ ServiceAccount tokens are rotated — use projected tokens with expiry
□ RBAC permissions reviewed and pruned quarterly
✅ Configuration and Secrets
□ No credentials hardcoded in container images or YAML manifests
□ No Secret YAML files with real values committed to Git
□ etcd encryption at rest enabled for Secrets
□ All non-sensitive config externalised to ConfigMaps
□ All sensitive config externalised to Secrets
□ Secrets sourced from a vault (AWS Secrets Manager, HashiCorp Vault, etc.) where possible
□ ConfigMaps are immutable in production (new version = new ConfigMap + rolling update)
□ No :latest image tags (config drift risk from image rebuilds)
✅ Storage
□ All stateful workloads use PersistentVolumeClaims — no hostPath in production
□ StorageClass reclaimPolicy is Retain for all production databases
□ PVC snapshots scheduled for all critical stateful workloads
□ PVC backup restore tested — not just created
□ StatefulSets used for all databases and clustered stateful apps
□ Headless Services defined for all StatefulSets
□ Storage capacity monitored and alerted (alert at 80% full)
✅ Observability
□ Prometheus + Grafana installed and scraping all workloads
□ node-exporter installed on all nodes
□ kube-state-metrics installed
□ Custom /metrics endpoint on all application services
□ ServiceMonitors defined for all applications
□ The four Golden Signals monitored for every service (latency, traffic, errors, saturation)
□ Alerting configured for: pod crash loop, high error rate, node memory pressure,
disk filling, deployment replica mismatch, certificate expiry
□ Alerts route to on-call rotation (PagerDuty, OpsGenie, etc.)
□ Grafana dashboards exported as JSON and committed to Git
□ Prometheus retention set appropriately (15-30 days minimum)
□ Log aggregation configured (Loki, ELK, Cloudwatch, Datadog)
□ Distributed tracing considered for complex microservices (Jaeger, Tempo)
✅ Reliability and Availability
□ Deployments use RollingUpdate strategy (never Recreate in production)
□ maxUnavailable: 0 set for zero-downtime deployments
□ minReadySeconds: 10+ set to ensure pods are stable before marking ready
□ All critical Deployments have replicas >= 3
□ Pod anti-affinity rules spread pods across nodes and availability zones
□ Horizontal Pod Autoscaler configured for variable-load services
□ Cluster Autoscaler configured for variable node counts (cloud clusters)
□ PodDisruptionBudgets set so node drains don't kill all replicas simultaneously
□ Readiness gates defined for services with external dependencies (database, cache)
□ Termination grace period is sufficient for graceful shutdown
# The complete reliable Deployment spec — covers most checklist items above
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0 # zero downtime
minReadySeconds: 10 # pod must be ready 10s before being counted
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
serviceAccountName: web-app-sa
automountServiceAccountToken: false
terminationGracePeriodSeconds: 60 # time for graceful shutdown
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: web-app
topologyKey: kubernetes.io/hostname # spread across nodes
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: web-app
topologyKey: topology.kubernetes.io/zone # prefer different AZs
containers:
- name: web-app
image: my-registry/web-app:v2.1.4 # pinned tag — never :latest
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 2
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"] # brief delay before shutdown
# lets load balancer drain
volumeMounts:
- name: tmp
mountPath: /tmp # writable scratch space
- name: cache
mountPath: /app/cache # writable cache space
volumes:
- name: tmp
emptyDir: {}
- name: cache
emptyDir: {}
✅ Deployment Process
□ All manifests are in version control (Git)
□ Changes are applied via GitOps (ArgoCD/Flux) or a CI/CD pipeline — never kubectl apply by hand in production
□ Image tags are immutable — a tag once pushed is never overwritten
□ Canary or blue/green deployment strategy used for high-risk changes
□ Deployment change-cause annotation set (kubernetes.io/change-cause)
□ Helm releases versioned — helm history shows clear release trail
□ Rollback procedure documented and tested before it's needed
□ Staging environment mirrors production configuration as closely as possible
✅ Cost and Efficiency
□ Resource requests are tuned based on actual usage (not guessed)
□ Vertical Pod Autoscaler (VPA) used in recommendation mode to right-size requests
□ Unused namespaces and workloads cleaned up regularly
□ Spot/preemptible instances used for non-critical workloads
□ Node rightsizing reviewed quarterly
□ Cloud provider cost explorer reviewed monthly
□ LoadBalancer Services consolidated behind Ingress
The Operational Practices That Actually Matter
Checklists get you to the starting line. Operational discipline keeps you there. Here are the practices that separate teams that sleep through the night from teams that don’t.
Treat your runbooks like code. Every alert should have a corresponding runbook — a document that tells the on-call engineer exactly what to check, what to do, and what escalation looks like. Store runbooks in Git, link them from alert annotations.
Test your disaster recovery before disaster strikes. Restore from your etcd backup. Delete a PVC and restore from snapshot. Drain a node and watch pods reschedule. Kill the control plane node and verify the cluster recovers. Do this quarterly, not in production for the first time.
Own your upgrade cadence. Kubernetes releases three minor versions per year. Support for old versions drops after ~14 months. Skipping upgrades is not avoiding risk — it’s accumulating it. Maintain a policy: stay within two minor versions of current.
Gate production with staging. If staging doesn’t mirror production closely, it provides false confidence. Same manifests, same Helm values structure, same monitoring, different scale. Every production deployment should have passed through staging first.
Measure before you optimise. Before scaling up, check Prometheus. Is that pod actually using its CPU limit? Are you running 10 replicas of a service getting 5 requests per minute? Right-sizing based on metrics saves more money than any other efficiency effort.
The Journey We’ve Taken
Let’s look back at what you now know.
From Episode 01 where we asked “what is a container orchestrator?” to here — you can now:
- Reason about the control plane — what the API Server, Scheduler, Controller Manager, and etcd each do and what breaks when they fail
- Deploy and manage workloads — Pods, Deployments, ReplicaSets, StatefulSets, and the self-healing loop that underlies all of them
- Connect services — ClusterIP, NodePort, LoadBalancer, CoreDNS, Endpoints, and how traffic actually flows from request to pod
- Externalise configuration — ConfigMaps for non-sensitive config, Secrets for credentials, with honest security context about what “secret” actually means
- Persist data — PersistentVolumes, PVCs, StorageClasses, and the StatefulSet guarantees that make databases possible
- Lock down the network — NetworkPolicy default-deny, surgical allow rules, the DNS gotcha, and the AND/OR selector trap
- Expose services properly — Ingress Controllers, path-based and host-based routing, TLS termination, cert-manager automation
- Control who can do what — RBAC Roles, ClusterRoles, bindings, ServiceAccounts, Pod Security Admission, security contexts
- Organise your cluster — Namespaces, ResourceQuotas, LimitRanges, and the four namespace patterns that work in production
- Observe what’s happening — Prometheus metrics collection, PromQL queries, Grafana dashboards, Alertmanager routing
- Manage configuration across environments — Kustomize overlays without templates, without copying YAML
- Package and distribute applications — Helm charts, values files, release management, rollback
- Build from scratch — a real multi-node cluster on EC2 using kubeadm
- Harden for production — the complete checklist of security, reliability, and operational practices
That is the knowledge set of a proficient Kubernetes engineer. Not someone who runs kubectl apply and hopes — someone who understands what the cluster is doing, can diagnose what went wrong, and can make deliberate decisions about security, reliability, and cost.
Where to Go From Here
The series ends here. Your Kubernetes education doesn’t.
For deeper security: CKS (Certified Kubernetes Security Specialist) certification. Falco for runtime security. OPA/Gatekeeper for policy enforcement.
For GitOps: ArgoCD or Flux. ApplicationSets for multi-cluster management. The GitOps Toolkit.
For service mesh: Istio or Linkerd — mTLS between services, traffic management, observability beyond metrics.
For multi-cluster: Cluster API for cluster lifecycle management. Fleet or ArgoCD ApplicationSets for multi-cluster deployments.
For certifications: CKA (Certified Kubernetes Administrator) is the benchmark. The exam is hands-on — all the lab exercises throughout this series are direct preparation for it.
Final Hands-On: Package and Harden
# 1. Create a Helm chart for the application from this series
helm create web-app
cd web-app
# 2. Update values.yaml with secure defaults
# (use the values.yaml from this article as a starting point)
# 3. Lint the chart
helm lint .
# 4. Render and review the full output
helm template my-release . --values values-production.yaml
# 5. Deploy to a test namespace
kubectl create namespace helm-test
helm install my-release . \
--namespace helm-test \
--set image.tag=latest
# 6. Verify deployment
kubectl get all -n helm-test
helm status my-release -n helm-test
# 7. Simulate an upgrade
helm upgrade my-release . \
--namespace helm-test \
--set replicaCount=3
# 8. Check the release history
helm history my-release -n helm-test
# 9. Rollback
helm rollback my-release 1 -n helm-test
kubectl get pods -n helm-test --watch
# 10. Apply the production hardening checklist to your deployment
# Check each item in the YAML:
# - security context present?
# - resource limits set?
# - probes configured?
# - anti-affinity rules in place?
# - image tag pinned?
# 11. Clean up
helm uninstall my-release -n helm-test
kubectl delete namespace helm-test
Thank You
This series started with a simple premise: teach Kubernetes the way it should be taught — with honest explanations, real examples, deliberate hands-on exercises, and no hand-waving over the hard parts.
Every episode was designed so that by the end, you don’t just know the commands — you understand what the cluster is doing and why. That understanding is what makes the difference when something breaks at 2 AM.
If this series helped you, share it with someone who’s just starting their Kubernetes journey. And if you have feedback — what landed, what was unclear, what topic you’d like covered next — leave a comment. This series was written for you. The next one will be too.
You started at zero. You’re a hero now. Go build something.