A cluster you can’t observe is a cluster you can’t trust. Here’s how to build the observability stack that tells you what’s happening inside your cluster — before your users do.
Kubernetes Zero to Hero · Episode 09 ·
We’ve built a lot over this series. Workloads running. Storage persisting. Traffic routing. Access controlled. Network isolated. But there’s one question we haven’t been able to answer with confidence:
Is everything actually healthy right now?
Not just “are the pods Running” — that’s the beginning, not the end. Is the API response time spiking? Is a node running out of memory? Is the database connection pool exhausted? Is a deployment silently failing readiness probes? Is disk usage creeping toward 90%?
Without observability, your cluster is a black box. You find out about problems when users report them — which is always too late. Prometheus and Grafana are the industry-standard answer. Together they give you metrics collection, storage, querying, visualisation, and alerting — the complete picture of what’s happening inside your cluster at any moment.
This episode builds the full stack from scratch.
The Observability Stack: What Each Piece Does
Before installing anything, understand the role each component plays:
┌─────────────────────────────────────────────────────────────┐
│ Your Cluster │
│ │
│ Pods → expose /metrics endpoint (Prometheus format) │
│ Nodes → node-exporter scrapes OS/hardware metrics │
│ K8s API → kube-state-metrics exposes object state │
│ │ │
│ ▼ │
│ Prometheus (scrapes & stores metrics) │
│ │ │
│ AlertManager (routes alerts) │
│ │ │
│ Grafana (visualises metrics) │
└─────────────────────────────────────────────────────────────┘
Prometheus — scrapes metrics from targets on a schedule, stores them in a time-series database, and evaluates alerting rules. It pulls metrics rather than receiving them pushed — targets expose an HTTP endpoint and Prometheus calls it.
Alertmanager — receives alerts from Prometheus and routes them to the right destination (Slack, PagerDuty, email, webhooks). Handles deduplication, grouping, and silencing.
Grafana — connects to Prometheus as a data source and renders dashboards. The visual layer — charts, heatmaps, stat panels, tables.
node-exporter — a DaemonSet that runs on every node and exposes host-level metrics: CPU, memory, disk I/O, network, filesystem usage.
kube-state-metrics — a service that listens to the Kubernetes API and exposes metrics about the state of Kubernetes objects: deployment replica counts, pod phase, PVC binding status, and more.
Installing the Stack: kube-prometheus-stack
The fastest and most production-ready way to install all of this is with the kube-prometheus-stack Helm chart — a curated bundle that installs Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics together, pre-configured to work with each other, with 20+ pre-built dashboards included.
Step 1: Install Helm
# Linux/macOS
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# Verify
helm version
version.BuildInfo{Version:"v3.13.0", ...}
Step 2: Add the Helm Repository
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update
# See available chart versions
helm search repo prometheus-community/kube-prometheus-stack
Step 3: Create a Monitoring Namespace
kubectl create namespace monitoring
kubectl label namespace monitoring purpose=monitoring
Step 4: Install kube-prometheus-stack
# Install with sensible defaults
helm install prometheus-stack \
prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--set grafana.adminPassword=YourSecurePassword123 \
--set prometheus.prometheusSpec.retention=15d \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=standard \
--set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=20Gi
For a values file approach (recommended for production):
# monitoring-values.yaml
grafana:
adminPassword: "YourSecurePassword123"
persistence:
enabled: true
size: 5Gi
ingress:
enabled: true
ingressClassName: nginx
hosts:
- grafana.example.com
tls:
- secretName: grafana-tls
hosts:
- grafana.example.com
prometheus:
prometheusSpec:
retention: 15d # keep 15 days of metrics
retentionSize: "18GB"
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 20Gi
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: 1000m
memory: 2Gi
alertmanager:
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 2Gi
nodeExporter:
enabled: true # metrics from every node
kubeStateMetrics:
enabled: true # metrics about K8s object state
helm install prometheus-stack \
prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--values monitoring-values.yaml
# Watch everything come up
kubectl get pods -n monitoring --watch
After a few minutes:
kubectl get pods -n monitoring
NAME READY STATUS AGE
alertmanager-prometheus-stack-alertmanager-0 2/2 Running 2m
prometheus-stack-grafana-7d8f9c-x2k9f 3/3 Running 2m
prometheus-stack-kube-state-metrics-6b9f8c-p3mzr 1/1 Running 2m
prometheus-stack-operator-5d6b7c-hn7wq 1/1 Running 2m
prometheus-stack-prometheus-node-exporter-4jxzq 1/1 Running 2m
prometheus-stack-prometheus-node-exporter-8p9wk 1/1 Running 2m
prometheus-prometheus-stack-prometheus-0 2/2 Running 2m
Accessing the Dashboards
Grafana
# Port-forward to access Grafana locally
kubectl port-forward -n monitoring \
svc/prometheus-stack-grafana 3000:80
# Open http://localhost:3000
# Username: admin
# Password: YourSecurePassword123
Navigate to Dashboards → Browse — you’ll find 20+ pre-built dashboards covering:
- Kubernetes / Cluster Resources
- Kubernetes / Nodes
- Kubernetes / Pods
- Kubernetes / Namespaces
- Node Exporter / Full
- Alertmanager / Overview
These are production-ready dashboards built by the Kubernetes community. You get deep cluster visibility in the first minute.
Prometheus UI
kubectl port-forward -n monitoring \
svc/prometheus-stack-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090
The Prometheus UI lets you run raw PromQL queries and inspect scrape targets — useful for debugging metrics collection.
Alertmanager UI
kubectl port-forward -n monitoring \
svc/prometheus-stack-kube-prometheus-alertmanager 9093:9093
# Open http://localhost:9093
Understanding PromQL: Querying Your Metrics
PromQL (Prometheus Query Language) is how you extract insight from your metrics. You don’t need to master it immediately — the pre-built dashboards cover most needs — but knowing the basics unlocks Grafana’s full power and helps you write custom alerts.
Metric Types
# Counter — always increasing (requests served, errors occurred)
http_requests_total{method="GET", status="200"}
# Gauge — can go up or down (current memory usage, replica count)
kube_deployment_status_replicas_ready
# Histogram — distribution of values (request duration, response size)
apiserver_request_duration_seconds_bucket
# Summary — similar to histogram, pre-computed percentiles
go_gc_duration_seconds
Essential PromQL Patterns
# Current CPU usage per node (percentage)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory usage per node (bytes used)
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
# Memory usage per node (percentage)
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
# Pod restart count in the last hour
increase(kube_pod_container_status_restarts_total[1h])
# Pods not in Running state
kube_pod_status_phase{phase!="Running", phase!="Succeeded"}
# Deployment replicas desired vs ready
kube_deployment_spec_replicas - kube_deployment_status_replicas_ready
# HTTP request rate (requests per second)
rate(http_requests_total[5m])
# HTTP error rate (percentage of requests that are 5xx)
rate(http_requests_total{status=~"5.."}[5m])
/
rate(http_requests_total[5m]) * 100
# 99th percentile request latency
histogram_quantile(0.99,
rate(apiserver_request_duration_seconds_bucket[5m])
)
# PVC usage percentage
(kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100
# Nodes with memory pressure
kube_node_status_condition{condition="MemoryPressure", status="true"}
💡 Start with rate() for counters, not raw values. Counters only go up — the raw value (http_requests_total = 45,392) tells you the total since the pod started, not the current rate. rate(http_requests_total[5m]) gives you requests per second averaged over 5 minutes — actually useful.
Instrumenting Your Own Applications
The pre-built dashboards cover cluster infrastructure. For application-level metrics — request rates, error rates, business metrics — you instrument your own code.
Adding a /metrics Endpoint
Python (with prometheus-client):
from prometheus_client import Counter, Histogram, start_http_server
import time
# Define metrics
REQUEST_COUNT = Counter(
'http_requests_total',
'Total HTTP requests',
['method', 'endpoint', 'status']
)
REQUEST_LATENCY = Histogram(
'http_request_duration_seconds',
'HTTP request latency',
['endpoint']
)
# Instrument your handlers
@app.route('/api/orders')
def get_orders():
with REQUEST_LATENCY.labels(endpoint='/api/orders').time():
result = fetch_orders()
REQUEST_COUNT.labels(
method='GET',
endpoint='/api/orders',
status='200'
).inc()
return result
# Start metrics server on port 8000
start_http_server(8000)
Go (with prometheus/client_golang):
import "github.com/prometheus/client_golang/prometheus"
var httpRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total HTTP requests",
},
[]string{"method", "endpoint", "status"},
)
func init() {
prometheus.MustRegister(httpRequests)
}// In your handler:
httpRequests.WithLabelValues("GET", "/api/orders", "200").Inc()
Node.js (with prom-client):
const client = require('prom-client');
const collectDefaultMetrics = client.collectDefaultMetrics;
collectDefaultMetrics();const httpRequests = new client.Counter({
name: 'http_requests_total',
help: 'Total HTTP requests',
labelNames: ['method', 'route', 'status_code'],
});// Express middleware
app.use((req, res, next) => {
res.on('finish', () => {
httpRequests.inc({
method: req.method,
route: req.path,
status_code: res.statusCode,
});
});
next();
});
// Expose metrics endpoint
app.get('/metrics', async (req, res) => {
res.set('Content-Type', client.register.contentType);
res.end(await client.register.metrics());
});
Telling Prometheus to Scrape Your App
# ServiceMonitor — tells Prometheus which services to scrape
# (kube-prometheus-stack watches for these automatically)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-monitor
namespace: production
labels:
release: prometheus-stack # must match the Prometheus operator's selector
spec:
selector:
matchLabels:
app: my-app # selects Services with this label
endpoints:
- port: metrics # port name on the Service
interval: 30s # scrape every 30 seconds
path: /metrics # path to scrape
# The Service must expose the metrics port with a named port
apiVersion: v1
kind: Service
metadata:
name: my-app-service
labels:
app: my-app
spec:
selector:
app: my-app
ports:
- name: http
port: 8080
targetPort: 8080
- name: metrics # named port referenced by ServiceMonitor
port: 8000
targetPort: 8000
# Verify Prometheus is scraping your app
# Open Prometheus UI → Status → Targets
# Your app should appear with State: UP
kubectl port-forward -n monitoring \
svc/prometheus-stack-kube-prometheus-prometheus 9090:9090
# Navigate to: http://localhost:9090/targets
Alerting: Knowing Before Your Users Do
Writing Alert Rules
# custom-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: custom-alerts
namespace: production
labels:
release: prometheus-stack # must match operator selector
spec:
groups:
- name: pod-alerts
interval: 1m
rules:
# Alert when a pod keeps crashing
- alert: PodCrashLooping
expr: |
increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m # must be true for 5 minutes before firing
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} is crash looping"
description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has restarted {{ $value }} times in 15 minutes."
# Alert when a deployment has unavailable replicas
- alert: DeploymentReplicasMismatch
expr: |
kube_deployment_spec_replicas
!=
kube_deployment_status_replicas_ready
for: 10m
labels:
severity: critical
annotations:
summary: "Deployment {{ $labels.deployment }} has mismatched replicas"
description: "Deployment {{ $labels.deployment }} in {{ $labels.namespace }} has {{ $value }} replicas unavailable."
# Alert when node memory is high
- alert: NodeMemoryHigh
expr: |
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
for: 5m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.instance }} memory usage above 85%"
description: "Node memory usage is {{ $value | humanize }}%."
# Alert when disk is filling up
- alert: PersistentVolumeFillingUp
expr: |
(kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "PVC {{ $labels.persistentvolumeclaim }} is {{ $value | humanize }}% full"
description: "PVC {{ $labels.persistentvolumeclaim }} in {{ $labels.namespace }} will fill up within 24 hours at current rate."
# Alert on high HTTP error rate
- alert: HighHTTPErrorRate
expr: |
rate(http_requests_total{status=~"5.."}[5m])
/
rate(http_requests_total[5m]) * 100 > 5
for: 2m
labels:
severity: critical
annotations:
summary: "High HTTP error rate on {{ $labels.job }}"
description: "{{ $value | humanize }}% of requests are returning 5xx errors."
kubectl apply -f custom-alerts.yaml
# Verify Prometheus picked up the rules
# Open Prometheus UI → Alerts
# Your custom rules should appear
Configuring Alertmanager: Routing to Slack
# alertmanager-config.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: slack-alerts
namespace: production
labels:
release: prometheus-stack
spec:
route:
groupBy: ['alertname', 'namespace']
groupWait: 30s # wait 30s before sending first alert in a group
groupInterval: 5m # send grouped updates every 5 minutes
repeatInterval: 4h # re-send unresolved alerts every 4 hours
receiver: slack-warning
routes:
- matchers:
- name: severity
value: critical
receiver: slack-critical # critical alerts go to a different channel
receivers:
- name: slack-warning
slackConfigs:
- apiURL:
name: slack-webhook-secret
key: webhook-url
channel: '#k8s-alerts'
sendResolved: true
title: '{{ .GroupLabels.alertname }}'
text: |
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
{{ end }}
- name: slack-critical
slackConfigs:
- apiURL:
name: slack-webhook-secret
key: webhook-url
channel: '#k8s-critical'
sendResolved: true
# Create the Slack webhook Secret
kubectl create secret generic slack-webhook-secret \
--from-literal=webhook-url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
-n production
kubectl apply -f alertmanager-config.yaml
Building a Custom Grafana Dashboard
The pre-built dashboards are excellent for infrastructure. For your application metrics, build a custom dashboard:
# Access Grafana
kubectl port-forward -n monitoring svc/prometheus-stack-grafana 3000:80
# In Grafana:
# 1. Click + → New Dashboard
# 2. Add Panel → choose visualization type
# 3. Write PromQL in the query box
# 4. Title and configure the panel
# 5. Save the dashboard
Essential panels for a service dashboard:
# Panel 1: Request rate (Time series)
rate(http_requests_total{job="my-app"}[5m])
# Panel 2: Error rate percentage (Stat panel)
rate(http_requests_total{job="my-app", status=~"5.."}[5m])
/
rate(http_requests_total{job="my-app"}[5m]) * 100
# Panel 3: 50th/95th/99th percentile latency (Time series)
histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{job="my-app"}[5m]))
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="my-app"}[5m]))
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="my-app"}[5m]))
# Panel 4: Pod restart count (Stat)
sum(increase(kube_pod_container_status_restarts_total{namespace="production"}[1h]))
# Panel 5: Ready replicas vs desired (Gauge)
kube_deployment_status_replicas_ready{deployment="my-app"}
/
kube_deployment_spec_replicas{deployment="my-app"}
# Panel 6: Memory usage per pod (Time series)
container_memory_working_set_bytes{namespace="production", container!=""}
💡 Export your dashboards as JSON and store them in Git. Grafana dashboards are stateful — if the Grafana pod is deleted without persistence, your dashboards are gone. Export via Dashboard → Share → Export, commit the JSON, and use Grafana’s dashboard provisioning to restore them automatically.
The Golden Signals: What to Always Monitor
Google’s SRE Book defines the four golden signals — the minimum viable observability for any service:
Latency — How long does a request take? Distinguish successful from failed request latency.
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
Traffic — How much demand is your service receiving?
rate(http_requests_total[5m])
Errors — How many requests are failing, and why?
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])Saturation — How full is your service? CPU, memory, connection pools.
container_memory_working_set_bytes / container_spec_memory_limit_bytes
Build a dashboard with these four for every service you run. If you only have these four, you have enough to detect and characterise nearly any production incident.
Debugging the Monitoring Stack
Prometheus not scraping a target
# Check Prometheus targets
kubectl port-forward -n monitoring \
svc/prometheus-stack-kube-prometheus-prometheus 9090:9090
# Navigate to Status → Targets
# Look for State: DOWN and the error message
# Common causes:
# 1. ServiceMonitor label doesn't match Prometheus operator's selector
kubectl get prometheus -n monitoring -o yaml | grep serviceMonitorSelector
# 2. Service port name doesn't match ServiceMonitor endpoint port
kubectl describe servicemonitor my-app-monitor -n production
# 3. Pod not exposing metrics correctly
kubectl exec -it <pod> -- wget -qO- http://localhost:8000/metrics | head -20
Alert not firing
# Check alert rule is loaded
# Prometheus UI → Alerts → find your rule
# State should be: inactive (ok), pending (condition met, waiting), firing
# Verify the PromQL expression returns data
# Prometheus UI → Graph → paste your alert expr
# Check Alertmanager received the alert
kubectl port-forward -n monitoring \
svc/prometheus-stack-kube-prometheus-alertmanager 9093:9093
# Navigate to http://localhost:9093/#/alerts
Grafana shows “No data”
# 1. Is Prometheus the data source?
# Grafana → Configuration → Data Sources → check Prometheus URL
# 2. Does the metric exist in Prometheus?
# Run the PromQL in Prometheus UI first
# 3. Is the time range appropriate?
# A 5m rate on a 1h old metric with no recent data shows nothing
# 4. Check metric name spelling
# Prometheus UI → Graph → start typing the metric name, use autocomplete
Hands-On Lab
# 1. Install kube-prometheus-stack
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
helm install prometheus-stack \
prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--set grafana.adminPassword=admin123 \
--set prometheus.prometheusSpec.retention=2d
# 2. Wait for everything to come up
kubectl get pods -n monitoring --watch
# 3. Access Grafana
kubectl port-forward -n monitoring svc/prometheus-stack-grafana 3000:80 &
# Open http://localhost:3000 (admin/admin123)
# 4. Explore the pre-built dashboards
# Dashboards → Kubernetes → Compute Resources → Cluster
# 5. Access Prometheus and run your first query
kubectl port-forward -n monitoring \
svc/prometheus-stack-kube-prometheus-prometheus 9090:9090 &
# Query: kube_pod_status_phase — see all pods and their phases
# 6. Deploy a test app and create load
kubectl create deployment load-test --image=nginx:1.25 --replicas=3
kubectl expose deployment load-test --port=80
# Generate some traffic
kubectl run load-gen --image=busybox \
-- sh -c "while true; do wget -q -O- http://load-test; sleep 0.1; done"
# 7. Watch node CPU metrics in Prometheus
# Query: rate(node_cpu_seconds_total{mode!="idle"}[1m])
# 8. Apply a custom alert rule
kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: lab-alerts
namespace: monitoring
labels:
release: prometheus-stack
spec:
groups:
- name: lab
rules:
- alert: TooManyPodRestarts
expr: increase(kube_pod_container_status_restarts_total[10m]) > 2
for: 1m
labels:
severity: warning
annotations:
summary: "Pod restarting too often"
EOF
# 9. Trigger the alert — create a crashing pod
kubectl run crasher --image=busybox -- sh -c "exit 1"
# Watch in Prometheus UI → Alerts for TooManyPodRestarts to appear
# 10. Clean up
kubectl delete deployment load-test load-gen
kubectl delete pod load-gen crasher
kubectl delete svc load-test
helm uninstall prometheus-stack -n monitoring
kubectl delete namespace monitoring
Step 9 is the payoff — deliberately causing a pod crash loop and watching your own alert fire in the Prometheus UI makes the alerting pipeline feel real and trustworthy rather than theoretical.
What’s Next
Episode 10 — the final episode in the series — covers Production Hardening and Helm Charts. We’ll package everything we’ve built into reusable Helm charts, apply the production hardening checklist across all the topics we’ve covered, and discuss the operational practices that separate a cluster you can sleep through from one that wakes you up at 2 AM.
Episode → Topic
EP 01 Introduction to Kubernetes
EP 02 Kubernetes Architecture & Components
EP 03 Pods, Deployments & ReplicaSets
EP 04 Services & Networking
EP 05 ConfigMaps, Secrets & Environment Config
EP 06 Storage: PersistentVolumes & StatefulSets
Bonus Setting Up a Cluster with kubeadm on EC2
Bonus Network Policy Deep Dive
EP 07 Ingress Controllers & Load Balancing
EP 08 RBAC & Security Fundamentals Bonus Namespaces Deep Dive
EP 09 ← You are here Monitoring with Prometheus & Grafana
EP 10 Production Hardening & Helm Charts
Before the Next Article
Three things to try before moving on:
- Open the Kubernetes / Compute Resources / Namespace dashboard in Grafana, filter to your namespace, and identify which pod is consuming the most memory.
- Write a PromQL query that shows you all pods with more than 0 restarts in the last hour. Run it in the Prometheus UI.
- Modify the PodCrashLooping alert rule threshold — change > 3 to > 1 — and deliberately crash a pod. Watch the alert move from inactive → pending → firing.
If you do all three, you’re not just someone who installed monitoring — you’re someone who can use it.
A cluster without observability is a cluster you’re flying blind in. You now have eyes — real-time metrics, persistent dashboards, and alerts that fire before your users notice. That’s the difference between reacting to incidents and preventing them. See you in the final episode.