kube-state-metrics Troubleshooting:
CrashLoopBackOff, OOMKilled & High CPU

kube-state-metrics is generally a low-maintenance component, but in large clusters or after upgrades, several issues commonly occur. This guide walks through every major failure mode — CrashLoopBackOff, OOMKilled, high CPU usage, and metrics not appearing — with root causes and fixes.

kube-state-metrics CrashLoopBackOff

A kube-state-metrics pod in CrashLoopBackOff restarts repeatedly because something is preventing it from starting or staying running.

Step 1: Check the crash logs

kubectl logs deploy/kube-state-metrics -n monitoring --previous
kubectl describe pod -l app.kubernetes.io/name=kube-state-metrics -n monitoring

Common Cause A: RBAC Permission Denied

After a Kubernetes upgrade, new API versions may require additional RBAC rules. Logs show:

E0101 error listing *v1.Pod: pods is forbidden:
  User "system:serviceaccount:monitoring:kube-state-metrics"
  cannot list resource "pods" in API group "" at the cluster scope

Fix: Re-apply or upgrade the Helm chart to get the updated ClusterRole, or manually add the missing resource to the ClusterRole:

helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring --reuse-values

Common Cause B: Invalid Flag / Configuration

Logs show unknown flag: --some-flag. This happens after upgrading kube-state-metrics when old custom flags were deprecated.

# Check current startup flags
kubectl get deploy kube-state-metrics -n monitoring \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

kube-state-metrics OOMKilled (Memory Issues)

In clusters with thousands of pods or nodes, kube-state-metrics builds a large in-memory cache of all objects. The default memory limit (often 128Mi) is insufficient.

Symptoms

kubectl describe pod -l app.kubernetes.io/name=kube-state-metrics -n monitoring
# Last State: Terminated  Reason: OOMKilled

Fix: Increase Memory Limits

# Rule of thumb: ~10MB per 1,000 pods
helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --set resources.limits.memory=512Mi \
  --set resources.requests.memory=256Mi \
  --reuse-values

Fix: Reduce Object Scope

# Only monitor specific namespaces
--set namespaces="production,staging"

# Only enable collectors you need
--set collectors="{pods,deployments,nodes}"

kube-state-metrics High CPU Usage

kube-state-metrics CPU usage spikes when:

• Prometheus scrape interval is too short (< 15 seconds)
• The cluster has high object churn (many pod creates/deletes per minute)
• Too many label allowlist entries increase per-metric computation

Diagnose CPU usage

kubectl top pod -l app.kubernetes.io/name=kube-state-metrics -n monitoring

# Check the KSM self-monitoring metrics (requires selfMonitor.enabled=true)
curl http://localhost:8080/metrics | grep process_cpu

Fixes for High CPU

# Increase Prometheus scrape interval (reduce load)
# In prometheus.yml or ServiceMonitor:
scrapeInterval: 60s

# Enable sharding to distribute CPU across replicas
helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
  --set replicaCount=3 \
  --set autosharding.enabled=true \
  --reuse-values

kube-state-metrics Metrics Not Appearing in Prometheus

If metrics are missing from Prometheus, check in order:

1. Is KSM running? kubectl get pods -n monitoring | grep kube-state-metrics
2. Is the service reachable? kubectl port-forward svc/kube-state-metrics 8080:8080 -n monitoring then curl http://localhost:8080/metrics | head -20
3. Is Prometheus scraping it? Check Prometheus UI → Status → Targets for job="kube-state-metrics"
4. Is the ServiceMonitor matching? Check the label selector on the ServiceMonitor matches the Service labels
5. Namespace issue? Ensure honor_labels: true is set in the scrape config

kube-state-metrics Pod Not Starting After Upgrade

After a Helm upgrade, the new pod may fail to start while the old pod is still terminating. This is normal with RollingUpdate strategy. If the new pod stays in Pending or Init:Error:

# Check events
kubectl describe pod -l app.kubernetes.io/name=kube-state-metrics -n monitoring | tail -20

# Common: PodSecurityPolicy or PodSecurity admission blocking the new pod
# Check pod security admission mode for the namespace
kubectl get namespace monitoring -o yaml | grep labels -A 5

Memory Usage Sizing Reference