This is a Go runtime panic, not a Kubernetes-specific error — it means the process dereferenced a pointer that was nil instead of pointing to a valid object, and it crashed instead of continuing. In kube-state-metrics it almost always shows up as a CrashLoopBackOff right after install or right after upgrading Kubernetes itself, and the fix depends on which collector triggered it.
What the Crash Looks Like
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x...]
goroutine 1 [running]:
k8s.io/kube-state-metrics/v2/internal/store.cronJobMetricFamilies...
/go/src/k8s.io/kube-state-metrics/internal/store/cronjob.go:120 +0x...
main.main()
/go/src/k8s.io/kube-state-metrics/main.go:...The important line is the one naming a store/<resource>.go file — that tells you which collector panicked. kubectl logs against the crashed container (add --previous once it's already restarted) is where you'll find this stack trace:
kubectl logs deploy/kube-state-metrics -n monitoring --previousKnown Trigger Patterns
This panic isn't one bug — it's a symptom that recurs across several different collectors, each with a different root cause. It's tracked upstream in issues such as kubernetes/kube-state-metrics#1898 and #362. The common threads:
| Trigger | Affected collectors | Why it happens |
|---|---|---|
API server returns a field as null that the collector assumes is always set | CronJob, HPA, PodDisruptionBudget | Optional spec/status fields (e.g. LastScheduleTime, .Status.CurrentMetrics) are nil until the object's controller populates them — a freshly created object can be scraped mid-reconcile |
| Kubernetes API version mismatch after a cluster upgrade | StatefulSet, Deployment | KSM was built against an older client-go/API schema; a field the running cluster no longer sends (or sends in a different shape) decodes to a nil pointer |
| Very large cluster, high object churn | Pod, generic reflector | A watched object is deleted between the list and the processing step, and the delete-handling path assumes the object is still present |
| Custom Resource State metrics misconfiguration | Custom Resource collector | A CustomResourceStateMetrics config path points at a field that doesn't exist on some (but not all) instances of the CRD |
Fix 1: Upgrade kube-state-metrics
Because most instances of this panic are collector bugs fixed as they're reported, the highest-success-rate fix is simply running a current release — check the version history for what's current, then:
helm repo update
helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
--namespace monitoring --reuse-valuesFix 2: Disable the Offending Collector as a Workaround
If you can't upgrade immediately (e.g. you're pinned to a KSM version for compatibility reasons), disable the specific collector named in the stack trace to stop the crash loop while you plan the upgrade:
# Example: stack trace points at store/cronjob.go — drop cronjobs
--set collectors="{pods,deployments,nodes,statefulsets,services}"
# Or via kubectl args if not using Helm:
--resources=pods,deployments,nodes,statefulsets,serviceskube_cronjob_* metrics at all) until you re-enable it after upgrading.Fix 3: Check for a Kubernetes/KSM Version Mismatch
If the panic started immediately after a Kubernetes cluster upgrade (not a KSM upgrade), confirm your KSM version is in the supported compatibility range for your cluster's Kubernetes minor version — running a KSM release older than what the project supports for your API server version is the single most common cause of this class of panic. Check the version compatibility table and upgrade KSM to match.
Diagnostic Checklist
- Get the stack trace:
kubectl logs deploy/kube-state-metrics -n monitoring --previous - Identify the collector from the file path in the trace (e.g.
store/cronjob.go) - Check whether the crash correlates with a specific object — did it start after creating/updating a particular CronJob, HPA, or CRD instance?
- Check your KSM version against the compatibility matrix for your Kubernetes version
- If blocked, disable the collector via
--resourcesas a temporary mitigation - Upgrade kube-state-metrics to the latest patch release
For CrashLoopBackOff caused by RBAC instead of a panic, see the ClusterRole & RBAC guide. For OOMKilled and CPU-related crashes, see the general troubleshooting guide.