kube-state-metrics + Prometheus:
Scrape Config, ServiceMonitor & Operator

Once kube-state-metrics is deployed, you need Prometheus to scrape it and store the metrics as time-series data. This guide covers three integration patterns: static scrape config, Prometheus Operator ServiceMonitor, and the all-in-one kube-prometheus-stack Helm chart.

Method 1 — Static Prometheus Scrape Config

If you manage your Prometheus configuration directly (not via the Operator), add this job to your prometheus.yml:

scrape_configs:
  - job_name: kube-state-metrics
    honor_labels: true
    honor_timestamps: true
    scrape_interval: 30s
    static_configs:
      - targets:
          - kube-state-metrics.monitoring.svc.cluster.local:8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: cluster
        replacement: production

Method 2 — ServiceMonitor (Prometheus Operator)

If you're running the Prometheus Operator (part of kube-prometheus-stack), use a ServiceMonitor CRD instead of editing prometheus.yml directly:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    release: prometheus   # Must match your Prometheus CR's serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - port: http
      interval: 30s
      scrapeTimeout: 10s
      honorLabels: true
      honorTimestamps: true

Verify Prometheus picked up the ServiceMonitor target:

# Open Prometheus UI
kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring

# Then browse to http://localhost:9090/targets
# Look for job="kube-state-metrics" with State=UP

Method 3 — kube-prometheus-stack (All-in-One)

The easiest path for a fresh setup — kube-prometheus-stack installs Prometheus, Grafana, Alertmanager, node-exporter, and kube-state-metrics together with pre-wired scrape configs and dashboards:

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set kubeStateMetrics.enabled=true \
  --set grafana.enabled=true

Verifying kube-state-metrics Prometheus Scrape

After configuring the scrape job, verify data is flowing with a quick PromQL query:

# Count total pods by phase
count by (phase) (kube_pod_status_phase == 1)

# Check deployment health
kube_deployment_status_replicas_available
  / kube_deployment_spec_replicas

# Node ready status
kube_node_status_condition{condition="Ready",status="true"}

Fixing prometheus-kube-state-metrics CrashLoopBackOff

If kube-state-metrics enters CrashLoopBackOff after a Prometheus Operator upgrade, the most common causes are:

1. RBAC mismatch — The ClusterRole is missing permissions for new API groups. Run kubectl describe pod -n monitoring and look for Forbidden errors.
2. Resource exhaustion — In large clusters, KSM needs higher memory limits. Check with kubectl top pod -n monitoring.
3. Version incompatibility — Ensure kube-state-metrics version matches your Kubernetes version. See the compatibility matrix.

Essential kube-state-metrics Prometheus Alert Rules

Add these to your Prometheus alerting rules after confirming the scrape is working:

groups:
  - name: kube-state-metrics-alerts
    rules:
      - alert: KSMDown
        expr: absent(kube_pod_info)
        for: 5m
        labels: {severity: critical}
        annotations:
          summary: "kube-state-metrics is not scraping"

      - alert: KubePodNotRunning
        expr: kube_pod_status_phase{phase!="Running",phase!="Succeeded"} == 1
        for: 15m
        labels: {severity: warning}