KubeStateMetrics Documentation v2.13

kube-state-metrics (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of objects — it is not focused on the health of individual Kubernetes components, but rather on the health of various objects inside, such as deployments, nodes, and pods.

Quick Start

The fastest way to get KSM running is via Helm:

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-state-metrics \
  prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --create-namespace

kubectl rollout status deployment/kube-state-metrics \
  -n monitoring
# deployment "kube-state-metrics" successfully rolled out

kubectl port-forward svc/kube-state-metrics 8080:8080 -n monitoring
curl http://localhost:8080/metrics | head -50

Requirements

Helm Chart Installation

The Helm chart is the recommended installation method for production clusters. It handles RBAC, ServiceAccount, Service, and Deployment configuration automatically.

helm install kube-state-metrics \
  prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --create-namespace \
  --set replicaCount=2 \
  --set autosharding.enabled=true \
  --set resources.requests.cpu=50m \
  --set resources.requests.memory=128Mi \
  --set resources.limits.cpu=250m \
  --set resources.limits.memory=256Mi

Key Helm Values

Direct kubectl Installation

kubectl apply -f \
  https://github.com/kubernetes/kube-state-metrics/releases/latest/download/kube-state-metrics.yaml

Configuration Flags

RBAC Configuration

KSM requires read access to all Kubernetes objects it monitors. Below is the minimal ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
  - apiGroups: [""]
    resources: [configmaps, secrets, nodes, pods, services,
      serviceaccounts, resourcequotas, replicationcontrollers,
      limitranges, persistentvolumeclaims, persistentvolumes,
      namespaces, endpoints]
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: [statefulsets, daemonsets, deployments, replicasets]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: [cronjobs, jobs]
    verbs: ["list", "watch"]
  - apiGroups: ["autoscaling"]
    resources: [horizontalpodautoscalers]
    verbs: ["list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses, networkpolicies]
    verbs: ["list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: [storageclasses, volumeattachments]
    verbs: ["list", "watch"]

Allow & Deny Lists

# Only expose specific metrics (allowlist)
--metric-allowlist=kube_pod_status_phase,kube_deployment_.*,kube_node_.*

# Exclude specific metrics (denylist)
--metric-denylist=kube_secret_.*,kube_configmap_.*

# Include specific Kubernetes labels as metric labels
--metric-labels-allowlist=pods=[app,version,tier],deployments=[app]

Horizontal Sharding

For large clusters (>5,000 nodes or >50,000 pods), distribute the KSM workload across multiple instances:

# Shard 0 of 4
./kube-state-metrics \
  --resources=pods,deployments \
  --shard=0 \
  --total-shards=4

With AutoSharding enabled in the Helm chart, KSM automatically assigns shards based on the StatefulSet ordinal:

helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
  --set replicaCount=4 \
  --set autosharding.enabled=true

Custom Resource Support

Expose metrics from your own CRDs without writing any code:

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: mycompany.io
        version: v1alpha1
        kind: MyApp
      labelsFromPath:
        app_name: [metadata, name]
        namespace: [metadata, namespace]
      metrics:
        - name: myapp_ready
          help: "Whether the MyApp resource is ready"
          each:
            type: Gauge
            gauge:
              path: [status, ready]

Prometheus Integration

scrape_configs:
  - job_name: kube-state-metrics
    honor_labels: true
    static_configs:
      - targets:
          - kube-state-metrics.monitoring.svc.cluster.local:8080

With Prometheus Operator, use a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
    - port: http
      interval: 30s
      honorLabels: true

Grafana Dashboards

Alert Rules

groups:
  - name: kubernetes-apps
    rules:
      - alert: KubePodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[15m]) > 0
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "Pod {{ $labels.pod }} is crash looping"

      - alert: KubeDeploymentReplicasMismatch
        expr: |
          kube_deployment_spec_replicas !=
          kube_deployment_status_replicas_available
        for: 15m
        labels: {severity: warning}

      - alert: KubeNodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 15m
        labels: {severity: critical}

      - alert: KubePVCPending
        expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
        for: 5m
        labels: {severity: warning}

      - alert: KubeHPAMaxedOut
        expr: |
          kube_horizontalpodautoscaler_status_current_replicas ==
          kube_horizontalpodautoscaler_spec_max_replicas
        for: 30m
        labels: {severity: warning}