Kube State Metrics Configuration for Better Monitoring

If kube-state-metrics is already running in your cluster, the real work has not even started yet.

Out of the box, it exposes a lot of data. Some of it is useful. Some of it is noise. And without proper kube state metrics configuration, it can quickly become hard to manage, expensive to store, and difficult to trust.

This guide walks you through how to control what gets exposed, how it is structured. Also how to shape it into something your monitoring stack can actually use, whether you are working with Prometheus, Grafana, or alerting systems.

What Kube State Metrics Configuration Actually Controls

At its core, configuration is about one thing: control. Kube-state-metrics reads Kubernetes object states and turns them into metrics. But without configuration, you do not decide:

• which resources are included
• which metrics are exposed
• how detailed those metrics are
• how labels are attached
• how custom resources are handled

Instead of accepting defaults, configuration lets you shape the data to match your environment.

How Configuration Works (High-Level View)

Kube state metrics configuration is not controlled from a single place. Instead, it is shaped across multiple layers that work together to define what data you collect, how it is structured, and how it is ultimately used.

At a high level, there are three primary ways to configure kube-state-metrics, each influencing a different part of your monitoring pipeline.

1. Command Flags (Most Common)

Command flags are the core of kube state metrics configuration. They are passed when the container starts and directly control what the service exposes. Through flags, you can define:

• which collectors are enabled or disabled
• how labels are handled and exposed
• which Kubernetes resources are included
• whether to limit metrics to specific namespaces

This layer gives you the most direct control over metric generation. In most setups, fine-tuning flags is the difference between a clean monitoring system and one overloaded with unnecessary data.

2. YAML Configuration (Kubernetes Deployment)

When deploying kube-state-metrics in Kubernetes, configuration is typically managed through YAML, either via raw manifests or Helm values. Here, you define:

• container arguments (including flags)
• environment-level settings
• CPU and memory resource limits
• deployment-specific options

While flags control behavior, YAML controls how that behavior is deployed and maintained.

If you are using Helm, configuration becomes significantly more manageable through structured values files. This makes it easier to version, update, and standardize your kube state metrics config across environments. It is worth understanding how the Helm setup works.

3. Prometheus Scrape Configuration (Indirect Control)

This layer sits outside kube-state-metrics but plays a critical role in shaping your final dataset. Your Prometheus configuration decides:

• which metrics endpoints are scraped
• how frequently data is collected
• which labels are preserved, modified, or dropped

Even if kube-state-metrics exposes a wide range of metrics, Prometheus acts as a filter determining what actually gets stored and queried.

How These Layers Work Together

Each layer serves a different purpose:

• Flags → control what metrics are generated
• YAML → controls how the service is deployed and managed
• Prometheus → controls what data is retained and used

When aligned properly, they create a monitoring setup that is:

• efficient (no unnecessary data)
• scalable (handles growth without breaking)
• usable (clean metrics for dashboards and alerts)

Misalignment, on the other hand, often leads to duplicate data, missing insights, or performance issues.

Effective kube state metrics configuration is not about tweaking one setting. It is about understanding how these layers interact and optimizing them together.

Key Flags and Arguments You Should Know

In any real-world setup, kube state metrics configuration is driven mostly through flags. These arguments define what gets exposed, how detailed it is, and how manageable your metrics remain over time.

Instead of using defaults, tuning these flags helps you reduce noise, control scale, and keep your monitoring system predictable.

--resources

This flag controls which Kubernetes resource types kube-state-metrics will track.

Example:

--resources=pods,deployments,nodes

By default, kube-state-metrics can expose metrics for a wide range of objects. But in most environments, you do not need all of them. A focused approach works better:

• include only the resources you actively monitor
• avoid exposing rarely used objects like jobs or cronjobs unless required
• keep your metric set aligned with your dashboards and alerts

This keeps queries faster and reduces unnecessary metric volume.

--metric-labels-allowlist

This flag defines which Kubernetes labels should be converted into Prometheus labels.

Example:

--metric-labels-allowlist=pods=[app,version]

Kubernetes labels are powerful, but exposing all of them as metrics can quickly create high-cardinality data. A practical approach:

• allow only stable, meaningful labels (like app, team, environment)
• avoid dynamic or frequently changing labels
• align label selection with how you filter dashboards and alerts

Used carefully, this flag turns raw metrics into something actually usable.

--collectors

Collectors determine which internal modules are active, each responsible for exposing metrics for a specific resource type.

Example:

--collectors=pods,deployments

Instead of relying on defaults, it is often better to explicitly define collectors based on your monitoring needs. For example:

• infrastructure-focused setups → nodes, pods
• application-focused setups → deployments, replicasets
• minimal setups → only pods + deployments

This gives you tighter control over what data enters your monitoring pipeline.

--namespace

This flag limits metric collection to specific Kubernetes namespaces.

Example:

--namespace=production

In multi-tenant or multi-environment clusters, this becomes especially useful. You can:

• isolate production metrics from staging
• run separate monitoring pipelines per environment
• reduce clutter when debugging specific workloads

It also helps maintain clearer boundaries when different teams share the same cluster.

Putting These Flags Together

These flags are rarely used in isolation. In practice, they work best when combined:

--resources=pods,deployments \
--collectors=pods,deployments \
--metric-labels-allowlist=pods=[app] \
--namespace=production

This kind of setup:

• narrows scope
• controls metric size
• keeps monitoring aligned with real usage

The goal is not to configure everything, but it is to configure only what you need, and nothing more.

Understanding Collectors (What Gets Measured)

Collectors define what kube-state-metrics actually observes. Each collector is responsible for a specific Kubernetes object, such as:

• pods
• deployments
• nodes
• services

Why Collectors Matter More Than You Think

By default, many collectors are enabled. That sounds helpful but it often creates:

• unnecessary metrics
• higher storage usage
• slower queries

A better approach is intentional selection.

Practical Approach

Instead of enabling everything:

• start with core resources (pods, deployments, nodes)
• expand only when needed
• disable unused collectors

This keeps your monitoring focused and efficient.

Labels: Power and Risk in One Place

Labels are what make your metrics usable. They allow you to:

• filter dashboards
• group workloads
• create targeted alerts

But they also introduce one of the biggest problems in monitoring that is high cardinality.

What Is High Cardinality?

When labels create too many unique combinations, your metrics explode in size.

Example:

pod name + version + environment + instance ID

Now multiply that across hundreds of pods.

Smart Label Strategy

Instead of exposing everything:

• allow only meaningful labels
• avoid dynamic values (like random IDs)
• focus on stable identifiers (app, environment)

This keeps your metrics usable without overwhelming your system.

YAML Configuration in Practice

Most setups use YAML to define kube-state-metrics behavior. A simplified example:

containers:
  - name: kube-state-metrics
    image: kube-state-metrics:latest
    args:
      - --resources=pods,deployments
      - --metric-labels-allowlist=pods=[app]

What Matters in YAML

When configuring via YAML, focus on:

• clarity (keep args readable)
• maintainability (avoid overly complex configs)
• version control (track changes over time)

Treat your configuration as part of your infrastructure, not a one-time setup.

Custom Resources (CRDs): When You Need Them

By default, kube-state-metrics focuses on built-in Kubernetes objects. But sometimes, that is not enough.

When Custom Resources Make Sense

You might need them if:

• your platform uses CRDs heavily
• you want visibility into custom controllers
• standard metrics do not reflect your system behavior

Important Consideration

Custom resource support is powerful but not always necessary. If your setup works with standard resources, keep it simple.

Only introduce CRDs when there is a clear monitoring gap.

Common Configuration Mistakes to Avoid

Even experienced users run into these issues.

1. Enabling Everything by Default

More data does not mean better monitoring. It usually means:

• more noise
• higher cost
• slower queries

2. Ignoring Label Explosion

Uncontrolled labels can quietly break your monitoring setup. Always limit what you expose.

3. Mixing Too Many Config Layers

Flags, YAML, and Prometheus configs should work together, not against each other. Keep responsibilities clear.

4. Not Reviewing Metrics Output

Configuration is not “set and forget.” Check:

• what metrics are actually being exposed
• whether they are useful

5. Overcomplicating Early

Start simple. Then evolve based on real needs, not assumptions.

A Practical Configuration Workflow

If you want a clean, scalable setup, kube state metrics configuration should not be treated as a one-time task. It works best as an iterative process, where you start simple, observe real usage, and refine over time.

Step 1: Start Minimal

Begin with only the essentials. Enable core resources like pods, deployments, and nodes, and keep your label exposure tightly controlled.

This gives you a clean baseline to work from. Instead of dealing with hundreds of unnecessary metrics, you are starting with a focused dataset that is easier to understand, query, and validate.

Step 2: Observe Usage

Once your setup is live, spend time reviewing how the metrics are actually being used. Look at your dashboards, Prometheus queries, and any alerting rules already in place.

This step is where gaps become visible. You will quickly notice what is missing, what is redundant, and which metrics are never touched.

Step 3: Expand Gradually

Based on what you observe, start expanding your configuration in a controlled way. Add collectors only when they serve a clear purpose, and refine labels to improve filtering and grouping.

Avoid the temptation to enable everything at once. Gradual expansion keeps your monitoring system stable and prevents unnecessary complexity from creeping in.

Step 4: Optimize

Over time, revisit your configuration and clean it up. Remove metrics that are no longer useful, reduce label noise, and simplify queries wherever possible.

Optimization is what keeps your monitoring setup sustainable. Without it, even a well-configured system can slowly turn into something difficult to manage.

This workflow ensures your kube state metrics configuration evolves with your system, instead of becoming outdated or overly complex.

Conclusion

Kube-state-metrics is not just something you install, it is something you shape. Without configuration, you get raw data. With configuration, you get useful insight.

Focus on:

• limiting what you collect
• controlling labels carefully
• keeping your setup simple and intentional

That is what turns metrics into something you can actually rely on.

FAQ Section