Español

Never Put user_id In A Metric Label

One innocent metric label can 10x your metrics bill. The mechanism has a name: cardinality. In a time-series database like Prometheus, every unique combination of label values is a separate time series — stored, indexed, and queried forever. Put user_id on a metric and you didn't add a column; you multiplied your series count by the number of users.

Why one label detonates your storage

Prometheus is explicit about it: "every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored." A single counter with a label that takes 50,000 values is 50,000 series. Add a second unbounded label — say request_id — and you multiply, not add.

The docs even quantify the ceiling. A per-user quota metric across 10,000 nodes and 10,000 users would create "a double digit number of millions" of series — "too much for the current implementation of Prometheus." The guidance: keep the cardinality of a metric below 10, and treat anything over 100 as a design smell to fix.

The rule: labels are for bounded dimensions

A metric label answers "which bucket?" — and the set of buckets must be small, known, and stable. If new values appear every time a user signs up or a request arrives, the dimension is unbounded and belongs nowhere near a metric.

Note route, not the raw URL: /orders/{id} is one label value; /orders/8412, /orders/8413 … are millions. The templated form is the bounded version of the same idea.

Where the high-cardinality IDs actually go

You still need per-user and per-request detail — just not in metrics. The three-pillars answer:

• Logs carry the unbounded fields. A log line is cheap to write with user_id, request_id, and the full URL; you query it on demand instead of indexing it forever.
• Traces give you the single-request view, with IDs as span attributes.
• Exemplars link the two: a metric bucket can attach a sample trace ID, so you jump from "5xx rate spiked" straight to an example failing trace — without ever labeling the metric by request.

Metrics tell you that something is wrong and how much; logs and traces tell you which user and why.

Power moves

• Watch cardinality before it bites. In Prometheus, topk(20, count by (__name__)({__name__=~".+"})) surfaces your fattest metrics; count(count by (label) (metric)) shows how many values a single label carries. Alert when a label crosses a threshold.
• Template the path at ingestion. Normalize /orders/8412 → /orders/{id} in your HTTP middleware before it becomes a label. Most frameworks expose the matched route pattern for exactly this.
• Aggregate away unused dimensions. Recording rules (or Grafana Adaptive Metrics) roll a high-cardinality series down to the labels you actually query — Grafana reports ~35% average metrics-cost cuts from this alone.
• A label added is a label forever. Removing a label later is a breaking change to every dashboard and alert built on it. Add dimensions conservatively; it's far cheaper than clawing them back.

Resources

• Metric and label naming — Prometheus (the cardinality warning)
• Instrumentation: "Do not overuse labels" — Prometheus (keep cardinality below 10)
• What is high cardinality? — Grafana Labs
• How to manage high cardinality metrics in Prometheus and Kubernetes — Grafana Labs
• View exemplars: link metrics to traces — Grafana