Add proposal for per-tenant cardinality API by CharlieTLe · Pull Request #7335 · cortexproject/cortex

CharlieTLe · 2026-03-07T18:38:35Z

Summary

Proposal for a per-tenant cardinality API (GET /api/v1/cardinality) that exposes cardinality statistics (top metrics by series count, top labels by value count, top label-value pairs by series count) across two data sources:

source=head: Fans out to ingesters via the distributor, aggregates TSDB head stats with RF-based deduplication.
source=blocks: Fans out to store gateways via BlocksFinder + GetClientsFor, computes cardinality from block indexes with per-block caching.

Key design points:

start/end required for blocks path, rejected for head path (head cannot sub-filter)
Per-tenant limits: cardinality_api_enabled, cardinality_max_query_range, cardinality_max_concurrent_requests, cardinality_query_timeout
Standard {status, data} Prometheus response envelope with approximated field for block overlap / partial results
Phased rollout: head path first, blocks path second, behind per-tenant feature flag

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

yeya24 · 2026-03-09T22:02:03Z

+| `labelValueCountByLabelName` | No | Portable to block storage |
+| `seriesCountByLabelValuePair` | No | Portable to block storage |
+| `memoryInBytesByLabelName` | **Yes** | In-memory byte usage has no analogue in object storage |
+| `minTime` / `maxTime` | **Yes** | Reflects head time range, not total storage |


Do we need to add those head specific fields?

…ore gateways Add source=blocks query parameter to analyze cardinality from compacted blocks in object storage. The blocks path fans out to store gateways, which compute statistics from block index headers (cheap label value counts) and posting list expansion (exact series counts per metric). Results are cached per immutable block. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

…plify Address feedback from PR cortexproject#7335 review: - Rename endpoint from /api/v1/status/tsdb to /api/v1/cardinality - Drop Prometheus compatibility as a goal - Add start/end time range query parameters - Drop head-specific fields (numLabelPairs, memoryInBytesByLabelName, minTime, maxTime) to unify response across both sources - Remove API Compatibility and Field Portability sections Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

…limit Make start/end required for source=blocks to prevent unbounded block scanning. Add cardinality_max_query_range per-tenant limit (default 24h) to give operators control over the blast radius. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

Critical: - Fix blocks path aggregation: no SG RF division since GetClientsFor routes each block to exactly one store gateway Significant: - Add min_time, max_time, block_ids to store gateway CardinalityRequest - Specify MaxErrors=0 for head path with availability implications - Add consistency check and retry logic for blocks path - Document RF division as best-effort approximation Moderate: - Wrap responses in standard {status, data} Prometheus envelope - Change HTTP 422 to HTTP 400 for limit violations - Add Error Responses section with all validation scenarios - Add approximated field for block overlap and partial results - Add Observability section with metrics - Add per-tenant concurrency limit and query timeout - Reject start/end for source=head instead of silently ignoring Low: - Add Rollout Plan with phased approach and feature flag - Document rolling upgrade compatibility (Unimplemented handling) - Document Query Frontend bypass - Improve caching: full results keyed by ULID, limit at response time - Add missing files to implementation section - Move shared proto to pkg/cortexpb/cardinality.proto - Rename TSDBStatus* to Cardinality* throughout - Add limit upper bound (max 512) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

friedrichg

Looks like a great feature. thanks for working on this!

friedrichg · 2026-03-16T20:08:21Z

+- Zero read-time cost — statistics are available immediately from block metadata.
+- The compactor already reads the full block index during compaction and validation (`GatherIndexHealthStats`).
+
+**Cons:**


I think one extra con here is that we would need to align with block ranges

SungJin1212

LGTM, I think it would be good to configure the Admin UI using those APIs.

yeya24 · 2026-03-30T03:48:34Z

+
+The HTTP handler parses the `source` parameter and delegates to the appropriate backend. The endpoint is registered via `NewQuerierHandler` in `pkg/api/handlers.go` and does **not** go through the Query Frontend — it is served directly by the Querier. The Query Frontend's splitting, caching, and retry logic is designed for PromQL queries and does not apply to cardinality statistics. The Querier's own per-tenant concurrency limit provides sufficient request control (see [Per-Tenant Limits](#per-tenant-limits)).
+
+#### Head Path (`source=head`)


head or blocks seems more an implementation detail of the underlying storage. Probably better to just enforce with time range?
I think from a user perspective it feels very confusing to see these storage details

yeya24 · 2026-03-30T03:50:15Z

+
+#### Blocks Path (`source=blocks`)
+
+The request flows through the Querier's HTTP handler, which fans out to store gateways using the same `BlocksFinder` + `GetClientsFor` pattern used by `LabelNames`, `LabelValues`, and `Series`:


Umm I am confused I thought head and blocks are both ingesters but for inmemory and on disk part. Should we clarify it? These names feel a bit confusing to me

yeya24 · 2026-03-30T03:51:34Z

+
+#### Store Gateway Service
+
+A new `Cardinality` RPC is added to the StoreGateway service in `pkg/storegateway/storegatewaypb/gateway.proto`:


This would require changes to Thanos because we use Thanos Store Gateway underneath? So need a proposal there as well

* Add proposal for per-tenant TSDB status API Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com> * Extend TSDB status proposal with long-term storage cardinality via store gateways Add source=blocks query parameter to analyze cardinality from compacted blocks in object storage. The blocks path fans out to store gateways, which compute statistics from block index headers (cheap label value counts) and posting list expansion (exact series counts per metric). Results are cached per immutable block. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com> * Update proposal based on PR review: rename to Cardinality API and simplify Address feedback from PR cortexproject#7335 review: - Rename endpoint from /api/v1/status/tsdb to /api/v1/cardinality - Drop Prometheus compatibility as a goal - Add start/end time range query parameters - Drop head-specific fields (numLabelPairs, memoryInBytesByLabelName, minTime, maxTime) to unify response across both sources - Remove API Compatibility and Field Portability sections Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com> * Require start/end for blocks path and add per-tenant max query range limit Make start/end required for source=blocks to prevent unbounded block scanning. Add cardinality_max_query_range per-tenant limit (default 24h) to give operators control over the blast radius. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com> * Address all review findings from proposal review Critical: - Fix blocks path aggregation: no SG RF division since GetClientsFor routes each block to exactly one store gateway Significant: - Add min_time, max_time, block_ids to store gateway CardinalityRequest - Specify MaxErrors=0 for head path with availability implications - Add consistency check and retry logic for blocks path - Document RF division as best-effort approximation Moderate: - Wrap responses in standard {status, data} Prometheus envelope - Change HTTP 422 to HTTP 400 for limit violations - Add Error Responses section with all validation scenarios - Add approximated field for block overlap and partial results - Add Observability section with metrics - Add per-tenant concurrency limit and query timeout - Reject start/end for source=head instead of silently ignoring Low: - Add Rollout Plan with phased approach and feature flag - Document rolling upgrade compatibility (Unimplemented handling) - Document Query Frontend bypass - Improve caching: full results keyed by ULID, limit at response time - Add missing files to implementation section - Move shared proto to pkg/cortexpb/cardinality.proto - Rename TSDBStatus* to Cardinality* throughout - Add limit upper bound (max 512) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com> * Rename proposal file to per-tenant-cardinality-api.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com> --------- Signed-off-by: Charlie Le <charlie_le@apple.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

Add proposal for per-tenant TSDB status API

bd518d8

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

pull-request-size bot added the size/L label Mar 7, 2026

dosubot bot added the type/feature label Mar 7, 2026

CharlieTLe mentioned this pull request Mar 7, 2026

Add per-tenant TSDB cardinality status API endpoint #7332

Closed

4 tasks

yeya24 reviewed Mar 9, 2026

View reviewed changes

CharlieTLe and others added 5 commits March 13, 2026 12:38

Rename proposal file to per-tenant-cardinality-api.md

8199b43

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Charlie Le <charlie_le@apple.com>

CharlieTLe changed the title ~~Add proposal for per-tenant TSDB status API~~ Add proposal for per-tenant cardinality API Mar 13, 2026

friedrichg approved these changes Mar 16, 2026

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 16, 2026

SungJin1212 approved these changes Mar 20, 2026

View reviewed changes

CharlieTLe merged commit 38a0c36 into cortexproject:master Mar 30, 2026
64 of 65 checks passed

CharlieTLe mentioned this pull request Mar 30, 2026

Add per-tenant cardinality API endpoint #7384

Open

5 tasks

yeya24 reviewed Mar 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add proposal for per-tenant cardinality API#7335

Add proposal for per-tenant cardinality API#7335
CharlieTLe merged 6 commits intocortexproject:masterfrom
CharlieTLe:proposal/per-tenant-tsdb-status-api

CharlieTLe commented Mar 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeya24 Mar 9, 2026

Uh oh!

friedrichg left a comment

Uh oh!

friedrichg Mar 16, 2026

Uh oh!

SungJin1212 left a comment

Uh oh!

Uh oh!

yeya24 Mar 30, 2026

Uh oh!

yeya24 Mar 30, 2026

Uh oh!

yeya24 Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		The HTTP handler parses the `source` parameter and delegates to the appropriate backend. The endpoint is registered via `NewQuerierHandler` in `pkg/api/handlers.go` and does not go through the Query Frontend — it is served directly by the Querier. The Query Frontend's splitting, caching, and retry logic is designed for PromQL queries and does not apply to cardinality statistics. The Querier's own per-tenant concurrency limit provides sufficient request control (see [Per-Tenant Limits](#per-tenant-limits)).

		#### Head Path (`source=head`)


		#### Blocks Path (`source=blocks`)

		The request flows through the Querier's HTTP handler, which fans out to store gateways using the same `BlocksFinder` + `GetClientsFor` pattern used by `LabelNames`, `LabelValues`, and `Series`:


		#### Store Gateway Service

		A new `Cardinality` RPC is added to the StoreGateway service in `pkg/storegateway/storegatewaypb/gateway.proto`:

Conversation

CharlieTLe commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeya24 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

friedrichg left a comment

Choose a reason for hiding this comment

Uh oh!

friedrichg Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

SungJin1212 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yeya24 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

yeya24 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

yeya24 Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CharlieTLe commented Mar 7, 2026 •

edited

Loading