Add prometheus exporter

[thomtrp]

Create a metric endpoint and expose prometheus gauge for event stream count.

[cubic-dev-ai]

No issues found across 7 files

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds Prometheus metrics support to expose live event stream counts via a new /metrics endpoint.

Key Changes:

• Added prom-client dependency for Prometheus integration
• Created PrometheusService to manage Prometheus registry and gauge metrics with support for custom collect callbacks
• Created PrometheusController exposing public /metrics endpoint for Prometheus scraping
• Added scanAndCountSetMembers() method to CacheStorageService for efficiently counting Redis set members across multiple keys matching a pattern
• Integrated metrics collection in EventStreamService to track twenty_event_streams_live_total gauge
• Default Node.js metrics are automatically collected via collectDefaultMetrics()

Implementation:
The implementation follows NestJS patterns with proper module imports and dependency injection. The metrics endpoint is marked as public (no authentication required) which is standard for Prometheus scrape targets. Error handling is implemented in the gauge collection callback to prevent metrics scraping failures from affecting the service.

Confidence Score: 4/5

• This PR is safe to merge with low risk
• The implementation is well-structured and follows established patterns in the codebase. However, the score is 4 instead of 5 because there are no tests for the new functionality, which is flagged by custom instruction 458c2737-e49d-49c4-a233-2605d1516703. The code quality is good with proper error handling, appropriate use of Redis operations, and clean separation of concerns. The public metrics endpoint is appropriately secured with guards.
• No files require special attention - all changes are straightforward and well-implemented

Important Files Changed

Filename	Overview
packages/twenty-server/src/engine/core-modules/cache-storage/services/cache-storage.service.ts	Added scanAndCountSetMembers method to count Redis set members matching a pattern
packages/twenty-server/src/engine/core-modules/metrics/prometheus.controller.ts	New controller exposing /metrics endpoint for Prometheus scraping with public access
packages/twenty-server/src/engine/core-modules/metrics/prometheus.service.ts	New service managing Prometheus registry, gauge creation, and metrics collection
packages/twenty-server/src/engine/subscriptions/event-stream.service.ts	Integrated Prometheus gauge tracking live event stream count with error handling

Sequence Diagram

sequenceDiagram
    participant Prometheus as Prometheus Server
    participant Controller as PrometheusController
    participant Service as PrometheusService
    participant EventStream as EventStreamService
    participant Cache as CacheStorageService
    participant Redis as Redis

    Note over Service: On Module Init
    Service->>Service: collectDefaultMetrics()
    EventStream->>Service: createGauge(config)
    Service->>Service: Register gauge with collect callback

    Note over Prometheus: Periodic Scrape
    Prometheus->>Controller: GET /metrics
    Controller->>Service: getMetrics()
    Service->>Service: Trigger collect callbacks
    Service->>EventStream: collect() callback executed
    EventStream->>EventStream: getTotalActiveStreamCount()
    EventStream->>Cache: scanAndCountSetMembers('workspace:*:activeStreams')
    Cache->>Redis: SCAN with pattern matching
    Redis-->>Cache: Return matching keys
    Cache->>Redis: SCARD for each key
    Redis-->>Cache: Return set member counts
    Cache-->>EventStream: Return total count
    EventStream->>Service: gauge.set(count)
    Service-->>Controller: Return formatted metrics
    Controller-->>Prometheus: 200 OK with metrics data

Loading

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​opentelemetry/​exporter-prometheus@​0.211.0

View full report

[github-actions]

🚀 Preview Environment Ready!

Your preview environment is available at: http://bore.pub:61555

This environment will automatically shut down when the PR is closed or after 5 hours.

[etiennejouan]

Looks good but I wonder why we don't rely on OTEL API (for all metrics collection) ?

[github-actions]

TODOs/FIXMEs:

• //TODO : Define meter name usage in monitoring: packages/twenty-server/src/engine/core-modules/metrics/metrics.service.ts
• //TODO : Define meter name usage in monitoring: packages/twenty-server/src/engine/core-modules/metrics/metrics.service.ts

Generated by 🚫 dangerJS against 9861253

[etiennejouan]

Tested and it works great ! 👏

(Difference with previous implementation is that all metrics are exported on /metrics endpoint, not only the one you want)

[Weiko]

This could be painful (N network calls), can we use Redis pipeline for that instead?

[Weiko]

Probably even better with LUA