Skip to content

Object Storage seamless migration#3275

Merged
matt-aitken merged 17 commits intomainfrom
object-storage-migration
Apr 1, 2026
Merged

Object Storage seamless migration#3275
matt-aitken merged 17 commits intomainfrom
object-storage-migration

Conversation

@matt-aitken
Copy link
Copy Markdown
Member

This allows seamless migration to different object storage.

Existing runs that have offloaded payloads/outputs will continue to use the default object store (configured using OBJECT_STORE_* env vars).

You can add additional stores by setting new env vars:

  • OBJECT_STORE_DEFAULT_PROTOCOL this determines where new run large payloads will get stored.
  • If you set that you need to set new env vars for that protocol.

Example:

OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3

@mintlify
Copy link
Copy Markdown
Contributor

mintlify bot commented Mar 26, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
trigger 🟢 Ready View Preview Mar 26, 2026, 10:40 AM

@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Mar 26, 2026

🦋 Changeset detected

Latest commit: 4071aca

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 29 packages
Name Type
@trigger.dev/core Patch
@trigger.dev/build Patch
trigger.dev Patch
@trigger.dev/python Patch
@trigger.dev/redis-worker Patch
@trigger.dev/schema-to-json Patch
@trigger.dev/sdk Patch
@internal/cache Patch
@internal/clickhouse Patch
@internal/llm-model-catalog Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/schedule-engine Patch
@internal/testcontainers Patch
@internal/tracing Patch
@internal/tsql Patch
@internal/zod-worker Patch
d3-chat Patch
references-d3-openai-agents Patch
references-nextjs-realtime Patch
references-realtime-hooks-test Patch
references-realtime-streams Patch
references-telemetry Patch
@internal/sdk-compat-tests Patch
@trigger.dev/react-hooks Patch
@trigger.dev/rsc Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 26, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • ✅ Review completed - (🔄 Check again to review again)

Walkthrough

Replaces the R2-specific object-store implementation with a new multi-protocol object store module (apps/webapp/app/v3/objectStore.server.ts) and deletes the old r2.server.ts. Import sites were updated to use the new module and callsites now capture returned stored filenames. The new module provides protocol-prefixed URI parsing/formatting, a protocol-aware client registry, upload/download/presign APIs, and support for a configurable default upload protocol. Environment examples and documentation were extended with OBJECT_STORE_DEFAULT_PROTOCOL and protocol-prefixed provider variables (e.g., OBJECT_STORE_S3_*, OBJECT_STORE_R2_*).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Object Storage seamless migration' directly and clearly describes the main change—adding support for seamless migration to different object storage providers with multi-provider configuration.
Description check ✅ Passed The description provides a clear overview of the feature (seamless migration, multi-provider support, env var configuration) and includes a practical example, but is missing several template sections including explicit testing steps, a formal changelog entry, and the contributor checklist.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch object-storage-migration

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 26, 2026

🧭 Helm Chart Prerelease Published

Version: 4.0.5-pr3275.4071aca

Install:

helm upgrade --install trigger \
  oci://ghcr.io/triggerdotdev/charts/trigger \
  --version "4.0.5-pr3275.4071aca"

⚠️ This is a prerelease for testing. Do not use in production.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@matt-aitken matt-aitken force-pushed the object-storage-migration branch from b03061c to 346c588 Compare March 30, 2026 11:08
This allows seamless migration to different object storage.
- Fix objectStore test failures: replace manual PrismaClient(process.env.DATABASE_URL)
  with proper postgresAndMinioTest fixture (new combined testcontainer)
- Fix undefined prefix in uploadDataToObjectStore: pathname now conditional
- Remove duplicate MAXIMUM_DEV_QUEUE_SIZE/MAXIMUM_DEPLOYED_QUEUE_SIZE rows from docs
Delegate the object store client to aws4fetch (existing) of the official AWS S3Client for IAM login.
devin-ai-integration[bot]

This comment was marked as resolved.

For large outputs
- Old SDKs will continue to use the old object storage
- New SDKs can use migrated object storage
devin-ai-integration[bot]

This comment was marked as resolved.

@matt-aitken matt-aitken marked this pull request as ready for review March 31, 2026 19:53
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

View 8 additional findings in Devin Review.

Open in Devin Review

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 0 new potential issues.

View 8 additional findings in Devin Review.

Open in Devin Review

@matt-aitken matt-aitken merged commit 68e88d0 into main Apr 1, 2026
45 checks passed
@matt-aitken matt-aitken deleted the object-storage-migration branch April 1, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants