Skip to content

Set lambda timeout in service#15632

Merged
thomtrp merged 4 commits intomainfrom
tt-set-lambda-timeout-in-service
Nov 5, 2025
Merged

Set lambda timeout in service#15632
thomtrp merged 4 commits intomainfrom
tt-set-lambda-timeout-in-service

Conversation

@thomtrp
Copy link
Copy Markdown
Contributor

@thomtrp thomtrp commented Nov 5, 2025

Some functions keep running without a timeout being thrown. Doing it in service directly.

Comment on lines 103 to 113
],
});

const resultServerlessFunction = await this.serverlessService.execute(
functionToExecute,
payload,
version,
);
const resultServerlessFunction = await this.callWithTimeout({
callback: () =>
this.serverlessService.execute(functionToExecute, payload, version),
timeoutMs: DEFAULT_SERVERLESS_FUNCTION_TIMEOUT_MS,
});

if (this.twentyConfigService.get('SERVERLESS_LOGS_ENABLED')) {
/* eslint-disable no-console */
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: User-configured timeoutSeconds for serverless functions are ignored, always defaulting to 5 minutes.
Severity: CRITICAL | Confidence: 1.00

🔍 Detailed Analysis

The callWithTimeout function in serverless-function.service.ts uses a hardcoded DEFAULT_SERVERLESS_FUNCTION_TIMEOUT_MS (5 minutes) for all serverless function executions. This overrides and ignores the user-configured functionToExecute.timeoutSeconds value, which can range from 1 to 900 seconds. Consequently, functions configured with a timeout greater than 5 minutes will still be terminated after 5 minutes, and functions configured with a shorter timeout will run for the full 5 minutes before timing out.

💡 Suggested Fix

Modify serverless-function.service.ts to use functionToExecute.timeoutSeconds * 1_000 instead of DEFAULT_SERVERLESS_FUNCTION_TIMEOUT_MS in the callWithTimeout function.

🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location:
packages/twenty-server/src/engine/metadata-modules/serverless-function/serverless-function.service.ts#L103-L113

Potential issue: The `callWithTimeout` function in `serverless-function.service.ts` uses
a hardcoded `DEFAULT_SERVERLESS_FUNCTION_TIMEOUT_MS` (5 minutes) for all serverless
function executions. This overrides and ignores the user-configured
`functionToExecute.timeoutSeconds` value, which can range from 1 to 900 seconds.
Consequently, functions configured with a timeout greater than 5 minutes will still be
terminated after 5 minutes, and functions configured with a shorter timeout will run for
the full 5 minutes before timing out.

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR moves timeout enforcement from the Lambda driver to the service layer to ensure consistent timeout behavior.

Key Changes:

  • Lambda timeout now hardcoded to 900s (AWS maximum) in lambda.driver.ts
  • New callWithTimeout helper wraps serverless function execution with 5-minute timeout
  • New constant DEFAULT_SERVERLESS_FUNCTION_TIMEOUT_MS defines the service-level timeout

Issues Found:

  • Memory leak in callWithTimeout - timeout not cleared when callback completes successfully
  • Creates architectural inconsistency: local driver still uses serverlessFunction.timeoutSeconds directly, while Lambda ignores it and relies on service-level timeout

Confidence Score: 2/5

  • PR has a memory leak that will accumulate with each serverless function execution
  • The callWithTimeout implementation doesn't clear the timeout when the callback completes successfully, causing timer handles to accumulate in memory. This will lead to memory growth over time as functions are executed.
  • packages/twenty-server/src/engine/metadata-modules/serverless-function/serverless-function.service.ts needs immediate fix for timeout cleanup

Important Files Changed

File Analysis

Filename Score Overview
packages/twenty-server/src/engine/metadata-modules/serverless-function/serverless-function.service.ts 2/5 Added timeout wrapper with memory leak issue - timeout not cleared when callback completes
packages/twenty-server/src/engine/core-modules/serverless/drivers/lambda.driver.ts 4/5 Hardcoded Lambda timeout to 900s (max), moved timeout enforcement to service layer
packages/twenty-server/src/engine/metadata-modules/serverless-function/constants/default-serverless-function-timeout-ms.constant.ts 5/5 New constant file defining 5-minute default timeout

Sequence Diagram

sequenceDiagram
    participant Client
    participant ServerlessFunctionService
    participant ServerlessService
    participant LambdaDriver
    participant AWS Lambda

    Client->>ServerlessFunctionService: executeOneServerlessFunction(id, payload)
    ServerlessFunctionService->>ServerlessFunctionService: throttleExecution()
    ServerlessFunctionService->>ServerlessFunctionService: callWithTimeout (5 min)
    Note over ServerlessFunctionService: Starts 5-minute timeout timer
    ServerlessFunctionService->>ServerlessService: execute(function, payload, version)
    ServerlessService->>LambdaDriver: execute(serverlessFunction, payload, version)
    LambdaDriver->>LambdaDriver: build & waitFunctionUpdates
    LambdaDriver->>AWS Lambda: invoke (900s timeout)
    AWS Lambda-->>LambdaDriver: result
    LambdaDriver-->>ServerlessService: result
    ServerlessService-->>ServerlessFunctionService: result
    Note over ServerlessFunctionService: ⚠️ Timeout NOT cleared - memory leak
    ServerlessFunctionService-->>Client: result
Loading

3 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

Copy link
Copy Markdown
Contributor

@martmull martmull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to update the local.driver also

Please add a test to verify the error message complies with what is already existing

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Nov 5, 2025

🚀 Preview Environment Ready!

Your preview environment is available at: http://bore.pub:45320

This environment will automatically shut down when the PR is closed or after 5 hours.

@thomtrp thomtrp enabled auto-merge (squash) November 5, 2025 10:39

const timeoutPromise = new Promise<never>((_, reject) => {
timeoutId = setTimeout(
() => reject(new Error(`Execution timeout: ${timeoutMs / 1000}s`)),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
() => reject(new Error(`Execution timeout: ${timeoutMs / 1000}s`)),
() => reject(new ServerlessFunctionException(`Execution timeout: ${timeoutMs / 1000}s`), SERVERLESS_FUNCTION_TIMEOUT),

@thomtrp thomtrp force-pushed the tt-set-lambda-timeout-in-service branch from 28adf67 to e2845e6 Compare November 5, 2025 12:32
Copy link
Copy Markdown
Contributor

@martmull martmull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thank you

@thomtrp thomtrp merged commit f21b862 into main Nov 5, 2025
50 checks passed
@thomtrp thomtrp deleted the tt-set-lambda-timeout-in-service branch November 5, 2025 12:43
@sentry
Copy link
Copy Markdown

sentry bot commented Nov 6, 2025

Issues attributed to commits in this pull request

This pull request was merged and Sentry observed the following issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants