Skip to content

[REQUIRES_FULL_CACHE_FLUSH_WHEN_RELEASED] Refactor FlatEntity to be UniversalFlatEntity superset#17452

Merged
prastoin merged 40 commits intomainfrom
introduce-flat-entity-super-set
Jan 28, 2026
Merged

[REQUIRES_FULL_CACHE_FLUSH_WHEN_RELEASED] Refactor FlatEntity to be UniversalFlatEntity superset#17452
prastoin merged 40 commits intomainfrom
introduce-flat-entity-super-set

Conversation

@prastoin
Copy link
Copy Markdown
Contributor

@prastoin prastoin commented Jan 26, 2026

Introduction

In this PR we're refactoring the FlatEntity type to become a superset of the UniversalFlatEntity.
Right now we're storing all the extra properties in __universal property, at some point it might just be sibling to other entity and we might rely on the propertiesToCompare constants and TypeScript allowing passing a superset type into a smaller subset type

FromTo utils

The entity to flat entity method now computes the universal information, standardized a typing and pattern to do

Example

Also strictly type

    "bbb019ea-6205-498c-aea5-67bc53bce8a9": {
      "workspaceId": "20202020-1c25-4d02-bf25-6aeccf7ea419",
      "universalIdentifier": "20202020-d111-4d11-8d11-da5ab0a11002",
      "applicationId": "d01b010d-b984-465b-b40b-370e954e5188",
      "id": "bbb019ea-6205-498c-aea5-67bc53bce8a9",
      "pageLayoutTabId": "791a512f-169f-4209-b731-aa86716668c6",
      "title": "Deals by Company",
      "type": "GRAPH",
      "objectMetadataId": "9e14efea-df5b-4c0e-aba9-cfe455f32397",
      "gridPosition": { "row": 0, "column": 6, "rowSpan": 6, "columnSpan": 6 },
      "configuration": {
        "color": "orange",
        "orderBy": "FIELD_ASC",
        "timezone": "UTC",
        "displayLegend": true,
        "displayDataLabel": false,
        "showCenterMetric": true,
        "configurationType": "PIE_CHART",
        "firstDayOfTheWeek": 0,
        "aggregateOperation": "COUNT",
        "groupBySubFieldName": "name",
        "groupByFieldMetadataId": "6673ff18-63d2-47a1-8f85-2b9b09ca27a5",
        "aggregateFieldMetadataId": "8d64ee41-5dd4-4de6-945a-7c0c18399715"
      },
      "createdAt": "2026-01-28T14:08:52.140Z",
      "updatedAt": "2026-01-28T14:08:52.140Z",
      "deletedAt": null,
      "__universal": {
        "universalIdentifier": "20202020-d111-4d11-8d11-da5ab0a11002",
        "applicationUniversalIdentifier": "20202020-64aa-4b6f-b003-9c74b97cee20",
        "pageLayoutTabUniversalIdentifier": "20202020-d011-4d11-8d11-da5ab0a01001",
        "objectMetadataUniversalIdentifier": "20202020-9549-49dd-b2b2-883999db8938",
        "gridPosition": {
          "row": 0,
          "column": 6,
          "rowSpan": 6,
          "columnSpan": 6
        },
        "configuration": {
          "color": "orange",
          "orderBy": "FIELD_ASC",
          "timezone": "UTC",
          "displayLegend": true,
          "displayDataLabel": false,
          "showCenterMetric": true,
          "configurationType": "PIE_CHART",
          "firstDayOfTheWeek": 0,
          "aggregateOperation": "COUNT",
          "groupBySubFieldName": "name",
          "aggregateFieldMetadataUniversalIdentifier": "20202020-d01a-4131-8a31-f123456789ab",
          "groupByFieldMetadataUniversalIdentifier": "20202020-cbac-457e-b565-adece5fc815f"
        }
      }
    },

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 26, 2026

🚀 Preview Environment Ready!

Your preview environment is available at: http://bore.pub:45640

This environment will automatically shut down when the PR is closed or after 5 hours.

@prastoin prastoin force-pushed the introduce-flat-entity-super-set branch 2 times, most recently from 78bcbda to 2b1f43f Compare January 27, 2026 12:13
@prastoin prastoin changed the title Introduce flat entity super set Refactor FlatEntity to be UniversalFlatEntity superset Jan 27, 2026
@prastoin prastoin self-assigned this Jan 27, 2026
github-merge-queue bot pushed a commit that referenced this pull request Jan 27, 2026
## Introduction
Handling nested serializedRelation references mapping
Removed the brand signature omit for the moment
I want to determine if it's really problematic later in the devx 
## Motivation

```ts
@ObjectType('RatioAggregateConfig')
export class RatioAggregateConfigDTO {
  @field(() => UUIDScalarType)
  @IsUUID()
  @isnotempty()
  fieldMetadataId: SerializedRelation;

  @field(() => String)
  @IsString()
  @isnotempty()
  optionValue: string;
}


@ObjectType('AggregateChartConfiguration')
export class AggregateChartConfigurationDTO
  implements PageLayoutWidgetConfigurationBase
{
// ...
  @field(() => RatioAggregateConfigDTO, { nullable: true })
  @ValidateNested()
  @type(() => RatioAggregateConfigDTO)
  @IsOptional()
  ratioAggregateConfig?: RatioAggregateConfigDTO;
}

```

Blocking #17452
@prastoin prastoin marked this pull request as ready for review January 28, 2026 14:20
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Jan 28, 2026

Greptile Overview

Greptile Summary

This PR refactors the FlatEntity type system to become a superset of UniversalFlatEntity, enabling a gradual migration path toward universal identifiers across the metadata system.

Key Changes

  • Type System Refactoring: FlatEntity now includes an optional __universal property containing UniversalFlatEntityExtraProperties, making it backward-compatible while supporting universal identifiers
  • Standardized Transformation Pattern: Introduced FromEntityToFlatEntityArgs type to standardize the signature of all entity-to-flat-entity transformation utilities, ensuring consistent access to universal identifier maps
  • Universal Identifier Computation: All from*EntityToFlat* utilities now compute universal identifiers for related entities and store them in the __universal property
  • JSONB Property Handling: Added ALL_JSONB_PROPERTIES_BY_METADATA_NAME constant and enhanced FormatRecordSerializedRelationProperties type to correctly handle JSONB properties and avoid incorrectly renaming Record<string, X> properties
  • Helper Utilities: Created createIdToUniversalIdentifierMap for efficient ID-to-universal-identifier lookups and isFieldMetadataSettingsOfType for type-safe settings validation

Migration Impact

The refactoring touches 87 files across all metadata modules, updating cache services to fetch universal identifiers and pass them to transformation utilities. This enables the system to work with both regular IDs and universal identifiers simultaneously during the migration period.

Architecture Notes

The approach allows TypeScript's structural typing to accept FlatEntity (superset) where UniversalFlatEntity (subset) is expected, enabling incremental adoption without breaking existing code.

Confidence Score: 4/5

  • This PR is safe to merge with careful monitoring - the refactoring is well-structured and comprehensive but requires runtime validation
  • The refactoring follows a sound architectural pattern with clear type safety. The changes are systematic and consistent across all affected modules. However, the score is 4 rather than 5 because: (1) the universal identifier maps rely on runtime data completeness - missing entities will throw exceptions, (2) the scale of changes (87 files) increases risk despite consistency, and (3) thorough integration testing is needed to ensure all transformation utilities handle edge cases correctly, particularly for nullable relations and JSONB properties
  • Monitor transformation utilities that handle complex JSONB properties (widget configurations, field metadata settings) and ensure runtime exception handling for missing universal identifiers is appropriate

Important Files Changed

Filename Overview
packages/twenty-server/src/engine/metadata-modules/flat-entity/types/flat-entity-from.type.ts Refactored to make FlatEntity a superset of UniversalFlatEntity by adding optional __universal property
packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/universal-flat-entity-from.type.ts Extracted UniversalFlatEntityExtraProperties type to enable reuse in FlatEntity type definition
packages/twenty-server/src/engine/workspace-cache/types/from-entity-to-flat-entity-args.type.ts New type standardizing arguments for entity-to-flat-entity transformation functions
packages/twenty-server/src/engine/metadata-modules/flat-field-metadata/utils/from-field-metadata-entity-to-flat-field-metadata.util.ts Updated to compute __universal property with universal identifiers for all related entities and settings
packages/twenty-server/src/engine/metadata-modules/flat-page-layout-widget/utils/from-page-layout-widget-configuration-to-universal-configuration.util.ts New utility for transforming widget configuration field metadata IDs to universal identifiers
packages/twenty-server/src/engine/workspace-manager/workspace-migration/universal-flat-entity/types/format-record-serialized-relation-properties.type.ts Enhanced to handle Record<string, X> types correctly, preventing incorrect property renaming

Sequence Diagram

sequenceDiagram
    participant Service as Cache Service
    participant Repo as TypeORM Repository
    participant Transform as Transform Utility
    participant MapUtil as Map Utilities
    participant Type as Type System

    Note over Service,Type: Entity to Flat Entity Transformation Flow

    Service->>Repo: Query entities with relations
    Repo-->>Service: Return entities[]
    
    Service->>Repo: Query related entities<br/>(applications, objects, fields)
    Repo-->>Service: Return related entities[]
    
    Service->>MapUtil: createIdToUniversalIdentifierMap(entities)
    MapUtil-->>Service: Map<id, universalIdentifier>
    
    Service->>MapUtil: regroupEntitiesByRelatedEntityId(relations)
    MapUtil-->>Service: Map<relatedId, RegroupedEntity[]>
    
    loop For each entity
        Service->>Transform: fromEntityToFlatEntity({<br/>entity, maps})
        
        Transform->>Type: Remove relation properties
        Type-->>Transform: Entity without relations
        
        Transform->>Type: Cast dates to strings
        Type-->>Transform: Entity with string dates
        
        Transform->>MapUtil: Lookup universal identifiers
        MapUtil-->>Transform: Universal identifiers
        
        Transform->>Transform: Build __universal property<br/>(universal IDs, JSONB transforms)
        
        Transform-->>Service: FlatEntity with __universal
        
        Service->>Service: Add to FlatEntityMaps
    end
    
    Service-->>Service: Return FlatEntityMaps<FlatEntity>
Loading

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@prastoin prastoin changed the title Refactor FlatEntity to be UniversalFlatEntity superset [REQUIRE_FULL_CACHE_FLUSH_WHEN_RELEASED] Refactor FlatEntity to be UniversalFlatEntity superset Jan 28, 2026
@prastoin prastoin changed the title [REQUIRE_FULL_CACHE_FLUSH_WHEN_RELEASED] Refactor FlatEntity to be UniversalFlatEntity superset [REQUIRES_FULL_CACHE_FLUSH_WHEN_RELEASED] Refactor FlatEntity to be UniversalFlatEntity superset Jan 28, 2026
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 87 files

Note: This PR contains a large number of files. cubic only reviews up to 75 files per PR, so some files may not have been reviewed.

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/twenty-server/src/engine/metadata-modules/logic-function/services/workspace-flat-logic-function-map-cache.service.ts">

<violation number="1" location="packages/twenty-server/src/engine/metadata-modules/logic-function/services/workspace-flat-logic-function-map-cache.service.ts:40">
P2: The application lookup omits soft-deleted rows. Since logic functions are loaded with `withDeleted: true` and conversion throws when an application ID is missing, a deleted application will break cache computation. Include `withDeleted: true` to keep the ID→universalIdentifier map complete.</violation>
</file>

<file name="packages/twenty-server/src/engine/metadata-modules/flat-webhook/services/workspace-flat-webhook-map-cache.service.ts">

<violation number="1" location="packages/twenty-server/src/engine/metadata-modules/flat-webhook/services/workspace-flat-webhook-map-cache.service.ts:36">
P2: Include soft-deleted applications in the lookup map to avoid throwing when a webhook references an application that was soft-deleted.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.


return universalIdentifier;
}): string | null => {
return fieldMetadataIdToUniversalIdentifierMap.get(fieldMetadataId) ?? null;
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: For the moment still requiring the many to many junctionFieldId though

Copy link
Copy Markdown
Contributor

@etiennejouan etiennejouan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LsuperGTM 👍


export type DeleteFieldAction =
BaseDeleteWorkspaceMigrationAction<'fieldMetadata'> & {
// Remove this
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For later ? or now ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upcoming PR !

@prastoin prastoin added this pull request to the merge queue Jan 28, 2026
Merged via the queue into main with commit fe9d6f3 Jan 28, 2026
99 of 101 checks passed
@prastoin prastoin deleted the introduce-flat-entity-super-set branch January 28, 2026 16:58
@twenty-eng-sync
Copy link
Copy Markdown

Hey @prastoin! After you've done the QA of your Pull Request, you can mark it as done here. Thank you!

1 similar comment
@twenty-eng-sync
Copy link
Copy Markdown

Hey @prastoin! After you've done the QA of your Pull Request, you can mark it as done here. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants