fix(compaction): make sure that only one task is running at a time, refactor compaction manager creation and add test by lucasfang · Pull Request #201 · alibaba/paimon-cpp

lucasfang · 2026-03-27T03:27:24Z

Purpose

Fix compaction manager to make sure only one task is running at a time.
Refactor key value table compaction manager creation by adding compaction manager factory.
Add UT for coverage.

Tests

NoopCompactManagerTest
- ShouldIgnoreAddedFilesAndStayIdle
- ShouldRejectUserTriggeredFullCompaction
- ShouldReturnEmptyCompactionResult
- ShouldAllowNoopLifecycleOperations
BitmapDeletionVectorTest
- PositionOutOfRangeShouldFail
- DeserializeShouldRejectInvalidMagicNumber
- DeserializeWithoutMagicNumberShouldRoundTrip
IOManagerTest
- CreateShouldReturnManagerWithGivenTempDir
- GenerateTempFilePathShouldContainPrefixAndSuffix
- GenerateTempFilePathShouldBeDifferentAcrossCalls
ForceUpLevel0CompactionTest
- TestMaxCompactIntervalConfiguration
- TestGentleIntervalShouldForcePickAfterThreshold
KeyValueFileStoreWriteTest
- TestWriteShouldSucceedWhenLookupEnabledWithIOManager
- TestWriteShouldSucceedWhenDefaultCompactRewriterPathEnabled
CoreOptionsTest
- TestPrepareCommitWaitCompaction
MergeTreeCompactManagerFactoryDirectTest
- TestCreateCompactStrategyDefaultUniversal
- TestCreateCompactStrategyLookupRadicalShouldNotSetMaxInterval
- TestCreateCompactStrategyLookupGentleShouldSetMaxInterval
- TestCreateCompactStrategyLookupGentleBehaviorShouldForceAfterThreshold
- TestCreateCompactStrategyLookupRadicalBehaviorShouldForceImmediately
- TestCreateCompactStrategyForceUpLevel0OptionShouldTakeEffect
- TestCreateCompactManagerWriteOnlyShouldReturnNoopCompactManager
- TestWriteShouldFailWhenLookupEnabledButIOManagerMissing
- TestCompactShouldUseNoopCompactManagerWhenWriteOnly
- TestWriteShouldFailForUnsupportedFullCompactionChangelogRewriter
- TestWriteShouldSucceedWhenLookupDeletionVectorEnabledWithIOManager
- TestWriteShouldFailWhenFirstRowWithDeletionVectors
- TestWriteShouldSucceedWhenLookupDeletionVectorEnabledWithSequenceField
- TestWriteShouldSucceedWhenLookupDeletionVectorEnabledWithLookupChangelog

API and Format

Add options for pk compaction.

lookup-wait

Generative AI tooling

Generated-by: GitHub Copilot (GPT-5.3-Codex)

Copilot

Pull request overview

Improves compaction task lifecycle handling to prevent overlapping compaction executions by ensuring outstanding compaction work is joined before starting a new task or closing writers.

Changes:

Make CancelCompaction() synchronously wait for the active compaction future to finish, removing the “detached futures” list.
Update writer close-path comments to reflect “cancel + wait” behavior.
Remove tracking/waiting of previously “cancelled” futures in the compaction future manager.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/paimon/core/mergetree/merge_tree_writer.h	Updates close-path comments to describe cancel + wait behavior.
src/paimon/core/compact/compact_future_manager.h	Changes cancellation semantics to move+wait the active future; removes cancelled futures tracking.
src/paimon/core/append/append_only_writer.cpp	Updates close-path comments to describe cancel + wait behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/paimon/core/compact/compact_future_manager.h

src/paimon/core/mergetree/merge_tree_writer.h

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/paimon/core/operation/key_value_file_store_write_test.cpp

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory_test.cpp

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory.cpp

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/paimon/core/operation/key_value_file_store_write_test.cpp

src/paimon/core/disk/io_manager_test.cpp

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory_test.cpp

src/paimon/core/disk/io_manager_test.cpp

src/paimon/core/compact/compact_future_manager.h

src/paimon/core/operation/key_value_file_store_write_test.cpp

lxy-9602 · 2026-03-30T08:55:04Z

src/paimon/core/operation/key_value_file_store_write_test.cpp

+
+TEST(KeyValueFileStoreWriteTest, TestWriteShouldSucceedWhenLookupEnabledWithIOManager) {
+    auto fields = {arrow::field("f0", arrow::utf8(), /*nullable=*/false)};
+    arrow::Schema typed_schema(fields);


I'm not clear on the purpose of these two tests. If they're intended to test lookup behavior during compaction, then we should at least call PrepareCommit(wait_compaction=true).

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory.h

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory_test.cpp

fix(compaction): make sure that only one task is running at a time.

a90abda

Copilot AI review requested due to automatic review settings March 27, 2026 03:27

Copilot AI reviewed Mar 27, 2026

View reviewed changes

src/paimon/core/compact/compact_future_manager.h Outdated Show resolved Hide resolved

src/paimon/core/compact/compact_future_manager.h Outdated Show resolved Hide resolved

src/paimon/core/mergetree/merge_tree_writer.h Outdated Show resolved Hide resolved

Copilot started reviewing on behalf of lucasfang March 27, 2026 05:50 View session

lucasfang added 3 commits March 27, 2026 16:35

fix

555b931

refactor(compaction): add merge tree compact manager factory

5537bb7

fix

8664647

lucasfang requested a review from Copilot March 27, 2026 10:27

Copilot started reviewing on behalf of lucasfang March 27, 2026 12:33 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

src/paimon/core/operation/key_value_file_store_write_test.cpp Show resolved Hide resolved

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory_test.cpp Show resolved Hide resolved

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory.cpp Show resolved Hide resolved

lucasfang requested a review from Copilot March 30, 2026 01:45

Copilot started reviewing on behalf of lucasfang March 30, 2026 01:45 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

src/paimon/core/operation/key_value_file_store_write_test.cpp Outdated Show resolved Hide resolved

src/paimon/core/disk/io_manager_test.cpp Show resolved Hide resolved

src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory_test.cpp Outdated Show resolved Hide resolved

fix

02a3276

lucasfang force-pushed the compaction_dev branch from 8465a81 to 02a3276 Compare March 30, 2026 06:49

lucasfang changed the title ~~fix(compaction): make sure that only one task is running at a time.~~ fix(compaction): make sure that only one task is running at a time, refactor compaction manager creation and add test Mar 30, 2026