Fix id collisions between AssignUniqueId operators - #30955
Open
hqbhoho wants to merge 1 commit into
Open
Conversation
hqbhoho
force-pushed
the
fix-assign-unique-id-per-task-pool
branch
2 times, most recently
from
August 31, 2026 10:21
33e185c to
38c28a3
Compare
hqbhoho
force-pushed
the
fix-assign-unique-id-per-task-pool
branch
2 times, most recently
from
August 31, 2026 10:37
279150e to
ce61f4d
Compare
Praveen2112
reviewed
Aug 31, 2026
hqbhoho
force-pushed
the
fix-assign-unique-id-per-task-pool
branch
from
August 31, 2026 11:19
ce61f4d to
950dac7
Compare
hqbhoho
force-pushed
the
fix-assign-unique-id-per-task-pool
branch
from
September 2, 2026 07:56
950dac7 to
86f1a00
Compare
ebyhr
reviewed
Sep 3, 2026
ebyhr
reviewed
Sep 4, 2026
The target and source AssignUniqueId operators of a MERGE can execute in the same task and drew overlapping ids from separate pools, causing false MERGE_TARGET_ROW_MULTIPLE_MATCHES failures.
hqbhoho
force-pushed
the
fix-assign-unique-id-per-task-pool
branch
from
September 4, 2026 06:15
dbe2637 to
39a1b71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
AssignUniqueIdembeds only the stage id and partition id of the task in the id mask, while row ids were drawn from a per-factory pool starting at zero. TwoAssignUniqueIdnodes executing in the same task therefore assign the same ids to different rows.MERGEhits this when the join is colocated with the partitioning of the target table (e.g. a bucket-partitioned Iceberg target withjoin_distribution_type=PARTITIONED): the target scan stays in the join stage, so the target and sourceAssignUniqueIdnodes run in the same task. After the right join coalesces the two id columns,MarkDistinctover(unique_id, case_number)treats unrelated rows as duplicate matches and fails with a falseMERGE_TARGET_ROW_MULTIPLE_MATCHES.The fix shares a single row id pool (
AtomicLong) across allAssignUniqueIdoperator factories in a task, held byLocalExecutionPlanContextand passed to every factory. Ids are then unique within the whole query: the stage/partition mask separates tasks, and the shared pool separates operators within a task. This follows the operator-level direction suggested by @Praveen2112 in the issue, and subsumes the factory-duplicate()pool sharing from commit 0e880c8.There is no execution-time cost: the per-row path is unchanged, and the shared pool is touched once per 2^20 ids per driver (a single
getAndAdd).Validation:
TestAssignUniqueIdOperatorcovers id disjointness for factories sharing a pool and for duplicated factories.TestIssue30639reproduces the issue end to end; it fails on current master withOne MERGE target table row matched more than one source rowand passes with this change. The test pinsbucket_execution_enabledand asserts bothAssignUniqueIdnodes plan into one fragment, so it cannot become vacuous if planner defaults change.TestMergepasses unchanged; the plan shape is not affected.Additional context and related issues
MERGEconsumer by widening theMarkDistinctkey; the regression test here is based on the reproducer by @Harmuth94 from that PR.Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: