Skip to content

fix(delete): correct predicate translation and identifier quoting - #245

Open
qiuyuhang wants to merge 2 commits into
lance-format:mainfrom
qiuyuhang:fix-delete-predicate-handling
Open

fix(delete): correct predicate translation and identifier quoting#245
qiuyuhang wants to merge 2 commits into
lance-format:mainfrom
qiuyuhang:fix-delete-predicate-handling

Conversation

@qiuyuhang

Copy link
Copy Markdown

Two bugs in DELETE predicate handling, both verified on main. Either one can
silently delete the wrong data (or none at all).

1. Pushed-down table filters were translated against the wrong column

DELETE built its Lance filter IR from get->table_filters via
BuildLanceTableFilterIRParts, which treats the filter key as a
scan-projection index and remaps it through column_ids. During DELETE the
keys are already physical column ids, so a predicate on any column other
than the first was translated against the wrong column:

ATTACH 'repro' AS ns (TYPE LANCE);
CREATE TABLE ns.main.t(id INT, name VARCHAR, flag INT);
INSERT INTO ns.main.t VALUES (1,'a',0),(2,'b',1),(3,'c',0),(4,'d',1);
DELETE FROM ns.main.t WHERE flag = 1;
SELECT * FROM ns.main.t ORDER BY id;
-- expected: ids 1, 3
-- actual: empty table, all four rows deleted

Fix: wire DELETE to TryBuildLanceTableFilterIRParts, which consumes
table_filters keyed by physical column id directly. Also skip
OPTIONAL_FILTER / DYNAMIC_FILTER entries: they are pruning hints pushed
as PUSHED_DOWN_PARTIALLY while the exact predicate remains in the plan as
a residual LogicalFilter (collected separately), so translating them into
hard delete predicates is unnecessary and breaks cases like string IN
lists.

2. Unquoted identifiers broke predicates on SQL-keyword column names

The delete FFI turns the filter IR into SQL via expr_to_sql and passes it
to Lance's string-based delete API. The default unparser dialect never
quotes identifiers, so a column whose name is a SQL keyword (name,
value, key, desc, ...) is emitted bare and the re-parsed predicate
matches nothing — the DELETE succeeds but removes zero rows:

ATTACH 'repro' AS ns (TYPE LANCE);
CREATE TABLE ns.main.k(name VARCHAR, id INT);
INSERT INTO ns.main.k VALUES ('x',1),('y',2);
DELETE FROM ns.main.k WHERE name = 'x';
SELECT * FROM ns.main.k ORDER BY id;
-- expected: only ('y', 2)
-- actual: both rows still present

Fix: unparse with a CustomDialect that always backtick-quotes identifiers.
Lance's filter parser uses backticks for identifier quoting; double quotes
would be read as string literals. Applied to both delete FFI entry points.

Yuhang Qiu added 2 commits August 25, 2026 04:06
DELETE built its Lance filter IR from get->table_filters via
BuildLanceTableFilterIRParts, which treats the filter key as a
scan-projection index and remaps it through column_ids. During DELETE the
keys are already physical column ids, so any predicate on a non-first
column was translated against the wrong column, deleting incorrect rows.

Wire DELETE to TryBuildLanceTableFilterIRParts, which consumes
table_filters keyed by physical column id directly. Skip
OPTIONAL_FILTER / DYNAMIC_FILTER entries: they are pruning hints pushed as
PUSHED_DOWN_PARTIALLY while the exact predicate is retained as a residual
LogicalFilter (collected separately), so translating them as hard delete
predicates is unnecessary and breaks cases like string IN lists.
The delete FFI turns the filter IR into a SQL string via expr_to_sql and
feeds it back to Lance's string-based delete API. The default unparser
dialect never quotes identifiers, so a column whose name is a SQL keyword
(name, value, key, desc, table, order, ...) was emitted bare and the
re-parsed predicate silently matched no rows, deleting nothing.

Unparse with a CustomDialect that always backtick-quotes identifiers
(Lance's filter parser uses backtick quoting; double quotes would be
treated as string literals). Applies to both delete FFI entry points.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: request changes.

The physical-column mapping and Lance-dialect identifier quoting address the two demonstrated silent failures, but pruning hints are skipped only at the top level, so a valid mixed-predicate DELETE still fails.

Please distinguish encoded, safely skipped, and unsupported filter nodes so AND translation can retain required children while omitting optional pruning hints, and add an end-to-end regression for that mixed case.

Comment thread src/lance_filter_ir.cpp
// for correctness: DuckDB pushes them as PUSHED_DOWN_PARTIALLY and keeps
// the exact predicate as a residual LogicalFilter (collected separately for
// DELETE). Skip them here rather than failing translation.
if (filter->filter_type == TableFilterType::OPTIONAL_FILTER ||

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skips optional filters only when the optional node is the top-level table-filter entry. DuckDB can instead produce CONJUNCTION_AND(required comparison, OPTIONAL_FILTER(IN ...)); recursive translation reaches the optional child, returns false, and rejects a valid DELETE. Use a tri-state result (encoded / skip / unsupported) so AND can omit safely optional children while preserving required conjuncts, while unsupported required nodes still fail closed.

Reproducer run against this head
ATTACH 'mixed_filter_repro.lance' AS ns (TYPE LANCE);
CREATE TABLE ns.main.t(s VARCHAR);
INSERT INTO ns.main.t VALUES ('a'), ('b'), ('c'), ('d');
DELETE FROM ns.main.t WHERE s >= 'c' AND s IN ('b', 'd');
SELECT * FROM ns.main.t ORDER BY s;

Expected: DELETE succeeds and the query returns a, b, c.
Observed: DELETE reports Not implemented Error: unsupported DELETE predicate for Lance: pushed-down table filter could not be translated to Lance filter IR, the debug CLI exits 134 during Tokio session cleanup, and a fresh read returns all four rows.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-changes Latest Gatekeeper recommendation requests changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant