fix(delete): correct predicate translation and identifier quoting - #245
fix(delete): correct predicate translation and identifier quoting#245qiuyuhang wants to merge 2 commits into
Conversation
DELETE built its Lance filter IR from get->table_filters via BuildLanceTableFilterIRParts, which treats the filter key as a scan-projection index and remaps it through column_ids. During DELETE the keys are already physical column ids, so any predicate on a non-first column was translated against the wrong column, deleting incorrect rows. Wire DELETE to TryBuildLanceTableFilterIRParts, which consumes table_filters keyed by physical column id directly. Skip OPTIONAL_FILTER / DYNAMIC_FILTER entries: they are pruning hints pushed as PUSHED_DOWN_PARTIALLY while the exact predicate is retained as a residual LogicalFilter (collected separately), so translating them as hard delete predicates is unnecessary and breaks cases like string IN lists.
The delete FFI turns the filter IR into a SQL string via expr_to_sql and feeds it back to Lance's string-based delete API. The default unparser dialect never quotes identifiers, so a column whose name is a SQL keyword (name, value, key, desc, table, order, ...) was emitted bare and the re-parsed predicate silently matched no rows, deleting nothing. Unparse with a CustomDialect that always backtick-quotes identifiers (Lance's filter parser uses backtick quoting; double quotes would be treated as string literals). Applies to both delete FFI entry points.
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The physical-column mapping and Lance-dialect identifier quoting address the two demonstrated silent failures, but pruning hints are skipped only at the top level, so a valid mixed-predicate DELETE still fails.
Please distinguish encoded, safely skipped, and unsupported filter nodes so AND translation can retain required children while omitting optional pruning hints, and add an end-to-end regression for that mixed case.
| // for correctness: DuckDB pushes them as PUSHED_DOWN_PARTIALLY and keeps | ||
| // the exact predicate as a residual LogicalFilter (collected separately for | ||
| // DELETE). Skip them here rather than failing translation. | ||
| if (filter->filter_type == TableFilterType::OPTIONAL_FILTER || |
There was a problem hiding this comment.
This skips optional filters only when the optional node is the top-level table-filter entry. DuckDB can instead produce CONJUNCTION_AND(required comparison, OPTIONAL_FILTER(IN ...)); recursive translation reaches the optional child, returns false, and rejects a valid DELETE. Use a tri-state result (encoded / skip / unsupported) so AND can omit safely optional children while preserving required conjuncts, while unsupported required nodes still fail closed.
Reproducer run against this head
ATTACH 'mixed_filter_repro.lance' AS ns (TYPE LANCE);
CREATE TABLE ns.main.t(s VARCHAR);
INSERT INTO ns.main.t VALUES ('a'), ('b'), ('c'), ('d');
DELETE FROM ns.main.t WHERE s >= 'c' AND s IN ('b', 'd');
SELECT * FROM ns.main.t ORDER BY s;Expected: DELETE succeeds and the query returns a, b, c.
Observed: DELETE reports Not implemented Error: unsupported DELETE predicate for Lance: pushed-down table filter could not be translated to Lance filter IR, the debug CLI exits 134 during Tokio session cleanup, and a fresh read returns all four rows.
Two bugs in DELETE predicate handling, both verified on main. Either one can
silently delete the wrong data (or none at all).
1. Pushed-down table filters were translated against the wrong column
DELETE built its Lance filter IR from
get->table_filtersviaBuildLanceTableFilterIRParts, which treats the filter key as ascan-projection index and remaps it through
column_ids. During DELETE thekeys are already physical column ids, so a predicate on any column other
than the first was translated against the wrong column:
Fix: wire DELETE to TryBuildLanceTableFilterIRParts, which consumes
table_filters keyed by physical column id directly. Also skip
OPTIONAL_FILTER / DYNAMIC_FILTER entries: they are pruning hints pushed
as PUSHED_DOWN_PARTIALLY while the exact predicate remains in the plan as
a residual LogicalFilter (collected separately), so translating them into
hard delete predicates is unnecessary and breaks cases like string IN
lists.
2. Unquoted identifiers broke predicates on SQL-keyword column names
The delete FFI turns the filter IR into SQL via expr_to_sql and passes it
to Lance's string-based delete API. The default unparser dialect never
quotes identifiers, so a column whose name is a SQL keyword (name,
value, key, desc, ...) is emitted bare and the re-parsed predicate
matches nothing — the DELETE succeeds but removes zero rows:
Fix: unparse with a CustomDialect that always backtick-quotes identifiers.
Lance's filter parser uses backticks for identifier quoting; double quotes
would be read as string literals. Applied to both delete FFI entry points.