feat(pushdown): push array_has_any/array_has_all

[RagingKore]

feat(pushdown): push array_has_any/array_has_all into Lance

Investigated, implemented and tested with the help of Claude Code.

Summary

No containment predicate on a LIST column could prefilter a search. LanceExprIRSupportsLogicalType
gated on the column's LogicalTypeId, and rejected LIST before it looked at the function, so
array_has_any and array_has_all never left DuckDB while id > 5, IN and LIKE on scalar columns
pushed normally. DuckDB applied the containment predicate after the search had already run, so callers had
to oversample k until the candidate pool covered the whole table, or the search silently missed matching
rows outside the top-k. prefilter := true neither helped nor errored, it quietly did nothing. Lance can
already answer these predicates from a LABEL_LIST index
(LabelListQueryParser::visit_scalar_function in lance-index); the extension never handed them over.
This PR does that, and fixes three further things that sat between sending the predicate and Lance using
the index.

Before and After

ns is any catalog from ATTACH '<dir>' AS ns (TYPE LANCE). ✅ means the predicate reaches Lance and
Lance filters, ❌ means it never left DuckDB. A scan has no prefilter argument and no top-k, so a missed
pushdown costs time. On a search it changes the answer: the search picks its top-k before applying the
predicate, so you get back fewer rows than you asked for.

Table addressed as	scan	lance_hybrid_search	lance_vector_search	lance_fts
'data/items.lance'	❌ → ✅	❌ → ✅	❌ → ✅	❌ → ✅
ns.main.items	❌ → ✅	❌ → ✅	❌ → ✅	❌ → ✅

Containment worked nowhere, under either way of addressing the table, and the index was unreachable by
construction. With a LABEL_LIST index on the list column the scan ✅ is an index lookup rather than a
full read; without one the predicate still pushes and Lance answers it by scanning. LABEL_LIST is the
only scalar index that answers containment.

The same defects blocked other predicates, so those are fixed by construction rather than as extra scope.
Cells with a single ✅ already worked and are unchanged.

Predicate	Table addressed as	scan	lance_hybrid_search	lance_vector_search	lance_fts
id > 5, =, numeric IN	'data/items.lance'	✅	✅	✅	✅
	ns.main.items	✅	✅	❌ → ✅	❌ → ✅
starts_with / LIKE / regexp	'data/items.lance'	✅	❌ → ✅	✅	❌ → ✅
	ns.main.items	✅	❌ → ✅	❌ → ✅	❌ → ✅
IS NULL on a list column	'data/items.lance'	❌ → ✅	❌ → ✅	❌ → ✅	❌ → ✅
	ns.main.items	❌ → ✅	❌ → ✅	❌ → ✅	❌ → ✅

The first two predicate groups resolve to a BTREE index when one exists on the compared column. No scalar
index answers IS NULL on a list column, so that row always scans.

One case this PR does not fix: IN over a VARCHAR column with two or more values throws
requires filter pushdown for prefilterable columns from all three search functions, before and after.
The same predicate pushes on a scan, and pushes from a search under prefilter := false, so the encoder
handles it; the prefilterable check rejects it. CONJUNCTION_OR over VARCHAR fails the same way.
Out of scope here, but worth an issue.

Design

LanceExprIRSupportsLogicalType rejected LogicalTypeId::LIST, so any filter touching a list column
bailed out before anything examined the function name, and the IR had no list literal. This admits LIST
to the type gate, adds a LIST literal tag (a length-prefixed sequence of the existing scalar payloads,
recursive), and gives containment branches to both the TableFilterSet and Expression builders. Rust
decodes the tag into a ScalarValue::List and resolves the two UDFs from datafusion-functions-nested,
promoted from a transitive dependency to an explicit one. The IR always carries the DataFusion spelling, so
DuckDB's list_has_any / list_has_all aliases stay decoupled from it.

Search Functions

lance_fts and lance_hybrid_search registered filter_pushdown and pushdown_expression but never
pushdown_complex_filter, so simple comparisons pushed while the optimizer dropped anything DuckDB models
as an expression. LancePushdownComplexFilter is typed against LanceKnnBindData, so this extracts its
body into CollectLancePushedFilterIRParts and gives both functions a hook over the LanceSearchBindData
they already share. This is wider than containment: starts_with, LIKE and regexp_matches have never
prefiltered from those two functions either, and now do.

Namespace-Backed Tables

lance_vector_search and lance_fts rejected prefilter := true on an attached namespace table without
an explicit filter := '<string>'. That guard was correct when #217 added it: query_table takes a filter
string, DuckDB supplies an IR, and no converter existed. #232 then added filter_expr_to_sql for namespace
scans, but nothing pointed the search path at it. The two search entry points now take the same
filter_ir arguments as lance_create_namespace_scan_stream_ir and share its machinery via
apply_filter_expr; a pushed predicate and an explicit filter combine with AND. The init-time check the
path-backed code already used replaces the bind-time guard. The effect is wider than the guard suggests: no pushed
predicate of any kind previously reached a namespace-backed search, and prefilter := true without an
explicit filter threw outright rather than degrading.

List Field Naming

DuckDB's Arrow exporter hardcodes the list child field name to l, while Arrow, pyarrow, Lance and
DataFusion use item, and Lance persists whatever it receives. DataFusion's array-signature coercion
normalizes every list argument to a child named item, so an l column gets wrapped in a cast and Lance's
maybe_indexed_column no longer matches it. The predicate pushed and prefiltered, but Lance answered it
with a scan:

full_filter=array_has_any(CAST(tags AS List(Utf8)), List([red])), refine_filter=array_has_any(…)

LanceNormalizeArrowListFieldNames rewrites the name to item at every write boundary: COPY, INSERT,
CREATE TABLE, CTAS, namespace create, and both ALTER TABLE paths. The plan then reads
refine_filter=-- with a ScalarIndexQuery … @tags_idx(LabelList).

Compatibility

Renaming on write broke appends to older datasets, since Lance rejects a schema that disagrees with the
destination. The writer now reads the destination schema in Append mode and matches whatever that dataset
already uses, realigning the imported data_type alongside it. New datasets get item, existing ones keep
l.

Two behaviour changes. lance_fts and lance_hybrid_search now genuinely prefilter, which changes results
for anyone who set prefilter := true and was silently getting post-filtering. And datasets written before
this change keep l and will not use the index; they read, append and prefilter correctly, Lance just
answers with a scan. One metadata-only rename fixes a dataset, with no data rewrite:
ALTER TABLE ns.main.t RENAME COLUMN "tags.l" TO item.

Deliberate Exclusions

Empty needles and NULL needle elements fall back to DuckDB, because DuckDB short-circuits an empty
list_has_all to true and ignores NULL elements where DataFusion compares them by value. A non-constant
second argument falls back too. Also excluded: the singular array_has / array_contains, and the && /
@> / <@ aliases, since DuckDB registers <@ without swapping its arguments, unlike Postgres. Only the
containment branches emit the LIST tag, keeping tags = ['a','b'] off a DataFusion list equality whose
nested comparison kernels are patchy. Admitting LIST to the type gate does push down IS NULL and
IS NOT NULL on list columns, which matches Arrow semantics.

Two scope notes on the index, found while testing and left alone. The LABEL_LIST index is reached for
VARCHAR[] and BIGINT[] element types. Narrower integer widths encode through the same I64 literal
tag, so DataFusion coerces the column instead of the literal and the planner falls back to a scan; the
predicate still pushes and still prefilters, it is only the index lookup that is lost. An element-type hint
on the LIST tag would close it. Separately, align_list_field_names descends into struct children but
not into nested lists, so a legacy List<List<T>> column would still fail to append.

Test Plan

• pushdown_filter_ir_containment.test: both functions and both DuckDB spellings across plain scans,
  lance_vector_search, lance_fts and lance_hybrid_search; VARCHAR[] and BIGINT[] elements; bound
  parameters; a k-semantics probe; index usage asserted through explain_verbose; every negative case
  checked for correct results and for not being pushed.
• pushdown_filter_ir_containment_namespace.test: the same ground against an attached catalog. The fixture
  makes the tagged rows the worst matches, so at k := 2 a search that filters after top-k returns
  nothing, and every assertion proves prefiltering from its row set alone.
• Two tests in search_functions.test asserted the removed bind-time error as the contract, and now assert
  that a WHERE clause prefilters without an explicit filter.
• Four Rust unit tests cover the append schema alignment directly.
• Some assertions check the filter IR byte count rather than a result set, because the failure mode is
  silent: a plan that stops pushing still returns the right rows, from a full scan.
• Full regression suite: 4250 assertions across 53 test cases. cargo clippy --all-targets clean.

Docs

docs/sql.md search filter semantics updated to match: pushed WHERE predicates and an explicit
filter combine with AND, and a new "Filter pushdown notes" section covers containment predicates,
the LABEL_LIST index, and the legacy l field-name remedy.

Note on Cargo.lock

The lock diff sits in its own commit and is unrelated. Cargo.toml on main already requires
datafusion = "54.0.0" while the committed lock pins 53.1.0, so --locked fails on main today and any
build regenerates it. This feature's own dependency change is one line of Cargo.toml.

[RagingKore]

cc @Xuanwo — this removes the bind-time guard from #217 by pointing the search path at the filter_expr_to_sql converter you added in #232; a pushed predicate and an explicit filter now combine with AND.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Search filtering must preserve one contract across path- and namespace-backed execution: required prefilters must be complete before top-k, while optional pushdown must fall back to DuckDB without failing the query. This revision violates both sides of that contract.

A viable revision would make complex-filter collection report completeness and use one stream-creation policy across dataset and namespace searches: reject unencodable predicates when prefilter=true, and retry without generated IR when prefilter=false while preserving explicit user filters.

[lance-gatekeeper]

Failure to encode this predicate is silently ignored. With prefilter=true, top-k is consequently computed before the missing predicate and DuckDB only filters the truncated result, producing incorrect output. Return completeness from this collector and reject required prefiltering whenever any predicate cannot be encoded; silent skipping is valid only for prefilter=false.

Executed reproducer

COPY (
  SELECT * FROM (VALUES
    (1::BIGINT, [0.0,0.0]::FLOAT[2], NULL::VARCHAR[]),
    (2::BIGINT, [1.0,0.0]::FLOAT[2], ['x']::VARCHAR[])
  ) t(id,vec,tags)
) TO 'test/.tmp/pr237_empty_needle_prefilter.lance'
  (FORMAT lance, MODE 'overwrite');

SELECT id
FROM lance_vector_search(
  'test/.tmp/pr237_empty_needle_prefilter.lance',
  'vec', [0.0,0.0]::FLOAT[2],
  k := 1, prefilter := true, use_index := false
)
WHERE array_has_all(tags, []::VARCHAR[]);

Expected 2; observed zero rows.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The two previously reported filtering defects are resolved: required filters are now checked for completeness, and optional namespace pushdown retries correctly.

The new rejection path still needs a safe lifecycle boundary. Path-backed searches open a Lance session before the complex-filter callback throws; at process teardown that session closes after Tokio's context is gone, causing a panic and leaving DuckDB hung. Reject before acquiring that session, or make session cleanup safe after runtime teardown.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The revision now preserves the search-filter contract across path- and namespace-backed execution: required predicates are complete or rejected, optional pushdown falls back without losing explicit filters, and rejected path-backed searches tear down cleanly.

The teardown guard is narrowly limited to the Tokio destroyed-thread-local state; normal session close still clears caches through the initialized runtime.