feat(optimizer): rewrite ORDER BY <distance> LIMIT k into Lance KNN scan

[martji]

Summary

Standard SQL SELECT ... ORDER BY array_distance(vec, q) LIMIT k previously fell through DuckDB's LogicalTopN over a full Lance scan and silently ignored any IVF / HNSW
index on the column — the only way to use the index was to switch to the bespoke lance_vector_search table function.

This PR adds an OptimizerExtension that rewrites the standard pattern into an index-backed plan with no user-visible change.

Design

The optimizer matches:

LogicalTopN(limit=k, offset=0, ASC dist)
└── LogicalProjection(... dist_func(vec, const_query) ...)
└── [LogicalFilter ...] // optional
└── LogicalGet(__lance_scan | __lance_table_scan | __lance_namespace_scan)

and replaces the underlying LogicalGet with a new internal __lance_knn_scan, collapsing TopN and any pushable Filter into a single GET. A covering WHERE clause is
encoded as Lance Filter IR so Lance pre-filters inside the index lookup.

Distance functions

SQL function	Metric
array_distance / list_distance	l2
array_cosine_distance / list_cosine_distance	cosine
array_negative_inner_product / list_negative_inner_product	dot

l2 and cosine are commutative; for dot, exactly one side must be a constant FLOAT[N] and the other a column reference.

Metric consistency

A new FFI lance_dataset_vector_index_metric returns the index's metric for a column, cached per-column on LanceDatasetCacheEntry (positive lookups only, so transient
errors don't poison the entry and late CREATE INDEX is picked up on the next plan). The optimizer falls back if the column has no vector index or the metric doesn't match
the distance function.

Session variables

Variable	Default	Purpose
lance_knn_nprobes	0 (Lance default)	IVF probe count
lance_knn_refine_factor	0	PQ refine factor
lance_knn_prefilter	true	Apply filter before / after the scan
lance_knn_use_index	true	Kill switch — false disables the rewrite
lance_knn_max_k	10000 (hard ceiling 1,000,000)	Caps k to bound index scan cost

Fallback

13 conditions keep the original plan unchanged (DESC, OFFSET, multi-ORDER-BY, non-constant query vector, unpushable WHERE, missing index, metric mismatch, k > max_k,
etc.). Correctness is unchanged in every fallback case.

Hardening

• Filter IR dedup — LogicalFilter, LIKE pushdown, and table_filters can each contribute IR fragments; byte-equal fragments are deduplicated so LIKE + KNN don't push
  the same predicate twice.
• Rebind verification — The __lance_knn_scan rebind path (prepared statement / EXPLAIN ANALYZE replan) re-verifies the index and metric. A clear "re-prepare" error
  replaces an opaque downstream failure when an index is dropped across contexts.
• 1 MiB blob cap — Rebound query / filter IR blobs are capped at 1 MiB; a typo'd internal call cannot trigger an eager multi-GB allocation.

Test plan

• test/sql/optimizer_knn_basic.test — rewrite shape; bidirectional EXCEPT against lance_vector_search for all three metrics
• test/sql/optimizer_knn_filter.test — WHERE pushdown, prefilter=true vs false row-count gap, IR-dedup byte window, LIKE + KNN interaction
• test/sql/optimizer_knn_fallback.test — every fallback condition, lance_knn_max_k lowering + 0 = default branch, late CREATE INDEX proving the metric cache only
  stores positive lookups
• Full regression GEN=ninja make test: 4203 assertions / 53 cases pass

[Xuanwo]

I think this rewrite is not semantics-preserving for standard SQL.

ORDER BY array_distance(vec, q) LIMIT k is an exact top-k query in DuckDB today. This PR rewrites it into a Lance KNN scan that may use ANN indices. With IVF/HNSW-style indices, the result depends on search parameters such as nprobes / refine_factor, so it can return a different top-k than the original full scan + TopN plan. The current tests mostly compare the rewritten query against lance_vector_search, and use num_partitions=1, so they prove parity with the existing approximate search path but not equivalence to the SQL query being optimized.

lance_knn_prefilter=false makes the semantic issue explicit: for WHERE ... ORDER BY distance LIMIT k, applying the filter after the global top-k is not equivalent to filtering first and then taking top-k. The tests currently assert this changed behavior, but a transparent optimizer rule cannot change the meaning of the SQL.

I think we need one of these designs instead:

• Keep this rewrite exact-only: only apply it when Lance can prove it will search all candidates needed to preserve exact SQL semantics.
• Or make approximate KNN an explicit SQL surface / opt-in mode, not a default optimizer rewrite for ordinary ORDER BY ... LIMIT.
• At minimum, the optimizer should bail out when filters are present and prefilter=false, since that is known to change results.

[martji]

Thanks for calling this out. I agree that we need to distinguish the exact full-scan TopN path from the vector-index path. DuckDB’s current plan for WHERE -> ORDER BY distance -> LIMIT computes an exact top-k over the filtered input, while an IVF/HNSW-backed vector search may be approximate depending on search parameters.

The design of this PR intentionally follows duckdb-vss: an ORDER BY array_distance(...) LIMIT k query can be optimized into a vector-index-backed scan automatically. This matches the behavior users typically expect from vector search systems and from duckdb-vss: when an index is available, search parameters such as nprobes, refine_factor, and ef_search may trade recall for performance instead of guaranteeing the same result set as a full scan + TopN plan.

That said, the WHERE + lance_knn_prefilter=false case you pointed out is a clear semantic problem for a transparent optimizer rule. It turns “filter first, then top-k” into “global top-k first, then filter”. I changed the optimizer to bail out in this case: when filters are present and lance_knn_prefilter=false, the rewrite is not applied and DuckDB keeps the original SQL plan. If users explicitly want post-filter KNN behavior, they can still use lance_vector_search(..., prefilter=false).

We also keep the global switch lance_knn_use_index. Its default remains true, preserving the automatic vector-index behavior consistent with duckdb-vss and vector search engines. When set to false, the optimizer does not rewrite the query, so DuckDB falls back to the full scan + TopN path. I added test coverage for this by comparing the lance_knn_use_index=false result against a materialized full-scan distance + DuckDB TopN baseline using bidirectional EXCEPT.

The updated policy is:

• Keep KNN optimizer rewrite enabled by default, matching duckdb-vss and vector-search index semantics.
• Provide lance_knn_use_index=false as the exact full-scan TopN fallback.
• Bail out for the known semantics-changing case: filters present with prefilter=false.
• Keep explicit approximate/post-filter search available through lance_vector_search, instead of transparently changing ordinary filtered SQL into post-filter behavior.