perf(exec): parallelize lance exec scan across fragments

[ringfa11]

Parallelize the Lance exec scan by splitting it into fragment partitions.

What

• Carry the DuckDB thread budget into the Lance exec path via a new LanceExecContext (repr(C) struct passed through the exec FFI, null-compatible — existing callers that pass NULL keep the previous single-partition behavior).
• Split the scan into min(threads, fragment_count) fragment partitions (round-robin) so scan+filter parallelize across cores.
• Scope each partition scan to its fragment subset with with_fragments; keep a single-partition full scan as fallback.

Why

Previously the exec scan ran as a single partition on one core. For large multi-fragment datasets, scan+filter now parallelize across the DuckDB thread budget.

Verification

• TPC-H SF10: Q6/Q1 results identical vs single partition; Q6 ~2x faster at threads=4.
• Confirmed __LANCE_EXEC pushdown via EXPLAIN (FORMAT JSON).

[ringfa11]

Benchmark results (A/B)

Validated against TPC-H SF10 (lineitem = 59,986,052 rows, 58 fragments) on a 64-core x86_64 Linux box.

Version	Code base	Scan shape
A (baseline)	upstream exec.rs (single partition)	always 1 partition
B (this PR)	this branch	min(threads, fragment_count) fragment partitions

Method: each build deployed independently, same .local data; queries warmed 2× then timed 5×, median reported. threads set per-session via DuckDB threads config.

Head-to-head at threads = 4

Query	A (single partition)	B (fragmented)	Speedup
Q6	0.953 s	0.303 s	3.14×
Q1	0.805 s	0.406 s	1.98×

Threads sweep (median seconds)

Q6 (scan+filter dominated):

threads	A	B	B/A
1	0.879	0.851	1.03×
2	0.901	0.502	1.79×
4	0.881	0.301	2.93×
8	0.893	0.208	4.29×
16	0.892	0.177	5.04×

Q1 (aggregate dominated):

threads	A	B	B/A
1	0.775	0.803	0.97×
2	0.781	0.508	1.54×
4	0.788	0.411	1.92×
8	0.791	0.393	2.01×
16	0.825	0.388	2.13×

Key observations:

• A is a flat line: adding threads gives zero speedup (~0.88–0.90 s Q6 / ~0.78–0.83 s Q1) — the single-partition scan is the hard bottleneck, as expected.
• B scales with threads: Q6 accelerates monotonically 0.851 → 0.177 s (5.04× at 16 threads); Q1 saturates at ~2× once the aggregate dominates the plan.
• Thread-driven partitioning beats a static num_cpus split: at 58 partitions Q1 was slower than single-partition (over-partitioning); following DuckDB's thread budget (4 partitions at threads=4) is optimal for both queries.

Correctness

Compared value-by-value against standard DuckDB reading the same SF10 parquet:

• Q6 revenue: 1230113636.0101 = 1230113636.0101 ✓
• Q1 all 4 group rows (sum/extendedprice/count): identical ✓
• aggregate count/sum/min/max: 59986052 / 1529738036.00 / 1992-01-02 / 104949.50 ✓
• avg(l_discount) differs only in output precision (0.050001 vs DuckDB 0.0500011776...), proven independent of partition count (rechecked at threads 1/8/16) — a pre-existing lance exec DECIMAL avg behavior, not introduced here.

No regression in multi-fragment partitioning; results identical vs single partition.

[ringfa11]

Hi @Xuanwo — your feat: exec pushdown with global aggregates (#124) introduced the unconditional LanceExecPushdown optimizer rewrite. This PR is a follow-up that addresses the single-partition bottleneck on the pushdown path it registers, and we'd love your eyes on it since you're most familiar with that design.

Context: When __LANCE_EXEC pushdown triggers, the Lance exec scan (exec.rs) hardcodes a single partition, so scan+filter runs on one core. For large multi-fragment datasets (e.g. TPC-H SF100) this regresses Q1/Q6 heavily (~15× in our environment). The fix carries DuckDB's thread budget into the exec path via a null-compatible LanceExecContext and splits the scan into min(threads, fragment_count) fragment partitions (round-robin, with_fragments).

Key question for you: the pushdown is registered unconditionally with no cost gate or hint. Since the exec path now parallelizes correctly, does that change your thinking on whether the unconditional rewrite is acceptable, or should a cost threshold still be considered? Would be great to get your read on whether this fix aligns with how you intended the pushdown path to scale.

Benchmark + correctness details are in the comment above. Thanks for taking a look!