feat: Enforce backend-owned schema model for Lance directory and REST catalogs by mikewhb · Pull Request #243 · lance-format/lance-duckdb

mikewhb · 2026-08-13T12:29:21Z

Background

DuckDB treats schemas created under an attached catalog as schemas owned by that catalog. For example, after attaching PostgreSQL, CREATE SCHEMA pg.sales creates a PostgreSQL schema, and
tables created under pg.sales are stored in PostgreSQL.

Legacy Lance directory namespaces behaved differently. Because they run with namespace manifests disabled, CREATE SCHEMA lance.sales created only a transient in-memory DuckDB schema. It was
not represented in Lance, disappeared after reattachment, and made table ownership and storage routing ambiguous.

This PR makes the Lance catalog follow a clear backend-owned schema model.

Changes

Legacy directory namespaces, including local and object-storage roots, continue to map their root to main.
CREATE SCHEMA is rejected for legacy directory namespaces because manifest-disabled directories cannot persist child namespaces.
REST child namespaces are exposed as DuckDB schemas and restored on ATTACH.
REST CREATE SCHEMA and DROP SCHEMA reuse Lance namespace APIs while preserving DuckDB conflict and dependency semantics.
REST CTAS uses its bound schema, so tables are created in the corresponding child namespace instead of the attachment root.
After a successful CTAS commit, only the target schema's lazy table discovery is invalidated, making new tables visible to existing connections.

REST services that do not support child-namespace discovery can still attach in main-only mode. Remote DDL that cannot participate in DuckDB rollback is rejected inside explicit transactions.

The existing REST delimiter contract and connection-local dataset cache behavior remain unchanged. Cross-connection dataset freshness and CREATE OR REPLACE handling are separate concerns.

Validation

Regression coverage includes:

legacy directory schema rejection;
directory and REST CTAS visibility;
REST child-schema lifecycle and CTAS routing;
schema conflict, dependency, and transaction behavior;
existing dataset cache reuse and precise invalidation.

Legacy directory attachments disable Lance namespace manifests and cannot persist child namespaces. Reject user-created schemas instead of falling back to transient DuckDB schema entries. Preserve the existing behavior that maps main directly to the catalog root.

Reuse Lance namespace list, create, and drop operations to map REST child namespaces to DuckDB schemas. Rebuild schema entries and install their existing REST table generators on ATTACH so schemas persist across connections.

Derive REST table operations from the bound Lance schema entry. Child-schema CTAS now targets its child namespace, while main retains the original namespace. Keep fully qualified table identifiers for namespace mutations and verify routing after reattachment.

Invalidate only the target schema's existing default generator after a directory copy or REST writer finalization succeeds. This makes committed CTAS tables visible across connections without touching the dataset cache or invalidating during planning.

Refresh only the replaced table entry after a successful CTAS commit so existing connections rematerialize its schema without flushing dataset caches. Carry namespace identifiers across the FFI as length-delimited segments and remove leaf-name retries that could redirect qualified operations to another namespace.

Use a database-shared generation map to invalidate only replaced dataset keys while preserving unrelated cross-query cache entries. Keep optional child-namespace discovery from blocking main-only backends, and reject remote schema DDL forms whose transaction or replacement semantics cannot be preserved.

Recognize structured unsupported namespace responses, preserve DuckDB duplicate-schema conflict semantics, and reject REST table DDL before remote side effects inside explicit transactions.

Keep PR 243 focused on namespace routing and catalog discovery. Restore connection-local dataset caching and move the database-wide generation boundary plus replacement freshness coverage to the dedicated cache-coherency work.

Remove the private LID1 FFI format because Lance REST still serializes identifiers with its configured delimiter. Preserve qualified child-namespace routing and fail-closed lookups while retaining the pre-existing REST identifier contract.

Invalidate only the target schema's lazy table generator after a successful CTAS commit. Leave existing table-entry replacement and dataset freshness semantics to the dedicated replacement and cache-coherency work.

mikewhb · 2026-08-14T04:09:32Z

@lance-community "Lance Gatekeeper — Pipeline could not complete" How to deal with it.

Return namespace and table listings through a shared typed string-list FFI instead of lossy newline framing. Reuse the ownership API for directory listings and scalar-index discovery. Normalize full table identifiers relative to the requested namespace, while accepting leaf-only compatibility responses and excluding deeper descendants. Reject appended REST segments containing the active delimiter before remote operations so quoted names cannot silently change hierarchy.

lance-gatekeeper

✅ Gate recommendation: approve.

1 fixed / 0 remain.

The current revision completes the REST identifier boundary: typed list results preserve element boundaries, appended delimiter-bearing segments fail closed, and Lance's full table identifiers are normalized to direct-child leaf names. Schema ownership, qualified CTAS routing, discovery refresh, conflict handling, and explicit-transaction guards remain intact.

mikewhb · 2026-08-14T09:21:33Z

@Xuanwo Could you review the REST multi-schema work on this branch? It maps REST child namespaces to DuckDB schemas and fixes REST CTAS to use the bound schema id. Directory catalogs are unchanged (no CREATE SCHEMA without manifest). Thanks.

mikewhb · 2026-08-15T00:53:15Z

@lance-community Lance Gatekeeper pipeline could not complete on the last run, could you please rerun it?

arrowbowang added 4 commits August 13, 2026 20:27

feat(catalog): expose REST child namespaces

fe6c789

Reuse Lance namespace list, create, and drop operations to map REST child namespaces to DuckDB schemas. Rebuild schema entries and install their existing REST table generators on ATTACH so schemas persist across connections.