Filesystem `layout="{table_name}"` makes `replace` delete prefix-named sibling tables

[mattfaltyn]

dlt version

1.29.0 source checkout from current devel at 91403f1a39058230361820de414b718a22d7a423.

Describe the problem

When the filesystem destination uses layout="{table_name}", replacing one table can silently delete an unrelated table whose name starts with the same prefix.

For example, after loading tables named event and events, replacing only event deletes the physical data file for events. The load completes successfully and rewrites event, so there is no failure or warning that unrelated data was removed.

The current write path automatically appends .<full_extension> when {ext} is omitted, producing files such as:

event.jsonl.gz
events.jsonl.gz

However, table-prefix extraction returns the bare prefix event. Replace truncation selects files with filepath.startswith(prefix), so both files are selected and deleted.

I reproduced this twice from a clean detached checkout of current upstream devel; both runs exited successfully with identical data loss.

Expected behavior

Replacing event should update only event.jsonl.gz; events.jsonl.gz should remain present and unchanged.

If an end-of-layout {table_name} cannot be handled safely, the destination should reject the layout during validation instead of accepting it and performing an overbroad delete.

Steps to reproduce

import tempfile
from pathlib import Path

import dlt

root = Path(tempfile.mkdtemp(prefix="dlt-fs-prefix-repro-"))
bucket = root / "bucket"

pipeline = dlt.pipeline(
    pipeline_name="prefix_repro",
    pipelines_dir=str(root / "pipelines"),
    destination=dlt.destinations.filesystem(
        bucket_url=str(bucket),
        layout="{table_name}",
    ),
    dataset_name="ds",
)

pipeline.run(
    dlt.resource([{"id": 1}], name="event"),
    loader_file_format="jsonl",
)
pipeline.run(
    dlt.resource([{"id": 2}], name="events"),
    loader_file_format="jsonl",
)

dataset = bucket / "ds"

def table_files():
    return sorted(path.name for path in dataset.glob("event*"))

print("before:", table_files())

pipeline.run(
    dlt.resource(
        [{"id": 3}],
        name="event",
        write_disposition="replace",
    ),
    loader_file_format="jsonl",
)

print("after:", table_files())

Actual output on current devel:

before: ['event.jsonl.gz', 'events.jsonl.gz']
after: ['event.jsonl.gz']

A control run changing only the layout to "{table_name}.{ext}" preserves both files:

before: ['event.jsonl.gz', 'events.jsonl.gz']
after: ['event.jsonl.gz', 'events.jsonl.gz']

Operating system

macOS

Runtime environment

Local

Python version

3.13

dlt data source

Inline Python resources containing dictionaries.

dlt destination

Filesystem & buckets

Other deployment details

The reproduction uses a local temporary filesystem and requires no credentials or external services. The affected prefix/deletion logic is shared by filesystem backends.

Additional information

The root cause appears to span these paths:

• create_path appends a dot and the full extension when {ext} is omitted.
• get_table_prefix_layout intends to retain the separator after {table_name}, specifically to avoid selecting similarly named tables, but returns bare {table_name} when the placeholder terminates the layout.
• _delete_table_files deletes the files returned by prefix listing.
• list_files_with_prefixes uses raw filepath.startswith(prefix).

The filesystem layout documentation says omitted extensions are automatically appended with a dot and that replace-compatible layouts need a separator after {table_name}.

A focused fix could make get_table_prefix_layout("{table_name}") derive "{table_name}.", matching the path produced by create_path. A stricter alternative is to reject the layout before loading.

Suggested regression coverage:

• Unit assertion for the exact end-of-layout case in tests/destinations/test_path_utils.py.
• Filesystem replace test with sibling tables event and events.

Focused validation on the clean upstream checkout:

tests/destinations/test_path_utils.py: 70 passed
local-only tests/load/filesystem/test_filesystem_client.py: 53 passed, 1 skipped

[AnnasMazhar]

Proposed fix

Root cause: get_table_prefix_layout("{table_name}") returns "{table_name}" (bare name, no trailing separator), but create_path produces "{table_name}.jsonl.gz" (appending .<ext> when {ext} is absent). When _delete_table_files calls list_files_with_prefixes which uses filepath.startswith(prefix), prefix "event" matches both event.jsonl.gz and events.jsonl.gz.

Fix: Append a dot separator when {ext} is omitted from layout

In get_table_prefix_layout(), after computing prefix, add:

# When layout ends with {table_name} and {ext} is absent, add a dot
# to match the separator that create_path appends before the extension
if prefix.endswith("{table_name}") and "ext" not in placeholders:
    prefix += "."

This makes get_table_prefix_layout("{table_name}") return "{table_name}.", which startswith matches exactly one file per table.

Test coverage:

1. Unit assertion in tests/destinations/test_path_utils.py for the end-of-layout case
2. Filesystem replace test with sibling tables event and events (per the reproducer in the issue)

Edge cases verified:

• Layouts with explicit {ext} ("{table_name}.{ext}") — no change, ext IS in placeholders, dot already in prefix
• Layouts with trailing path separators ("{table_name}/") — no change, prefix ends with /, not {table_name}
• Layouts with {schema_name}/{table_name} — no change, prefix includes the separator after table_name
• Layouts where table_needs_own_folder=True — no change, the {table_name}/ check catches it earlier

Happy to implement — just wanted alignment on the approach before coding.