Skip to content

[TPC-DS] Fix paired sales/returns table generation #398

Description

@kevinjqliu

This affects only TPC-DS because TPC-H has no sales/returns pair concept.

Follow-up to #354 and #391; part of #218.

Requirement

For each sales/returns pair:

  • Sales only → generate and output only sales.
  • Returns only → generate and output only returns.
  • Both → generate both in one shared pass and output both.

Generated work and output files must exactly match the selected tables.

Issues

1. Selecting one table produces both

Affected formats: DAT and CSV.

Selecting only sales or only returns generates and writes both tables.

2. Selecting both traverses the shared source twice

Affected format: Parquet.

Selecting both tables writes the correct outputs but generates each independently instead of sharing one source pass.

Behavior matrix

Using store_sales and store_returns as the representative pair:

Format Sales only Returns only Both
DAT¹ ❌ Outputs both ❌ Outputs both ✅ One pass, both outputs
CSV ❌ Outputs both ❌ Outputs both ✅ One pass, both outputs
Parquet ✅ Sales only ✅ Returns only ❌ Two source passes

¹ DAT is the TPC-DS equivalent of TBL.

Reproduction

Issue 1: DAT and CSV output both tables

# DAT: each command incorrectly produces both paired files
cargo run -p tpcgen-cli --bin tpcgen-cli tpcds dat --scale-factor=1 --tables=store_sales -o "$(mktemp -d)"
cargo run -p tpcgen-cli --bin tpcgen-cli tpcds dat --scale-factor=1 --tables=store_returns -o "$(mktemp -d)"

# CSV: each command incorrectly produces both paired files
cargo run -p tpcgen-cli --bin tpcgen-cli tpcds csv --scale-factor=1 --tables=store_sales -o "$(mktemp -d)"
cargo run -p tpcgen-cli --bin tpcgen-cli tpcds csv --scale-factor=1 --tables=store_returns -o "$(mktemp -d)"

Selecting both already works:

cargo run -p tpcgen-cli --bin tpcgen-cli tpcds dat --scale-factor=1 --tables=store_sales,store_returns -o "$(mktemp -d)"
cargo run -p tpcgen-cli --bin tpcgen-cli tpcds csv --scale-factor=1 --tables=store_sales,store_returns -o "$(mktemp -d)"

Issue 2: Parquet traverses the shared source twice

Single-table selection works:

cargo run -p tpcgen-cli --bin tpcgen-cli tpcds parquet --scale-factor=1 --tables=store_sales -o "$(mktemp -d)"
cargo run -p tpcgen-cli --bin tpcgen-cli tpcds parquet --scale-factor=1 --tables=store_returns -o "$(mktemp -d)"

Selecting both produces the correct files but traverses the shared source twice:

cargo run -p tpcgen-cli --bin tpcgen-cli tpcds parquet --scale-factor=1 --tables=store_sales,store_returns -o "$(mktemp -d)"

Acceptance criteria

  • DAT/CSV single-table selection generates and writes only the selected table.
  • DAT/CSV paired selection continues to generate both in one pass.
  • Parquet paired selection generates both in one shared pass.
  • Cover store, catalog, and web pairs across every affected format.

Repeated selection is already handled by #391 (TPC-DS) and #392 (TPC-H) and is outside this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions