Skip to content

feat: add Snowflake privatelink host injection and fix BigQuery preflight check#145

Open
MiroCillik wants to merge 6 commits intomasterfrom
miro-dbt-snowflake-insecure
Open

feat: add Snowflake privatelink host injection and fix BigQuery preflight check#145
MiroCillik wants to merge 6 commits intomasterfrom
miro-dbt-snowflake-insecure

Conversation

@MiroCillik
Copy link
Copy Markdown
Member

@MiroCillik MiroCillik commented Mar 24, 2026

AJDA-2149

Follow up to #132

Summary

  • Inject Snowflake host parameter into profiles.yml only for privatelink connections, where the hostname differs from what dbt infers from the account name
  • Fix BigQuery dataset preflight check: use dataset->update() (write operation) instead of dataset->reload() (read-only) to properly verify write permissions before dbt runs CREATE SCHEMA IF NOT EXISTS
  • Widen DbtProfilesYaml::dumpYaml() to accept additionalOptions injected into every output after merge

Test plan

  • Unit tests pass for DbtYamlCreateTest (profiles with host for privatelink URLs)
  • Unit tests pass for LocalBigQueryProviderTest (dataset write permission check)
  • CI tests-in-kbc passes (BigQuery jobs no longer fail with 403)
  • Verify Snowflake privatelink connections work with injected host in profiles.yml

Release Notes

  • Justification

    • Snowflake Private Link connections fail because dbt derives the hostname from the account name, which can produce incorrect hostnames (e.g. when the host contains underscores). The host parameter override is needed only for privatelink — regular Snowflake connections work fine with just the account.
    • BigQuery preflight check was using a read-only API call (datasets.get) to verify eventual consistency of IAM permissions, but dbt needs write permission (datasets.create). GCP IAM propagates read permissions faster than write, so the check passed but dbt still failed with 403.
  • Plans for Customer Communication

    • No customer communication needed. Host injection is automatic for privatelink URLs. BigQuery fix is transparent.
  • Impact Analysis

    • Low risk. Host is only injected when the Snowflake URL contains "privatelink" — no change for regular connections. BigQuery fix changes the preflight check from a read to a no-op write, matching what dbt actually needs.
  • Deployment Plan

    • Continuous deployment. No special rollout needed.
  • Rollback Plan

    • Fully reversible. Revert/redeploy previous version.
  • Post-Release Support Plan

    • No post-release support needed.

🤖 Generated with Claude Code

@MiroCillik MiroCillik force-pushed the miro-dbt-snowflake-insecure branch from b4b9a1e to e9d13a3 Compare March 31, 2026 12:47
…e remoteDwh

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MiroCillik MiroCillik force-pushed the miro-dbt-snowflake-insecure branch from e9d13a3 to fa1ccc3 Compare April 13, 2026 13:07
MiroCillik and others added 3 commits April 14, 2026 14:45
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dataset->reload() only tests read permission (bigquery.datasets.get),
but dbt needs write permission (bigquery.datasets.create) for
CREATE SCHEMA IF NOT EXISTS. GCP IAM propagates read permissions faster
than write, so the check passed but dbt still failed with 403.

Replaced with dataset->update([]) which requires bigquery.datasets.update
— same IAM role as datasets.create.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The dbt host parameter is only needed for privatelink connections where
the actual hostname differs from what dbt infers from the account name.
For regular Snowflake URLs, account alone is sufficient.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MiroCillik MiroCillik changed the title feat: add sfInsecureMode and host injection for Snowflake remoteDwh feat: add Snowflake privatelink host injection and fix BigQuery preflight check Apr 20, 2026
try {
$retryProxy->call(function () use ($dataset): void {
$dataset->reload();
$dataset->update([]);
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tohle je jeste fix predesleho PR na retry u BQ. Ten "reload" totiz overiril jen "read" a ty "write" permissions se propaguji pomaleji, takze ten retry pak nefungoval spravne.

MiroCillik and others added 2 commits April 20, 2026 10:22
The previous check (dataset->update) tested a dataset-level permission
(bigquery.datasets.update), but dbt's CREATE SCHEMA IF NOT EXISTS
requires the project-level bigquery.datasets.create permission. These
are different IAM bindings that propagate independently.

Now creates a temporary probe dataset and deletes it, testing the exact
permission dbt needs. A 409 (already exists) also proves the permission.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant