Skip to content

feat(lakehouse): support Azure Blob storage for DuckLake#2409

Merged
turtleDev merged 3 commits into
bruin-data:mainfrom
JakeHarveyy:feat/ducklake-azure-storage
Jul 22, 2026
Merged

feat(lakehouse): support Azure Blob storage for DuckLake#2409
turtleDev merged 3 commits into
bruin-data:mainfrom
JakeHarveyy:feat/ducklake-azure-storage

Conversation

@JakeHarveyy

Copy link
Copy Markdown
Contributor

What

Adds Azure Blob Storage as a storage: backend for the DuckDB DuckLake lakehouse connection, alongside the existing s3 and gcs backends.

duckdb:
  - name: my_ducklake
    lakehouse:
      format: ducklake
      catalog:
        type: postgres
        # ...
      storage:
        type: azure
        path: "az://my-container/path"
        auth:
          connection_string: "${AZURE_STORAGE_CONNECTION_STRING}"

Two auth modes:

  • connection_string — account-key auth.
  • account_name only — uses DuckDB's credential_chain provider (managed identity / az login / environment credentials).

Why

DuckLake stores its Parquet on object storage, and DuckDB fully supports Azure Blob via the azure extension — but Bruin's lakehouse connection only accepts s3/gcs, so DuckLake pipelines can't target Azure. This closes that gap for Azure deployments. Follows the same shape as the GCS (#1758) and S3-compatible (#1994/#1999) backends.

How

  • pkg/config/lakehouse.go: StorageTypeAzure + supportedStorageTypes; StorageAuth.ConnectionString/AccountName (+ IsAzure()).
  • pkg/duckdb/lakehouse.go: DuckLake validation for azure, loads the azure extension, generateAzureSecret() (CREATE SECRET (TYPE azure, …)), and the attach flow.
  • generateDuckLakeAttach is unchanged — az://… flows through as DATA_PATH and the secret's SCOPE authorizes it.

Generated SQL for type: azure:

INSTALL azure; LOAD azure;
SET GLOBAL azure_transport_option_type = 'curl';
CREATE OR REPLACE SECRET bruin_<alias>_storage (TYPE azure, CONNECTION_STRING '', SCOPE 'az://…');
-- + catalog secret + ATTACH 'ducklake:postgres:' (DATA_PATH 'az://…', …)

Note on the curl transport

azure_transport_option_type = 'curl' is set globally for azure storage. The azure extension's default transport ignores the system CA bundle and fails TLS to *.blob.core.windows.net on Linux/containers; curl uses libcurl + the standard CA paths (host needs ca-certificates). It's GLOBAL rather than session-scoped because DuckLake maintenance calls otherwise still fail — see duckdb/ducklake#776. Happy to make this configurable (or opt-in) if you'd prefer.

Scope

DuckLake only. Iceberg-on-Azure remains unsupported (untested).

Tests

pkg/duckdb/lakehouse_test.go: config validation (pass / missing-auth / missing-path), required extensions, generateAzureSecret golden strings (connection_string + credential_chain), and the full attach-statement sequence.

Docs

  • docs/platforms/duckdb.md: #### Azure config examples, auth modes, transport/CA note, maintenance-func caveat.
  • docs/getting-started/lakehouse.md: Azure column in the DuckLake support table.

Adds `storage.type: azure` to the DuckDB DuckLake lakehouse connection,
mirroring the existing s3/gcs backends. Two auth modes: an account-key
`connection_string`, or `account_name` alone to use DuckDB's credential
chain (managed identity / az login / env).

Bruin loads the `azure` extension and sets
`azure_transport_option_type = 'curl'` globally so TLS to
*.blob.core.windows.net uses the system CA bundle — required on
Linux/containers. GLOBAL rather than session because DuckLake maintenance
calls otherwise still fail (duckdb/ducklake#776).

Scope: DuckLake only; Iceberg-on-Azure remains unsupported.

Covered by unit tests (validation, required extensions, secret SQL,
attach statements) and documented in the DuckDB platform + lakehouse
getting-started pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

PR author is not in the allowed authors list.

@JakeHarveyy

Copy link
Copy Markdown
Contributor Author

Verified the unit tests pass locally (CI here is awaiting first-time-contributor approval):

go test -tags="no_duckdb_arrow" ./pkg/config/ ./pkg/duckdb/
ok  github.com/bruin-data/bruin/pkg/config   0.013s
ok  github.com/bruin-data/bruin/pkg/duckdb   0.025s

New azure coverage all green: TestLakehouseAttacher_GenerateAzureSecret (connection_string + credential_chain + empty), GetRequiredExtensions (azure ext), GenerateAttachStatements (asserts INSTALL/LOAD azure, SET GLOBAL azure_transport_option_type='curl', CREATE SECRET TYPE azure, ATTACH … DATA_PATH 'az://…'), and TestValidateLakehouseConfig azure pass/missing-auth/missing-path cases.

These are unit tests over the generated SQL; I haven't run an E2E against a live Azure storage account.

@turtleDev
turtleDev merged commit 8274d13 into bruin-data:main Jul 22, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants