feat: add bronze-to-silver bigquery template

Author: mohin-io

docs/getting-started/templates-docs/bronze-silver-bigquery-README.md

@@ -0,0 +1,67 @@
+# Bruin - Bronze to Silver BigQuery Template
+
+This template delivers an end-to-end example of the bronze-to-silver pattern in
+Bruin. It ingests raw exchange rates from the credential-free Frankfurter API
+into BigQuery and then builds curated metrics that are ready for analytics.
+
+The pipeline ships with:
+
+- `bronze.frankfurter_rates`: An ingestr asset that captures daily FX rates in a
+  bronze dataset.
+- `silver.fx_rate_enriched`: A BigQuery SQL asset that adds rolling averages,
+  day-over-day deltas, and quality checks.
+- `.bruin.yml`: A starter configuration showing how to connect Frankfurter and
+  Google Cloud.
+
+## Setup
+
+Initialize the template from the Bruin CLI:
+
+```bash
+bruin init bronze-silver-bigquery
+```
+
+Update the generated `.bruin.yml` with your Google Cloud project and service
+account. Frankfurter does not require credentials.
+
+```yaml
+default_environment: default
+environments:
+  default:
+    connections:
+      frankfurter:
+        - name: "frankfurter-default"
+      google_cloud_platform:
+        - name: "gcp-default"
+          project_id: "your-gcp-project-id"
+          service_account_file: "/path/to/service-account.json"
+```
+
+Create the `bronze` and `silver` datasets in BigQuery, or adjust the asset names
+to match datasets that already exist.
+
+## Running the pipeline
+
+From the template directory, validate the pipeline and then materialize it:
+
+```bash
+bruin validate .
+bruin run .
+```
+
+The run performs two steps:
+
+1. Loads raw currency rates from Frankfurter into `bronze.frankfurter_rates`.
+2. Generates rolling metrics in `silver.fx_rate_enriched`, complete with sensible
+   quality checks.
+
+## Quality checks
+
+- Bronze layer: not-null and positivity checks plus a custom validation that
+  ensures the base currency remains EUR.
+- Silver layer: not-null and positivity checks on analytics columns and a custom
+  query to guarantee rolling averages are present for historical data.
+
+Use this template as the foundation for more advanced multi-layer pipelines or
+swap the source for any other ingestr-compatible system that does not require
+credentials.

templates/bronze-silver-bigquery/.bruin.yml

@@ -0,0 +1,10 @@
+default_environment: default
+environments:
+  default:
+    connections:
+      frankfurter:
+        - name: "frankfurter-default"
+      google_cloud_platform:
+        - name: "gcp-default"
+          project_id: "your-gcp-project-id"
+          service_account_file: "/path/to/service-account.json"

templates/bronze-silver-bigquery/README.md

@@ -0,0 +1,78 @@
+# Bruin - Bronze to Silver BigQuery Template
+
+This template demonstrates a complete bronze-to-silver pattern built on top of
+BigQuery. It combines a raw ingestion step powered by ingestr with a curated
+transformation layer that adds analytics-friendly aggregates and data quality
+checks.
+
+## Pipeline Overview
+
+- `assets/bronze_raw_data.asset.yml` creates the **bronze** layer by loading
+  publicly available foreign exchange rates from the Frankfurter API into
+  BigQuery.
+- `assets/silver_aggregated.sql` materializes the **silver** layer by computing
+  rolling 7-day averages and day-over-day deltas for each currency.
+- `pipeline.yml` wires the assets together and defines default connections for
+  Frankfurter and Google Cloud.
+- `.bruin.yml` contains a starter configuration you can adapt with your own
+  GCP project and service account.
+
+## Initialize the Template
+
+```bash
+bruin init bronze-silver-bigquery
+```
+
+This creates a folder with the files listed above so you can run the template in
+place or tailor it to your environment.
+
+## Configure Connections
+
+Update `.bruin.yml` with your Google Cloud project details and service account.
+The Frankfurter source does not require credentials, but you can rename the
+connection if desired.
+
+```yaml
+default_environment: default
+environments:
+  default:
+    connections:
+      frankfurter:
+        - name: "frankfurter-default"
+      google_cloud_platform:
+        - name: "gcp-default"
+          project_id: "your-gcp-project-id"
+          service_account_file: "/path/to/service-account.json"
+```
+
+> **BigQuery dataset**: The template writes to the `bronze` and `silver`
+> datasets. Create them ahead of time or change the dataset names in the asset
+> definitions to match your environment.
+
+## Run the Pipeline
+
+Validate and execute the pipeline from the template directory:
+
+```bash
+bruin validate .
+bruin run .
+```
+
+The bronze asset loads historical exchange rates, and the silver asset enriches
+that data with rolling metrics suitable for downstream analytics and reporting.
+
+## Data Quality Highlights
+
+- Every column in the bronze asset is monitored for nulls, and a custom check
+  ensures the base currency remains `EUR`.
+- The silver layer adds positive and not-null checks, plus a custom validation
+  that guarantees rolling averages are present for data older than seven days.
+
+## Next Steps
+
+- Swap in another no-auth source such as the Chess template by updating the
+  `source_connection` and transformation logic.
+- Extend the silver layer with additional materializations (e.g. gold-level
+  dashboards or alerts).
+- Schedule the pipeline using the `schedule` and `start_date` fields in
+  `pipeline.yml` or integrate it with your orchestrator of choice.

templates/bronze-silver-bigquery/assets/bronze_raw_data.asset.yml

@@ -0,0 +1,45 @@
+name: bronze.frankfurter_rates
+type: ingestr
+
+description: |
+  Ingests daily exchange rate data from the public Frankfurter API into BigQuery
+  without requiring credentials. The bronze layer captures the raw payload so
+  downstream transformations can compute rolling trends and quality metrics.
+
+columns:
+  - name: date
+    type: timestamp
+    description: "UTC timestamp returned by the Frankfurter exchange rate feed"
+    checks:
+      - name: not_null
+  - name: currency_code
+    type: string
+    description: "ISO 4217 code for the quoted currency in the rates dataset"
+    checks:
+      - name: not_null
+  - name: base_currency
+    type: string
+    description: "ISO 4217 code of the base currency (EUR by default in Frankfurter)"
+    checks:
+      - name: not_null
+  - name: rate
+    type: numeric
+    description: "Exchange rate value for currency_code relative to base_currency"
+    checks:
+      - name: not_null
+      - name: positive
+
+custom_checks:
+  - name: ensure_eur_reference
+    description: "Verify that the base currency remains EUR for the raw snapshot"
+    value: 0
+    query: |
+      SELECT COUNT(*) AS mismatch_count
+      FROM bronze.frankfurter_rates
+      WHERE base_currency != 'EUR'
+
+parameters:
+  source_connection: frankfurter-default
+  source_table: exchange_rates
+  destination: bigquery
+  destination_table: bronze.frankfurter_rates

templates/bronze-silver-bigquery/assets/silver_aggregated.sql

@@ -0,0 +1,99 @@
+/* @bruin
+
+name: silver.fx_rate_enriched
+type: bq.sql
+
+materialization:
+   type: table
+
+description: |
+  Enriches the bronze Frankfurter exchange rates with rolling aggregates and
+  day-over-day deltas to illustrate a typical silver layer transformation in
+  BigQuery.
+
+depends:
+  - bronze.frankfurter_rates
+
+columns:
+  - name: date
+    type: date
+    description: "Date of the exchange rate observation"
+    primary_key: true
+    checks:
+      - name: not_null
+  - name: currency_code
+    type: string
+    description: "ISO 4217 code for the quoted currency"
+    primary_key: true
+    checks:
+      - name: not_null
+  - name: base_currency
+    type: string
+    description: "Reference currency that the rate is quoted against"
+    checks:
+      - name: not_null
+  - name: rate
+    type: numeric
+    description: "Spot exchange rate value for the currency on the given date"
+    checks:
+      - name: not_null
+      - name: positive
+  - name: avg_rate_7d
+    type: numeric
+    description: "Seven day rolling average to smooth short term volatility"
+    checks:
+      - name: not_null
+      - name: positive
+  - name: change_vs_prev_day
+    type: numeric
+    description: "Day-over-day change in the exchange rate"
+    checks:
+      - name: not_null
+
+custom_checks:
+  - name: ensure_rolling_average_populated
+    description: "Ensure rows older than a week carry a computed rolling average"
+    value: 0
+    query: |
+      SELECT COUNT(*) AS missing_average
+      FROM silver.fx_rate_enriched
+      WHERE avg_rate_7d IS NULL
+        AND date <= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
+
+@bruin */
+
+WITH cleaned_rates AS (
+  SELECT
+    DATE(date) AS rate_date,
+    currency_code,
+    base_currency,
+    SAFE_CAST(rate AS NUMERIC) AS rate
+  FROM `bronze.frankfurter_rates`
+  WHERE DATE(date) >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY)
+),
+enriched AS (
+  SELECT
+    rate_date,
+    currency_code,
+    base_currency,
+    rate,
+    AVG(rate) OVER (
+      PARTITION BY currency_code
+      ORDER BY rate_date
+      ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
+    ) AS avg_rate_7d,
+    COALESCE(
+      rate - LAG(rate) OVER (PARTITION BY currency_code ORDER BY rate_date),
+      0
+    ) AS change_vs_prev_day
+  FROM cleaned_rates
+)
+SELECT
+  rate_date AS date,
+  currency_code,
+  base_currency,
+  rate,
+  avg_rate_7d,
+  change_vs_prev_day
+FROM enriched
+ORDER BY date, currency_code;

templates/bronze-silver-bigquery/pipeline.yml

@@ -0,0 +1,7 @@
+name: bronze-silver-bigquery
+schedule: daily
+start_date: "2024-01-01"
+
+default_connections:
+  google_cloud_platform: gcp-default
+  frankfurter: frankfurter-default