fix(soda): detect missing columns in CSV/Parquet files via field_is_present

[armorbreak001]

Summary

Fixes #1065 — field_is_present always passes for CSV/Parquet files even when the column is missing from the data.

Root Cause

In create_view_with_schema_union(), when reading CSV/Parquet files via DuckDB, all columns from the datacontract schema are created in the table (even if they don't exist in the actual data file). Missing columns are filled with NULL values. Soda's when required column missing check only verifies column existence in the table — so it always passes, regardless of whether the column was actually in the source file.

Fix

Two coordinated changes:

1. duckdb_connection.py: After inserting data into the schema-based table, drop any columns that don't exist in the source data file. This ensures Soda's field_is_present check correctly detects missing columns.

2. check_soda_execute.py: Before running Soda scans, pre-filter constraint/quality checks (e.g., field_required, field_minimum) that reference dropped columns. These are marked as failed with reason "Column X not found in data file" and excluded from the SodaCL YAML to prevent SQL errors. field_is_present checks are intentionally left in so Soda can evaluate them normally (and correctly report them as failed).

Test Updates

Updated 2 tests to reflect the corrected behavior:

• test_csv_optional_field_missing_from_old_data — now expects field_is_present failure for missing optional field
• test_parquet_optional_field_missing_from_old_data — same for Parquet

All 10 schema evolution tests pass.

[armorbreak001]

Update: PyPI now has Linux x86_64 wheels for pyarrow==24.0.0. The CI failure was likely a temporary resolution issue. Triggered a CI rerun — if it still fails, the root cause may be different (e.g., uv.lock pinning an incompatible transitive dependency).

[jschoedl]

Closing in favour of #1163. Please avoid creating PRs if another one already exists for this issue, except when it is stale for a longer time.