|
| 1 | +# GitHub-release worker transport (`github://` / `github-auto://`) |
| 2 | + |
| 3 | +Run a VGI worker straight from a GitHub release — no container runtime, no manual |
| 4 | +download. The extension fetches the release asset, verifies its SHA256, extracts it |
| 5 | +into a per-user cache, and runs the worker over the ordinary subprocess transport. |
| 6 | + |
| 7 | +This is a **developer-experience convenience**. It overlaps `oci://` (which already |
| 8 | +gives registry distribution, signing, and isolation); the one thing it adds is "no |
| 9 | +container runtime needed." For locked-down / enterprise supply-chain control, prefer |
| 10 | +publishing an image and using `oci://` against your own registry. |
| 11 | + |
| 12 | +## The two schemes |
| 13 | + |
| 14 | +```sql |
| 15 | +-- Explicit: any repo, you name the asset and (recommended) pin its digest. |
| 16 | +ATTACH 'u' AS u (TYPE vgi, |
| 17 | + LOCATION 'github://Query-farm/vgi-units@v0.1.1/vgi-units-v0.1.1-osx_arm64.tar.gz#sha256=78168f…'); |
| 18 | + |
| 19 | +-- Auto: the publisher follows a naming convention; one line works on every platform. |
| 20 | +ATTACH 'u' AS u (TYPE vgi, LOCATION 'github-auto://Query-farm/vgi-units@v0.1.1'); |
| 21 | +``` |
| 22 | + |
| 23 | +Both accept the same LOCATION wherever a worker path is accepted: `ATTACH`, |
| 24 | +`vgi_table_function(...)`, `vgi_catalogs(...)`. |
| 25 | + |
| 26 | +### `github://owner/repo@tag/asset[#sha256=<hex>][#path=<member>]` |
| 27 | +- `asset` is the exact release asset filename (the segment after the last `/`; tags |
| 28 | + may themselves contain `/`). |
| 29 | +- `#sha256=<hex>` pins the downloaded asset's digest. **Strongly recommended** — it is |
| 30 | + the integrity guarantee, and it is enforced even on a cache hit. |
| 31 | +- `#path=<member>` selects which file inside a multi-file archive is the worker |
| 32 | + executable (only needed if auto-selection is ambiguous — see below). |
| 33 | + |
| 34 | +### `github-auto://owner/repo@tag[/prefix]` |
| 35 | +Builds the asset name from a fixed convention and verifies it against the published |
| 36 | +`.sha256` sidecar (so no manual pin is needed): |
| 37 | + |
| 38 | +``` |
| 39 | +{prefix}-{tag}-{platform}.tar.gz (prefix defaults to the repo name) |
| 40 | +``` |
| 41 | + |
| 42 | +`{platform}` is DuckDB's own platform string (`osx_arm64`, `linux_amd64`, |
| 43 | +`linux_arm64`, `linux_amd64_musl`, …). The digest is read from the first sidecar that |
| 44 | +exists among `{stem}.sha256` (GoReleaser style) or `{asset}.sha256`. |
| 45 | + |
| 46 | +It is a **deterministic convention**, not a heuristic: it constructs *one* exact name |
| 47 | +and fails with a clear "asset not found, available: …" listing if the publisher didn't |
| 48 | +follow it. |
| 49 | + |
| 50 | +## Publisher contract |
| 51 | + |
| 52 | +To support `github-auto://`, publish per-platform assets named |
| 53 | +`{repo}-{tag}-{platform}.tar.gz` (platform = the DuckDB platform string) plus a |
| 54 | +`.sha256` sidecar. The reference `Query-farm/vgi-units` release (a GoReleaser pipeline) |
| 55 | +is the model: `vgi-units-v0.1.1-osx_arm64.tar.gz` + `vgi-units-v0.1.1-osx_arm64.sha256`, |
| 56 | +each tarball containing the executable + README + LICENSE. |
| 57 | + |
| 58 | +**Build the worker relocatable.** The worker runs from wherever it was cached, with |
| 59 | +the caller's working directory inherited (we do *not* `chdir` into the cache). So: |
| 60 | +- link shared libraries with rpath `@loader_path` (macOS) / `$ORIGIN` (Linux); |
| 61 | +- resolve data files relative to the executable (`/proc/self/exe`, |
| 62 | + `_NSGetExecutablePath`), never relative to the current directory. |
| 63 | + |
| 64 | +If the worker is a single self-contained binary (static Go/Rust, etc.), nothing extra |
| 65 | +is required. |
| 66 | + |
| 67 | +## Verification |
| 68 | + |
| 69 | +- **SHA256 is enforced**: `github://` against the `#sha256=` pin, `github-auto://` |
| 70 | + against the `.sha256` sidecar. The archive bytes are verified *before* the tar |
| 71 | + parser runs, so extraction only ever sees trusted input. A mismatch throws. |
| 72 | +- **No allowlist**: consistent with every other LOCATION scheme (bare path runs any |
| 73 | + executable, `oci://` runs any image). The trust boundary is whoever writes the |
| 74 | + `ATTACH`. Note this is the first scheme that fetches and execs *remote* native code |
| 75 | + from a string — pin digests and treat LOCATION strings in shared catalogs/configs |
| 76 | + accordingly. |
| 77 | +- **Sigstore/cosign provenance** is a deferred follow-up (the reference release already |
| 78 | + ships `.cosign.bundle` sidecars). |
| 79 | + |
| 80 | +## Caching |
| 81 | + |
| 82 | +- Cache dir: `${XDG_CACHE_HOME:-~/.cache}/vgi/releases` (override with the |
| 83 | + `vgi_github_cache_dir` setting). It must be on an **exec-capable** filesystem (not a |
| 84 | + `noexec` runtime/tmp mount). |
| 85 | +- Keyed by **content digest** — two attaches of the same release share one cached, |
| 86 | + immutable extracted tree, across concurrent DuckDB processes (coordinate-keyed |
| 87 | + `flock`, atomic directory install). |
| 88 | +- macOS arm64: the extracted binary is ad-hoc code-signed (`codesign --deep -s -`) so |
| 89 | + unsigned Mach-O / nested dylibs aren't `SIGKILL`ed at exec. |
| 90 | +- No automatic eviction in v1 — clear with `SELECT * FROM vgi_github_cache_flush();` |
| 91 | + (or `rm -rf` the cache dir). |
| 92 | + |
| 93 | +## Diagnostics |
| 94 | + |
| 95 | +```sql |
| 96 | +SELECT * FROM vgi_github_cache(); -- owner, repo, tag, asset, digest, dir, entrypoint, age_seconds |
| 97 | +SELECT * FROM vgi_github_cache_flush(); -- clears the cache; returns the count removed |
| 98 | +``` |
| 99 | + |
| 100 | +## Formats & limits |
| 101 | + |
| 102 | +- Supported assets: bare executable, single-file `.zst`/`.gz`, and `.tar.gz` / |
| 103 | + `.tar.zst` archives (decompressed in-process; the full tree is extracted with |
| 104 | + tar-slip path sanitization). The single executable-bit member is the entrypoint; |
| 105 | + use `#path=<member>` to disambiguate. |
| 106 | +- `.zip` is not supported (publish `.tar.gz`). |
| 107 | +- **POSIX-only.** On non-POSIX builds these schemes throw a clear error at ATTACH. |
| 108 | +- Authenticated GitHub API access: set `$GITHUB_TOKEN` (or `$GH_TOKEN`) to avoid the |
| 109 | + 60-request/hour unauthenticated limit and to reach private repos. Asset *downloads* |
| 110 | + use the pre-signed CDN URL with no Authorization header. |
0 commit comments