feat: add portable CLI binaries for multi-platform Supabase CLI distribution#2033
feat: add portable CLI binaries for multi-platform Supabase CLI distribution#2033
Conversation
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
30 times, most recently
from
cebd363 to
57718b4
Compare
|
Here are the issues I had integrating the artefact into the CLI: PostgreSQL Bundle IssuesThis document describes the issues encountered with the current PostgreSQL bundle ( Goal: The bundle should be extractable and runnable without any CLI-side workarounds. 1. Code Signing on macOS (Critical)Status: Blocking - requires CLI workaround Issue: After extracting the zip, all binaries crash immediately with: Root Cause: The binaries have invalid or missing code signatures after extraction. macOS's security system (Gatekeeper) kills unsigned/invalid binaries. Current Workaround: Re-sign all executables and // binary.go:434-463 - codesignPostgresDir()
cmd := exec.Command("codesign", "-f", "-s", "-", path)Ideal Fix: Sign binaries during packaging, or use proper Mach-O flags that don't require signing. 2. Timezone Data - Hardcoded Nix Paths (Critical)Status: Blocking PostgREST - no effective workaround Issue: The postgres binary was built with: This causes Impact:
Attempted Workarounds:
Ideal Fix: Either:
3. safeupdate Extension MissingStatus: Fixed with CLI workaround Issue: The bundled migrations configure the session_preload_libraries=safeupdateBut the Current Workaround: Reset the setting after migrations run: -- process-compose.yaml.tmpl:56-57
ALTER ROLE authenticator RESET session_preload_libraries;Ideal Fix: Either:
4. pgsodium/vault getkey Script PathsStatus: Fixed with CLI workaround Issue: The Without these, postgres fails to start with pgsodium/vault enabled. Current Workaround: Append explicit paths after copying template: // sandbox.go:129-130
conf += fmt.Sprintf("\npgsodium.getkey_script = '%s'\n", pgsodiumScript)
conf += fmt.Sprintf("vault.getkey_script = '%s'\n", pgsodiumScript)Ideal Fix: Use relative paths in the template, or document the expected configuration. 5. pg_hba.conf AuthenticationStatus: Fixed with CLI workaround Issue: The bundled But Current Workaround: Override with trust auth for localhost: // sandbox.go:136-152
host all all 127.0.0.1/32 trustIdeal Fix: Either:
6. Locale Path - Hardcoded Nix Path (Warning)Status: Warning only - not blocking Issue: The binary tries to use a Nix-specific locale binary: Impact: Warning messages during initdb, but doesn't break functionality. Ideal Fix: Build without hardcoded Nix paths for locale detection. Summary: Ideal Bundle RequirementsFor the bundle to work without workarounds:
|
|
I asked Claude to compare what we're doing in the Docker image vs what's included in the artifact. How the Docker Image Works (Reference)The Supabase postgres Docker image ( Docker Initialization Flow
Why Docker Works and Bundle Doesn't
Recommended Bundle ChangesTo make the bundle work like the Docker image without CLI workarounds: 1. Add an Initialization ScriptInclude a # share/supabase-cli/bin/supabase-postgres-init.sh
#!/bin/bash
set -e
PGDATA="${PGDATA:-./data}"
POSTGRES_USER="${POSTGRES_USER:-supabase_admin}"
POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-postgres}"
# Get the directory where this script lives
SCRIPT_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
PGBIN="$SCRIPT_DIR/bin"
# Initialize if needed
if [ ! -s "$PGDATA/PG_VERSION" ]; then
"$PGBIN/initdb" -D "$PGDATA" -U "$POSTGRES_USER" --encoding=UTF8 --locale=C
# Set password for the superuser immediately (using single-user mode)
"$PGBIN/postgres" --single -D "$PGDATA" -j <<-EOSQL
ALTER USER "$POSTGRES_USER" WITH PASSWORD '$POSTGRES_PASSWORD';
EOSQL
# Copy config templates
cp "$SCRIPT_DIR/share/supabase-cli/config/postgresql.conf.template" "$PGDATA/postgresql.conf"
cp "$SCRIPT_DIR/share/supabase-cli/config/pg_hba.conf.template" "$PGDATA/pg_hba.conf"
fi
# Start postgres (or exec to replace shell)
exec "$PGBIN/postgres" -D "$PGDATA" "$@"2. Bundle Timezone DataOption A (Recommended): Build postgres WITHOUT
Option B: Include tzdata in bundle
3. Include safeupdate ExtensionAdd to bundle: 4. Fix Config Templatespg_hba.conf.template - Can use scram-sha-256 if init script sets password: postgresql.conf.template - Use relative/discoverable paths: # Use path relative to PGDATA or make it configurable
# pgsodium.getkey_script should be set by init script based on install location5. Sign macOS Binaries During BuildAdd to build process: # Sign all binaries and dylibs before creating zip
find . -type f \( -perm +111 -o -name "*.dylib" \) -exec codesign -s "Developer ID" {} \;Or use ad-hoc signing: find . -type f \( -perm +111 -o -name "*.dylib" \) -exec codesign -f -s - {} \;6. Build Without Nix-Specific PathsRemove from configure:
Ideal Bundle StructureSummary of Build Changes Required
With these changes, the CLI would only need to:
No workarounds needed. |
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
2 times, most recently
from
814d460 to
3065354
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@nix/packages/postgres-portable.nix`:
- Around line 223-242: The timezone-data verification block that checks for
timezone/ and timezonesets/ (the commented shell snippet in the file) must be
either re-enabled or documented: if the removal of --with-system-tzdata in
postgres.nix actually bundles tzdata, uncomment and restore the verification
block to assert $out/share/postgresql/timezone or $out/share/timezonesets (and
the alternate $out/share paths) and fail early on missing data; otherwise add a
clear comment above that block referencing the change to --with-system-tzdata in
postgres.nix and why the runtime check is intentionally disabled so reviewers
know this is deliberate.
🧹 Nitpick comments (3)
nix/checks.nix (1)
780-780: Duplicate section numbering: two "Section 8" headers.Line 780 has "Section 8: Verifying wrapper scripts" and line 838 has "Section 8: Checking for hardcoded /nix/store paths". The second should be "Section 10".
📝 Proposed fix
- echo "=== Section 8: Checking for hardcoded /nix/store paths in binaries ===" + echo "=== Section 10: Checking for hardcoded /nix/store paths in binaries ==="Also applies to: 838-838
nix/packages/cli-config/postgresql.conf.template (1)
35-43: Consider documenting portable relative path pattern.The commented getkey script paths require users to manually configure after copying. Per PR feedback, consider documenting the portable relative path pattern or providing a helper to auto-configure based on bundle location.
💡 Suggested documentation improvement
# pgsodium and vault configuration # Set the path to your pgsodium_getkey.sh script # Default location when using portable bundle: share/supabase-cli/config/pgsodium_getkey.sh +# For portable bundles, use paths relative to PGDATA or absolute paths to the bundle: +# Example: pgsodium.getkey_script = '/path/to/bundle/share/supabase-cli/config/pgsodium_getkey.sh' `#pgsodium.getkey_script` = '/path/to/pgsodium_getkey.sh' `#vault.getkey_script` = '/path/to/pgsodium_getkey.sh'.github/workflows/cli-release.yml (1)
188-194: Extension verification incomplete - missing CLI extensions.The check verifies
pg_graphql,pgcrypto,uuid-ossp,supabase_vault, but doesn't verifypgsodium,pg_net,pg_cron, orsupautilswhich are in the CLI extension set.🔧 Proposed fix
# Check for required extensions - for ext in pg_graphql pgcrypto uuid-ossp supabase_vault; do + for ext in pg_graphql pgcrypto uuid-ossp supabase_vault pgsodium pg_net pg_cron supautils; do if ! grep -q "$ext" extensions.log; then echo "Required extension $ext not found" exit 1
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@nix/packages/postgres-portable.nix`:
- Around line 334-373: The macOS Mach-O patching loop (the for bin in
$out/bin/.*-wrapped and for lib in $out/lib/*.dylib* blocks using
otool/install_name_tool) leaves binaries unsigned causing "Code Signature
Invalid" crashes; after completing install_name_tool changes add an ad-hoc
codesign step to re-sign each patched executable and dylib (e.g., run codesign
-f -s - on the same $bin and $lib files), gated by the existing stdenv.isDarwin
conditional and keeping errors non-fatal (redirect or || true) so packaging
continues.
- Around line 29-48: The migrations bundle references the safeupdate extension
but the CLI build excludes it via cliExtensions causing runtime failures; either
update the migration file 20220118070449_enable-safeupdate-postgrest.sql to wrap
the session_preload_libraries = 'safeupdate' change in a guard that checks for
the extension (similar to the conditional used in
20220224211803_fix-postgrest-supautils.sql), or add the pg-safeupdate.nix entry
into cliExtensions so psql_17_cli provides the safeupdate extension; modify the
migration SQL to use the same EXISTS/DO/IF logic as the supautils migration or
update postgres.nix to include pg-safeupdate.nix in cliExtensions accordingly.
🧹 Nitpick comments (2)
nix/packages/postgres-portable.nix (2)
79-91: Consider excluding additional system libraries.The exclusion list covers core glibc libraries but misses some commonly system-provided libraries that could cause ABI mismatches if bundled:
libgcc_s.so*- GCC runtime (tied to system compiler)libstdc++.so*- C++ standard library (if any C++ code is involved)libnsl.so*,libnss_*.so*- Name service librariesBundling these from Nix while the host uses different versions can cause subtle runtime issues.
💡 Suggested addition
case "$libname" in libc.so*|libc-*.so*|ld-linux*.so*|libdl.so*|libpthread.so*|libm.so*|libresolv.so*|librt.so*) return 0 # Exclude ;; + libgcc_s.so*|libstdc++.so*|libnsl.so*|libnss_*.so*) + return 0 # Exclude - system ABI-coupled libraries + ;; *) return 1 # Include ;;
163-180: Silent failures when copying dependencies may produce incomplete bundles.The
cp "$dep" $out/lib/ 2>/dev/null || truepattern silently ignores copy failures. If a required dependency fails to copy (e.g., permission issues, disk space), the bundle will be incomplete but the build succeeds.Consider at minimum logging when a copy fails, or tracking failures and reporting at the end.
💡 Suggested improvement
if ! should_exclude_library "$libname"; then - cp "$dep" $out/lib/ 2>/dev/null || true + if ! cp "$dep" $out/lib/ 2>/dev/null; then + echo "Warning: Failed to copy dependency $dep" + fi else
| + lib.optionalString stdenv.isDarwin '' | ||
| # On macOS, patch binaries to use relative library paths | ||
| # This makes the bundle portable across macOS systems for Supabase CLI | ||
| for bin in $out/bin/.*-wrapped; do | ||
| if [ -f "$bin" ] && file "$bin" | grep -q "Mach-O"; then | ||
| # Get all dylib dependencies from Nix store | ||
| otool -L "$bin" | grep /nix/store | awk '{print $1}' | while read dep; do | ||
| libname=$(basename "$dep") | ||
| # Check if we have this library in our lib directory | ||
| if [ -f "$out/lib/$libname" ]; then | ||
| echo "Patching $bin: $dep -> @rpath/$libname" | ||
| install_name_tool -change "$dep" "@rpath/$libname" "$bin" 2>/dev/null || true | ||
| fi | ||
| done | ||
| # Add @rpath to look in @executable_path/../lib | ||
| install_name_tool -add_rpath "@executable_path/../lib" "$bin" 2>/dev/null || true | ||
| fi | ||
| done | ||
|
|
||
| # Patch dylibs to use @rpath for their dependencies | ||
| for lib in $out/lib/*.dylib*; do | ||
| if [ -f "$lib" ] && file "$lib" | grep -q "Mach-O"; then | ||
| # First, fix the library's own ID to use @rpath | ||
| libname=$(basename "$lib") | ||
| install_name_tool -id "@rpath/$libname" "$lib" 2>/dev/null || true | ||
|
|
||
| # Add @rpath to the library itself so it can find other libraries | ||
| install_name_tool -add_rpath "@loader_path" "$lib" 2>/dev/null || true | ||
|
|
||
| # Then fix references to other libraries | ||
| otool -L "$lib" | grep /nix/store | awk '{print $1}' | while read dep; do | ||
| deplibname=$(basename "$dep") | ||
| if [ -f "$out/lib/$deplibname" ]; then | ||
| echo "Patching $lib: $dep -> @rpath/$deplibname" | ||
| install_name_tool -change "$dep" "@rpath/$deplibname" "$lib" 2>/dev/null || true | ||
| fi | ||
| done | ||
| fi | ||
| done | ||
| ''; |
There was a problem hiding this comment.
Missing macOS code signing causes "Code Signature Invalid" crashes.
Per reviewer feedback, extracted macOS binaries crash with "Code Signature Invalid" after the Mach-O patching. The current workaround requires users to manually re-sign with codesign -f -s -.
Consider adding ad-hoc signing after patching:
💡 Suggested addition
done
+
+ # Ad-hoc sign all patched binaries and dylibs to fix code signature invalidation
+ echo "Ad-hoc signing binaries and libraries for macOS..."
+ for bin in $out/bin/.*-wrapped; do
+ if [ -f "$bin" ]; then
+ codesign -f -s - "$bin" 2>/dev/null || echo "Warning: Failed to sign $bin"
+ fi
+ done
+ for lib in $out/lib/*.dylib*; do
+ if [ -f "$lib" ]; then
+ codesign -f -s - "$lib" 2>/dev/null || echo "Warning: Failed to sign $lib"
+ fi
+ done
'';🤖 Prompt for AI Agents
In `@nix/packages/postgres-portable.nix` around lines 334 - 373, The macOS Mach-O
patching loop (the for bin in $out/bin/.*-wrapped and for lib in
$out/lib/*.dylib* blocks using otool/install_name_tool) leaves binaries unsigned
causing "Code Signature Invalid" crashes; after completing install_name_tool
changes add an ad-hoc codesign step to re-sign each patched executable and dylib
(e.g., run codesign -f -s - on the same $bin and $lib files), gated by the
existing stdenv.isDarwin conditional and keeping errors non-fatal (redirect or
|| true) so packaging continues.
9bc3af3 to
345b8ff
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@nix/packages/postgres.nix`:
- Around line 194-201: The current assignment to postFixup is replacing upstream
fixup steps; instead, preserve upstream behavior and only remove the initdb
wrapper on Linux by appending a conditional snippet to the existing postFixup
(e.g., set postFixup = (old: old + lib.optionalString
pkgs.stdenv.hostPlatform.isLinux '' ... '') so you only inject the removal of
the initdb wrapper and do not discard other upstream fixups), reference
postFixup, lib.optionalString, pkgs.stdenv.hostPlatform.isLinux and the initdb
wrapper; if you truly intend to disable postFixup entirely for the CLI variant,
add an explicit comment documenting that decision.
nix/packages/postgres.nix
Outdated
| # CRITICAL FIX FOR LINUX: Override postFixup to remove initdb wrapper | ||
| # The upstream nixpkgs adds a wrapper on Linux that embeds glibc.bin path | ||
| # We remove this wrapper because 'locale' should be found from system PATH | ||
| postFixup = lib.optionalString pkgs.stdenv.hostPlatform.isLinux '' | ||
| # Do NOT wrap initdb with glibc.bin path | ||
| # The system's locale command will be used from PATH | ||
| # This is acceptable for portable CLI distribution | ||
| ''; |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's see the full context around the lines in question
head -210 nix/packages/postgres.nix | tail -40Repository: supabase/postgres
Length of output: 2545
🏁 Script executed:
# Check the structure of the entire postgres.nix file to understand the override pattern
wc -l nix/packages/postgres.nixRepository: supabase/postgres
Length of output: 90
🏁 Script executed:
# Look for the override block containing this code
grep -n "override\|postFixup" nix/packages/postgres.nix | head -20Repository: supabase/postgres
Length of output: 468
🏁 Script executed:
# Check the basePostgresql definition to see its postFixup
grep -n "basePostgresql" nix/packages/postgres.nix | head -10Repository: supabase/postgres
Length of output: 204
🏁 Script executed:
# Look for any upstream postFixup references or check what basePostgresql pulls from
grep -n "inherit\|postgresql\|pkgs\." nix/packages/postgres.nix | head -30Repository: supabase/postgres
Length of output: 1701
🏁 Script executed:
# Search for where basePostgresql is defined
grep -B 5 "basePostgresql\s*=" nix/packages/postgres.nixRepository: supabase/postgres
Length of output: 195
🏁 Script executed:
# Let's understand what upstream nixpkgs postgres does with postFixup
# Search for what basePostgresql (pkgs.postgresql_*) might contain
# Check if there are any references to postFixup behavior in comments or documentation
grep -n "postFixup\|preConfigure\|previousAttrs" nix/packages/postgres.nixRepository: supabase/postgres
Length of output: 484
🏁 Script executed:
# Let's check the git history or similar patterns to see if postFixup was intentionally appended before
grep -B 3 -A 3 "previousAttrs\." nix/packages/postgres.nixRepository: supabase/postgres
Length of output: 1484
🏁 Script executed:
# Let's check if there's any actual content that should be in postFixup
# The block appears empty - let's verify this is intentional
sed -n '194,203p' nix/packages/postgres.nixRepository: supabase/postgres
Length of output: 745
🏁 Script executed:
# Let's see if there are any other patterns or comments explaining the postFixup intent
sed -n '190,210p' nix/packages/postgres.nixRepository: supabase/postgres
Length of output: 1443
Clarify intent: does the CLI variant need postFixup disabled entirely, or just the initdb wrapper removed?
The current assignment postFixup = lib.optionalString pkgs.stdenv.hostPlatform.isLinux ''...'' replaces the entire upstream postFixup with an empty string on all platforms, discarding any upstream fixup steps that may be critical for portability. This differs from how preConfigure is handled (appended, not replaced).
If the goal is simply to avoid the initdb wrapper while preserving other fixups, use the safer pattern:
Recommended fix
- postFixup = lib.optionalString pkgs.stdenv.hostPlatform.isLinux ''
+ postFixup = (previousAttrs.postFixup or "") + lib.optionalString pkgs.stdenv.hostPlatform.isLinux ''
# Do NOT wrap initdb with glibc.bin path
# The system's locale command will be used from PATH
# This is acceptable for portable CLI distribution
'';If disabling postFixup entirely for the CLI variant is intentional and safe, document this explicitly.
🤖 Prompt for AI Agents
In `@nix/packages/postgres.nix` around lines 194 - 201, The current assignment to
postFixup is replacing upstream fixup steps; instead, preserve upstream behavior
and only remove the initdb wrapper on Linux by appending a conditional snippet
to the existing postFixup (e.g., set postFixup = (old: old + lib.optionalString
pkgs.stdenv.hostPlatform.isLinux '' ... '') so you only inject the removal of
the initdb wrapper and do not discard other upstream fixups), reference
postFixup, lib.optionalString, pkgs.stdenv.hostPlatform.isLinux and the initdb
wrapper; if you truly intend to disable postFixup entirely for the CLI variant,
add an explicit comment documenting that decision.
3b7417d to
d9cb921
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@nix/checks.nix`:
- Around line 215-256: The cliSkipTests array in nix/checks.nix includes "plv8"
but there is no corresponding test file; remove the "plv8" entry from the
cliSkipTests list (or alternatively add a matching test file under
nix/tests/sql/ if the test is required) so filteredTestList and the test
basename mapping no longer reference a non-existent test; update the
cliSkipTests array (symbol: cliSkipTests) and re-run the filter logic (symbol:
filteredTestList) to ensure no missing basenames remain.
In `@nix/packages/cli-config/pg_hba.conf.template`:
- Around line 5-9: The pg_hba.conf.template currently requires scram-sha-256 for
loopback TCP ("host all all 127.0.0.1/32 scram-sha-256" and "host all all
::1/128 scram-sha-256"), which will reject initdb-created passwordless superuser
connections; update the CLI-friendly template to use "trust" for those loopback
host rules or alternatively make the init wrapper set a password before enabling
SCRAM—modify the entries in pg_hba.conf.template for 127.0.0.1/32 and ::1/128 to
"trust" (or implement password-setting in the init script that runs before
switching these entries to scram-sha-256).
| # Tests to skip for CLI variants (require extensions not in CLI) | ||
| cliSkipTests = [ | ||
| # Extension-specific tests | ||
| "evtrigs" | ||
| "http" | ||
| "hypopg" | ||
| "index_advisor" | ||
| "pg_hashids" | ||
| "pg_jsonschema" | ||
| "pg_partman" | ||
| "pg_repack" | ||
| "pg_tle" | ||
| "pgtap" | ||
| "pgmq" | ||
| "pgroonga" | ||
| "pgrouting" | ||
| "plpgsql-check" | ||
| "plv8" | ||
| "postgis" | ||
| "postgres_fdw" | ||
| # Tests that depend on extensions not in CLI | ||
| "security" # depends on various extensions | ||
| "extensions_schema" # tests extension loading | ||
| "roles" # includes roles/schemas from extensions not in CLI (pgtle, pgmq, repack, topology) | ||
| # Version-specific extension tests | ||
| "z_17_ext_interface" | ||
| "z_17_pg_stat_monitor" | ||
| "z_17_pgvector" | ||
| "z_17_rum" | ||
| "z_17_roles" # version-specific roles test, includes pgtle_admin | ||
| ]; | ||
|
|
||
| # Convert filtered tests to a sorted list of basenames (without extension) | ||
| testList = pkgs.lib.mapAttrsToList ( | ||
| name: _: builtins.substring 0 (pkgs.lib.stringLength name - 4) name | ||
| ) filteredSqlTests; | ||
| sortedTestList = builtins.sort (a: b: a < b) testList; | ||
|
|
||
| # Filter out CLI-incompatible tests if this is a CLI variant | ||
| filteredTestList = | ||
| if isCliVariant then | ||
| builtins.filter (test: !(builtins.elem test cliSkipTests)) testList | ||
| else |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Compare cliSkipTests entries against nix/tests/sql/*.sql basenames.
python - <<'PY'
import re, pathlib
text = pathlib.Path("nix/checks.nix").read_text()
m = re.search(r"cliSkipTests\s*=\s*\[(.*?)\];", text, re.S)
skip = re.findall(r'"([^"]+)"', m.group(1)) if m else []
tests = [p.stem for p in pathlib.Path("nix/tests/sql").glob("*.sql")]
missing = [s for s in skip if s not in tests]
extra = [t for t in tests if t in skip]
print("Skip entries without matching test file:", missing)
print("Skip entries that match:", extra)
PYRepository: supabase/postgres
Length of output: 467
Remove plv8 from cliSkipTests—it has no corresponding test file.
Verification shows plv8 is listed in the skip list but lacks a matching test file in nix/tests/sql/. Most other entries (including plpgsql-check) correctly match their test basenames. Remove plv8 or create the missing test file.
🤖 Prompt for AI Agents
In `@nix/checks.nix` around lines 215 - 256, The cliSkipTests array in
nix/checks.nix includes "plv8" but there is no corresponding test file; remove
the "plv8" entry from the cliSkipTests list (or alternatively add a matching
test file under nix/tests/sql/ if the test is required) so filteredTestList and
the test basename mapping no longer reference a non-existent test; update the
cliSkipTests array (symbol: cliSkipTests) and re-run the filter logic (symbol:
filteredTestList) to ensure no missing basenames remain.
00bdb0f to
d3de8d8
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@nix/packages/postgres-portable.nix`:
- Around line 50-61: Uncomment and set the getkey_script paths in the
postgresql.conf.template so the portable bundle points to the installed script
path (installed by configBundle) — update the pgsodium.getkey_script and
vault.getkey_script entries in postgresql.conf.template to reference the bundle
location (share/supabase-cli/config/pgsodium_getkey.sh) or implement a startup
hook in the CLI to rewrite those config keys at launch; ensure references align
with the install path created by configBundle (the directory created in
installPhase) and document the behavior so pgsodium and vault can locate the
getkey script when the portable bundle is used.
| configBundle = stdenv.mkDerivation { | ||
| name = "cli-config-bundle"; | ||
| src = configDir; | ||
| dontPatchShebangs = true; | ||
| installPhase = '' | ||
| mkdir -p $out/share/supabase-cli/config | ||
| cp postgresql.conf.template $out/share/supabase-cli/config/ | ||
| cp pg_hba.conf.template $out/share/supabase-cli/config/ | ||
| cp pg_ident.conf.template $out/share/supabase-cli/config/ | ||
| cp pgsodium_getkey.sh $out/share/supabase-cli/config/ | ||
| chmod +x $out/share/supabase-cli/config/pgsodium_getkey.sh | ||
| ''; |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
fd -t f "\.(template|sh)$" nix/packages/Repository: supabase/postgres
Length of output: 244
🏁 Script executed:
echo "=== pg_hba.conf.template ===" && cat nix/packages/cli-config/pg_hba.conf.template && \
echo -e "\n=== pgsodium_getkey.sh ===" && cat nix/packages/cli-config/pgsodium_getkey.sh && \
echo -e "\n=== postgresql.conf.template ===" && head -50 nix/packages/cli-config/postgresql.conf.templateRepository: supabase/postgres
Length of output: 2208
🏁 Script executed:
cat -n nix/packages/postgres-portable.nix | head -80Repository: supabase/postgres
Length of output: 3062
🏁 Script executed:
rg -i "getkey_script|pgsodium.*path|postgresql\.conf" --type sh --type rs --type ts --type go -A 3 -B 1 | head -100Repository: supabase/postgres
Length of output: 87
🏁 Script executed:
rg -i "getkey_script|pgsodium.*path" -A 2 -B 1 | head -80Repository: supabase/postgres
Length of output: 5047
🏁 Script executed:
grep -r "run-server\|launcher\|wrapper" nix/packages/postgres-portable.nix || echo "Not found in postgres-portable.nix"Repository: supabase/postgres
Length of output: 345
🏁 Script executed:
rg "configBundle|postgresql\.conf\.template" nix/packages/ -A 5 -B 5 | head -60Repository: supabase/postgres
Length of output: 2061
🏁 Script executed:
cd nix/packages && grep -A 30 "buildPhase\|installPhase" postgres-portable.nix | grep -E "run-server|launcher|bin/" | head -20Repository: supabase/postgres
Length of output: 261
🏁 Script executed:
# Check what's actually included in the final output
grep -A 50 "stdenv.mkDerivation {" nix/packages/postgres-portable.nix | grep -E "mkdir|cp|bin/|share/" | head -30Repository: supabase/postgres
Length of output: 1025
🏁 Script executed:
# Look for documentation or CLI integration
find . -name "README*" -o -name "*.md" | head -10Repository: supabase/postgres
Length of output: 362
🏁 Script executed:
# Search for references to share/supabase-cli/config
rg "share/supabase-cli/config|cli-config" --type md --type txt -A 2 -B 1Repository: supabase/postgres
Length of output: 43
🏁 Script executed:
cat README.md | grep -A 20 -i "portable\|cli\|config" | head -50Repository: supabase/postgres
Length of output: 3259
🏁 Script executed:
# Search for where CLI might process these templates
rg "share/supabase-cli" -lRepository: supabase/postgres
Length of output: 144
🏁 Script executed:
# Look for any comments or issues about template configuration in the codebase
rg "placeholder|path.*template|getkey.*path" --type nix -B 2 -A 2 | head -80Repository: supabase/postgres
Length of output: 379
🏁 Script executed:
# Check if there's a mechanism in the nix package to handle path substitution
tail -100 nix/packages/postgres-portable.nixRepository: supabase/postgres
Length of output: 4291
Uncomment and configure getkey_script paths in postgresql.conf.template for portable bundle usage.
The template has pgsodium.getkey_script and vault.getkey_script commented out with a placeholder path. Since the portable bundle installs the script to share/supabase-cli/config/pgsodium_getkey.sh, these lines should be uncommented and configured to reference that location, or a documented mechanism must exist for the CLI to set these paths at startup. Without this configuration, the pgsodium and vault extensions won't function. The pg_hba.conf configuration (trust for local, scram-sha-256 for network) is appropriate for passwordless init.
🤖 Prompt for AI Agents
In `@nix/packages/postgres-portable.nix` around lines 50 - 61, Uncomment and set
the getkey_script paths in the postgresql.conf.template so the portable bundle
points to the installed script path (installed by configBundle) — update the
pgsodium.getkey_script and vault.getkey_script entries in
postgresql.conf.template to reference the bundle location
(share/supabase-cli/config/pgsodium_getkey.sh) or implement a startup hook in
the CLI to rewrite those config keys at launch; ensure references align with the
install path created by configBundle (the directory created in installPhase) and
document the behavior so pgsodium and vault can locate the getkey script when
the portable bundle is used.
7b41b7c to
fc53a9e
Compare
|
I've updated the PR to
I have not made any changes to the template - I think it's normal to update a config template at runtime Adhoc codesigning isn't working and is hard to debug in CI, I think it's ok to continue with client-side codesigning for macOS for now since that is what homebrew kegs do when installing Let's discuss the init with others on Slack and then we can update this PR with whatever is decided |
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
from
fc53a9e to
a35290e
Compare
|
@CodeRabbit pause |
✅ Actions performedReviews paused. |
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
from
a35290e to
47c345c
Compare
…ibution
This commit implements portable, self-contained PostgreSQL binaries for
the Supabase CLI across macOS (ARM), Linux (x64), and Linux (ARM64),
along with automated CI/CD workflows for building and releasing these
artifacts.
The Supabase CLI needs to ship PostgreSQL binaries that work on user
machines without requiring Nix or other system dependencies. This means
extracting the actual binaries from Nix's wrapper scripts, bundling all
necessary shared libraries, and patching them to use relative paths
instead of hardcoded Nix store paths.
A `variant` parameter was added to the postgres build pipeline to
distinguish between "full" (all extensions) and "cli" (minimal
extensions for Supabase CLI). The `cliExtensions` list contains 6
extensions required for running Supabase migrations: supautils,
pg_graphql, pgsodium, supabase_vault, pg_net, and pg_cron. Built-in
extensions (uuid-ossp, pgcrypto, pg_stat_statements) are included
automatically with PostgreSQL. `makeOurPostgresPkgs`/`makePostgresBin`
were modified to accept this parameter. A new `psql_17_cli` package is
created using `variant = "cli"`, while the full extension set is
preserved for base packages (`psql_15`, `psql_17`, `psql_orioledb-17`).
The portable CLI variant (`psql_17_cli_portable`) includes 6 extensions
for migration support while maintaining a significantly smaller size
than the full build. The implementation in
`nix/packages/postgres-portable.nix` extracts binaries from
`psql_17_cli` using a `resolve_binary()` function that follows wrapper
layers to find the actual ELF/Mach-O binaries behind Nix's environment
setup scripts.
All Nix-provided libraries (ICU, readline, zlib, etc.) are bundled while
excluding system libraries (`libc`, `libpthread`, `libm`, `glibc`,
`libdl`) that must come from the host. This distinction is critical:
Linux bundles must exclude glibc due to kernel ABI dependencies, while
macOS can include more libs due to its different linking model.
Dependency resolution runs multiple passes to catch transitive deps
(e.g., ICU → charset → etc.).
Platform-specific patching is applied: Linux binaries use the system
interpreter (`/lib64/ld-linux-*.so.2`) and `$ORIGIN`-based RPATHs, while
macOS binaries use `@rpath` with `@executable_path`. Wrapper scripts set
`LD_LIBRARY_PATH` (Linux) or `DYLD_LIBRARY_PATH` (macOS) to find bundled
libraries.
The bundle includes PostgreSQL config templates (`postgresql.conf`,
`pg_hba.conf`, `pg_ident.conf`) tailored for CLI usage with minimal
local dev settings, plus the complete Supabase migration script
(`migrate.sh`) with all init-scripts and migration SQL files (55 files,
236KB).
To achieve true portability, a `portable` parameter was added to the
PostgreSQL build in `nix/postgresql/generic.nix`. When `portable =
true`, three critical hardcoded paths are excluded from the build:
1. `--with-system-tzdata` is removed from configure flags, allowing
PostgreSQL to use bundled timezone data from the `share/` directory
instead of a hardcoded `/nix/store/.../tzdata` path;
2. the `locale-binary-path.patch` is excluded, so PostgreSQL calls
`locale -a` from system PATH rather than using an absolute path to
glibc's locale command;
3. the `postFixup` initdb wrapper is disabled to avoid hardcoding glibc
paths. The `portable` parameter defaults to `false` in
`nix/postgresql/default.nix` but is overridden to `true` for the CLI
variant in `nix/packages/postgres.nix` using `.override { portable =
true; }`.
This ensures standard PostgreSQL builds remain unchanged while the CLI
variant produces truly portable binaries without any `/nix/store`
references.
A GitHub Actions workflow builds portable binaries across all three
platforms using a matrix strategy. Each build runs automated portability
checks that verify no `/nix/store` references remain, validate RPATH
configuration, confirm transitive dependencies are bundled, ensure
system libraries are NOT bundled, and check wrapper scripts contain
proper library path setup.
Post-build testing validates binaries work without Nix (`postgres
--version`, `psql --version`), that Supabase migrations can run
successfully, and that tzdata is available.
On tagged releases (`v*-cli`), the workflow creates GitHub releases with
tarball artifacts and checksums.
The test infrastructure needed significant changes to support variants
with different extension sets. An `isCliVariant` parameter was added to
`makeCheckHarness`, and the hardcoded `shared_preload_libraries` list in
`postgresql.conf.in` was replaced with a `@PRELOAD_LIBRARIES@`
placeholder. A `generatePreloadLibs` script now parses `receipt.json` at
test time and dynamically builds the preload list based on available
extensions, removing the previous timescaledb removal hack for OrioleDB.
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
from
47c345c to
bd4b787
Compare
4 participants
What kind of change does this PR introduce?
Adds a new, portable
psql_17_cli_portabletarget.What is the current behavior?
All current builds are not portable (i.e. they have hard dependencies on the
/nix/store), and they all include the full list of extensions.What is the new behavior?
The
psql_17_cli_portabletarget can run without a/nix/storepresent, and only includes thesupautilsextension.Additional context
Summary by CodeRabbit