feat(build_transformer): add Caddy file server support with JSON API#304
feat(build_transformer): add Caddy file server support with JSON API#304
Conversation
Add a dedicated CaddyArtefactDownloader that uses Caddy's JSON directory listing API instead of HTML scraping. This provides more reliable and structured file discovery for KDF artifact downloads. Changes: - Add CaddyArtefactDownloader with JSON API support (Accept: application/json) - Update ArtefactDownloaderFactory to detect and route Caddy-hosted mirrors - Add --source caddy option to update_api_config CLI tool - Fix PowerShell Expand-Archive invocation on Windows (requires -Command flag) - Update BUILD_CONFIG_README with Caddy mirror documentation The Caddy JSON API returns structured directory listings which is more robust than parsing HTML. Recursive directory search is supported up to 3 levels deep.
|
Visit the preview URL for this PR (updated for commit 6758fed): https://komodo-playground--pr304-feat-caddy-mirror-su-j214jna9.web.app (expires Thu, 18 Dec 2025 10:36:27 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 2bfedd77fdea45b25ba7c784416e81f177aa5c47 |
There was a problem hiding this comment.
Pull request overview
This PR adds support for Caddy-based file servers with JSON API directory listings as an alternative to HTML-based mirror scraping and GitHub releases. It introduces a new CaddyArtefactDownloader class that leverages Caddy's structured JSON API for more reliable artifact discovery, routing for the devbuilds.gleec.com mirror, and fixes PowerShell extraction commands across all downloader implementations.
Key Changes:
- Added
CaddyArtefactDownloaderthat fetches directory listings via JSON API with recursive subdirectory search - Updated
ArtefactDownloaderFactoryto detect Caddy hosts and route appropriately - Fixed PowerShell
Expand-Archiveinvocation to use-Commandflag with single-string format across all downloaders
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
packages/komodo_wallet_cli/bin/update_api_config.dart |
Added caddy source option to CLI, updated help text and examples, added _fetchCaddyDownloadUrl and _searchCaddyDirectory methods for JSON API interaction |
packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/caddy_artefact_downloader.dart |
New file implementing CaddyArtefactDownloader with JSON API support, including CaddyFileEntry model and recursive directory search with hash matching |
packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/artefact_downloader_factory.dart |
Added _caddyHosts whitelist and _isCaddyServer detection method, updated factory logic to route Caddy URLs to CaddyArtefactDownloader |
packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/github_artefact_downloader.dart |
Fixed PowerShell extraction command to use -Command flag with properly quoted paths |
packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/dev_builds_artefact_downloader.dart |
Fixed PowerShell extraction command to use -Command flag with properly quoted paths |
packages/komodo_defi_framework/app_build/BUILD_CONFIG_README.md |
Updated documentation to explain three mirror types (GLEEC DevBuilds with JSON API, Nebula HTML, Komodo DevBuilds HTML), added clarity about Caddy-based servers |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| '-Command', | ||
| 'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force', |
There was a problem hiding this comment.
On Windows, filePath and destinationFolder are interpolated directly into a PowerShell -Command string in Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force, which allows PowerShell expression injection if an attacker can influence the archive filename or path (e.g., via $() in a downloaded asset name). Because these paths ultimately derive from remote URLs (GitHub release assets), a compromised or malicious source could craft a filename that executes arbitrary PowerShell code when this command runs. To fix this, avoid embedding untrusted values in a PowerShell script string and instead either use a non-shell unzip library in Dart or invoke PowerShell with parameters that are not subject to string interpolation (e.g., use a small wrapper script with param() and pass paths via -File/-ArgumentList).
| final result = await Process.run('powershell', [ | ||
| 'Expand-Archive', | ||
| '-Path', | ||
| filePath, | ||
| '-DestinationPath', | ||
| destinationFolder, | ||
| '-Command', | ||
| 'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force', |
There was a problem hiding this comment.
On Windows, filePath and destinationFolder are interpolated directly into a PowerShell -Command string in Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force, which enables PowerShell expression injection if an attacker can control the archive filename or path (e.g., by including $() in a downloaded file name). Here those paths come from remote HTTP downloads (dev builds mirrors), so a compromised or malicious mirror could craft filenames that execute arbitrary PowerShell commands when extraction runs. Mitigate this by not constructing a script string from untrusted data—either use a native unzip library in Dart or a PowerShell wrapper that accepts paths as parameters (via -File/-ArgumentList) without double-quoted interpolation.
| final result = await Process.run('powershell', [ | ||
| '-Command', | ||
| 'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force', |
There was a problem hiding this comment.
On Windows, filePath and destinationFolder are interpolated directly into a PowerShell -Command string in Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force, which allows PowerShell expression injection if a Caddy-hosted artifact filename or path can be influenced by an attacker (e.g., including $() in the url/basename returned by the JSON listing). Since Caddy directory listings and download URLs are driven by remote data, a compromised or malicious Caddy server could supply filenames that execute arbitrary PowerShell code when this extraction step runs. To harden this, avoid embedding untrusted paths in a PowerShell script string and instead use a safer extraction mechanism (e.g., a Dart unzip library or a PowerShell script invoked via -File with paths passed as arguments rather than interpolated into double-quoted strings).
Summary
CaddyArtefactDownloaderthat uses Caddy's JSON directory listing API instead of HTML scrapingArtefactDownloaderFactoryto detect and route Caddy-hosted mirrors (e.g.,devbuilds.gleec.com)--source caddyoption toupdate_api_configCLI tool for Caddy mirror supportExpand-Archiveinvocation on Windows (requires-Commandflag)Details
Caddy JSON API
Caddy provides structured JSON directory listings when the
Accept: application/jsonheader is included:[ { "name": "kdf_9e6f4e5-wasm.zip", "size": 11739893, "url": "./kdf_9e6f4e5-wasm.zip", "mod_time": "2025-12-10T20:37:05Z", "is_dir": false, "is_symlink": false } ]This is more reliable than HTML scraping and provides structured metadata.
Windows PowerShell Fix
The existing PowerShell extraction code was incorrect - cmdlets like
Expand-Archivemust be passed via the-Commandflag:CLI Usage
Test plan
1. Static analysis
cd packages/komodo_wallet_build_transformer dart analyze2. Test CLI with Caddy source
3. Verify build with downloaded artifacts
4. Test Windows extraction (manual)