test: add smoke tests for skills and plugin installs#54
Open
maksymilian-majer wants to merge 1 commit intocallstackincubator:mainfrom
Open
test: add smoke tests for skills and plugin installs#54maksymilian-majer wants to merge 1 commit intocallstackincubator:mainfrom
maksymilian-majer wants to merge 1 commit intocallstackincubator:mainfrom
Conversation
c20dcab to
b753115
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a Bun + TypeScript smoke-test suite for repository plumbing and distribution paths, explicitly separated from future skill evals.
It introduces deterministic coverage for:
SKILL.mdfrontmatter parsingIt also adds a GitHub Actions workflow that runs the smoke suite twice:
skillsCLI versionskills@latestas a lightweight forward-compatibility signalWhat changed
tests/manifest-smoke.test.tstests/codex-plugin-install-smoke.test.tstests/claude-layout-smoke.test.tstests/skill-install-smoke.test.tstests/helpers/.github/workflows/smoke-tests.ymlREADME.mdwith local validation guidanceWhy this is useful
This PR closes deterministic gaps that have already regressed in this repository:
SKILL.mdfrontmatter (725477b)590b981)The new suite is meant to catch repository-shape and installability regressions early, without using model credentials.
GitHub issues addressed
Most directly addressed
#49Codex plugin installThe new smoke tests now verify:
.agents/plugins/marketplace.jsonHOME.codex/pluginsPartially addressed
#27Unable to install skills for GeminiThis PR does not add Gemini-specific tests, but it does reduce the underlying repository-shape risk by validating:
SKILL.mddiscovery through a real installer pathExplicitly out of scope here
#36Hermes polyfills guidanceThat is a content-correctness / eval problem, not a smoke-test problem. This PR intentionally keeps smoke tests focused on packaging, metadata, installation, and distribution behavior.
Validation
Locally:
Previously validated on the working branch via GitHub Actions
workflow_dispatchbefore squashing, including pinned andlatestskillsCLI runs.Notes
evals/work.skillssmoke coverage is deterministic by default through a pinned CLI, while still checkingskills@latestin the same workflow.