Skip to content

test: add smoke tests for skills and plugin installs#54

Open
maksymilian-majer wants to merge 1 commit intocallstackincubator:mainfrom
maksymilian-majer:maks/smoke-tests-pr
Open

test: add smoke tests for skills and plugin installs#54
maksymilian-majer wants to merge 1 commit intocallstackincubator:mainfrom
maksymilian-majer:maks/smoke-tests-pr

Conversation

@maksymilian-majer
Copy link
Copy Markdown

@maksymilian-majer maksymilian-majer commented Apr 8, 2026

Summary

This PR adds a Bun + TypeScript smoke-test suite for repository plumbing and distribution paths, explicitly separated from future skill evals.

It introduces deterministic coverage for:

  • SKILL.md frontmatter parsing
  • metadata drift between shipped skills/plugins and marketplace manifests
  • Codex plugin installation in project and pseudo-global modes
  • Claude Code skill layout validation
  • standalone skill installation from a GitHub-style repo source

It also adds a GitHub Actions workflow that runs the smoke suite twice:

  • once against the pinned local skills CLI version
  • once against skills@latest as a lightweight forward-compatibility signal

What changed

  • added tests/manifest-smoke.test.ts
  • added tests/codex-plugin-install-smoke.test.ts
  • added tests/claude-layout-smoke.test.ts
  • added tests/skill-install-smoke.test.ts
  • added shared helpers under tests/helpers/
  • added .github/workflows/smoke-tests.yml
  • updated README.md with local validation guidance

Why this is useful

This PR closes deterministic gaps that have already regressed in this repository:

  • invalid SKILL.md frontmatter (725477b)
  • missing Claude marketplace entries (590b981)
  • install and marketplace drift outside skill bodies

The new suite is meant to catch repository-shape and installability regressions early, without using model credentials.

GitHub issues addressed

Most directly addressed

  • #49 Codex plugin install

The new smoke tests now verify:

  • the Codex plugin installer can build and run in CI
  • project install writes .agents/plugins/marketplace.json
  • pseudo-global install works with fake HOME
  • plugin payload is copied into .codex/plugins
  • marketplace entries are rewritten to installed local paths
  • repeated installs merge cleanly and remain stable

Partially addressed

  • #27 Unable to install skills for Gemini

This PR does not add Gemini-specific tests, but it does reduce the underlying repository-shape risk by validating:

  • standalone skill discovery from a GitHub-style repo source
  • project/global install layouts
  • SKILL.md discovery through a real installer path

Explicitly out of scope here

  • #36 Hermes polyfills guidance

That is a content-correctness / eval problem, not a smoke-test problem. This PR intentionally keeps smoke tests focused on packaging, metadata, installation, and distribution behavior.

Validation

Locally:

bun install
bun test
SMOKE_SKILLS_CLI_CHANNEL=latest bun test

Previously validated on the working branch via GitHub Actions workflow_dispatch before squashing, including pinned and latest skills CLI runs.

Notes

  • The smoke suite is intentionally separate from future evals/ work.
  • skills smoke coverage is deterministic by default through a pinned CLI, while still checking skills@latest in the same workflow.
  • Codex plugin smoke rebuilds the CLI before running install checks, so local runs do not accidentally validate stale bundled output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant