ci: use charmcraft test for example charm by tonyandrewmeyer · Pull Request #2440 · canonical/operator

tonyandrewmeyer · 2026-04-21T01:27:49Z

This PR updates the one example charm (httpbin-demo) to use charmcraft test (which is spread, underneath).

The charm is updated using charmcraft init --profile test-kubernetes --force. There are changes coming to Charmcraft in regards to "craft test" in 26.10, so for now it seems that using the existing profile setup is the best move. This is what creates the spread.yaml, spread/.extension (unchanged), and a default task (that I've replaced).
I've changed the default suite from spread/deploy to spread/integration.
I've changed the inline juju/lxd/microk8s install to use concierge prepare, installed astral-uv and tox, extended the exclude to include more Python artefacts, and increased the kill-timeout to 90s. This is probably the most debatable point -- perhaps it would be better to remain closer to what charmcraft provides, but I think we should stick with our recommended tooling, particularly Concierge.
There's a task.yaml for each Python integration test module (one, for this example charm, but set up in a way that supports more). These are pretty simple, taking the packed charm path from Charmcraft and running tox -e integration with the specific module.
A new example charms integration test workflow is added to collect (discover) the spread (charmcraft test) jobs, and use that to set up the matrix for testing. This means when a new task.yaml is added, it will automatically be added and run concurrently via the native GitHub functionality.

tox -e integration does not change in this PR. It's available for use, and still needs a local Juju controller set up, and something to do the packing.

I've added small notes to the docs as well. I don't think we want to have too much more than this at the moment, since the profiles and charmcraft test are still marked as experimental (even though they are somehow also considered to be used by everyone), and because we know that there are naming (and maybe other) changes coming in 26.10. I'd like input on whether having this extra setup in the tutorial charms particularly is too inconsistent with what you'd get strictly following the tutorial.

Preview for the CI how-to, preview for the integration test how-to, preview for the machine tutorial (the tip box), preview for the K8s tutorial start (again the tip box).

I've left the integration tests unchanged, but I wonder if the final K8s tutorial one should be changed to use a separate module for the COS test, to demonstrate that aspect working?

Example CI run in my fork.

I wonder if Concierge should learn how to handle this? Why does Concierge normally work but isn't via spread?

…rking on ubuntu-latest.

…nd networking on ubuntu-latest." This reverts commit c7892ee.

james-garner-canonical

I'm a bit wary about adding this to the tutorial charms without updating the tutorial as a whole to explain how/if you should use it. I wonder if it would be better to just do it for the httpbin charm currently. I guess the downside is that we'd need a separate example charm integration test workflow in that case -- but sounds like that's what you're doing separately.

tonyandrewmeyer · 2026-04-23T11:08:52Z

@dwilding @james-garner-canonical I've shrunk this down as discussed.

dwilding

I think this scope is much better.

I haven't finished reviewing, but I need to pause for today.

dwilding · 2026-04-24T05:15:26Z

+(write-integration-tests-for-a-charm-split-across-modules)=
+### Split tests across modules
+
+As your suite grows, split your integration tests across several `test_*.py` modules, grouped by feature. Tests within a module share a single `juju` fixture (and therefore a single Juju model), so keep tests that need to build on each other together. Across modules, each gets a fresh model, which makes modules independent of each other.


Suggested change

As your suite grows, split your integration tests across several `test_*.py` modules, grouped by feature. Tests within a module share a single `juju` fixture (and therefore a single Juju model), so keep tests that need to build on each other together. Across modules, each gets a fresh model, which makes modules independent of each other.

As your suite grows, split your integration tests across several `test_*.py` modules, grouped by feature. Tests within a module share a single `juju` fixture (and therefore a single Juju model), so keep tests that need to build on each other together. Each module gets a fresh model, which makes modules independent of each other.

dwilding · 2026-04-24T05:18:39Z

+A common split:
+
+- `test_charm.py` — smoke tests: pack, deploy, and check the charm reaches active status.
+- `test_<feature>.py` — one module per feature area (for example, `test_backups.py`, `test_tls.py`, `test_upgrade.py`, `test_scaling.py`).


I think the meaning is clear enough from the examples.

Suggested change

- `test_<feature>.py` — one module per feature area (for example, `test_backups.py`, `test_tls.py`, `test_upgrade.py`, `test_scaling.py`).

- `test_<feature>.py` — for example, `test_backups.py`, `test_tls.py`, `test_upgrade.py`, `test_scaling.py`.

dwilding · 2026-04-24T05:23:40Z

+- `test_charm.py` — smoke tests: pack, deploy, and check the charm reaches active status.
+- `test_<feature>.py` — one module per feature area (for example, `test_backups.py`, `test_tls.py`, `test_upgrade.py`, `test_scaling.py`).
+
+The main reason to split is **parallel CI execution**: each module can run as a separate job, so total wall-clock time is governed by the slowest module rather than the sum of all modules. `tox -e integration` still runs every module sequentially on a single machine; it's the CI matrix (see {ref}`set-up-ci-integration`) that turns module boundaries into parallel jobs. Adding a new `test_*.py` file then automatically adds a new CI job — no workflow changes needed.


I found this a bit confusing. The first example in our CI doc doesn't have a matrix, so new jobs wouldn't automatically be created for each test_*.py file, right?

My interpretation is more like this: Use separate modules so that you have the option of using a matrix in CI. If you're going to do that, we recommend using the charmcraft test approach described in the CI doc.

dwilding · 2026-04-24T05:24:48Z

+
+The main reason to split is **parallel CI execution**: each module can run as a separate job, so total wall-clock time is governed by the slowest module rather than the sum of all modules. `tox -e integration` still runs every module sequentially on a single machine; it's the CI matrix (see {ref}`set-up-ci-integration`) that turns module boundaries into parallel jobs. Adding a new `test_*.py` file then automatically adds a new CI job — no workflow changes needed.
+
+For real-world examples of module-per-feature splits, see:


I think this would be easier to read.

Suggested change

For real-world examples of module-per-feature splits, see:

For real-world examples of split tests, see:

dwilding · 2026-04-24T05:25:25Z

+- [postgresql-k8s-operator](https://github.com/canonical/postgresql-k8s-operator/tree/main/tests/integration) — the Kubernetes equivalent.
+- [opensearch-operator](https://github.com/canonical/opensearch-operator/tree/main/tests/integration) — larger suite with per-cloud backup modules.
+
+For a minimal scaffold that you can lift into your own charm, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm in the Ops repository.


Personal preference perhaps 🙂

Suggested change

For a minimal scaffold that you can lift into your own charm, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm in the Ops repository.

For a minimal scaffold that you can use in your own charm, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm in the Ops repository.

dwilding · 2026-04-24T05:34:07Z

+(set-up-ci-charmcraft-test)=
+## Run integration tests in parallel with `charmcraft test`
+
+If you initialised your charm with `charmcraft init --profile test-machine` or `--profile test-kubernetes` (both currently experimental), your charm includes a `spread.yaml` and one `spread/integration/<module>/task.yaml` per test module. You can use `charmcraft test` in CI to run each module as its own matrix job, so total wall-clock time is bounded by the slowest module rather than the sum of all modules. Adding a new `test_*.py` module — along with its `task.yaml` — automatically adds a new CI job.


This is quite dense. How about expanding it (visually) and with a bit more context on spread:

Suggested change

If you initialised your charm with `charmcraft init --profile test-machine` or `--profile test-kubernetes` (both currently experimental), your charm includes a `spread.yaml` and one `spread/integration/<module>/task.yaml` per test module. You can use `charmcraft test` in CI to run each module as its own matrix job, so total wall-clock time is bounded by the slowest module rather than the sum of all modules. Adding a new `test_*.py` module — along with its `task.yaml` — automatically adds a new CI job.

If you initialised your charm with `charmcraft init --profile test-machine` or `--profile test-kubernetes` (both currently experimental), your charm includes extra testing machinery:

- A [spread](https://github.com/canonical/spread) configuration file called `spread.yaml`.

- One file `spread/integration/<module>/task.yaml` per test module.

You can use `charmcraft test` in CI to run each module as its own matrix job, so total wall-clock time is bounded by the slowest module rather than the sum of all modules. Adding a new `test_*.py` module — along with its `task.yaml` — automatically adds a new CI job.

I'd also consider moving the sentence about automatically adding CI jobs. I'll make a separate suggestion about that.

dwilding · 2026-04-24T05:41:40Z

+        run: charmcraft test "craft:ubuntu-24.04:spread/integration/${{ matrix.task }}"
+```
+
+For a complete workflow that discovers modules dynamically (no hard-coded matrix), see the Ops repository's [example-charm-charmcraft-test.yaml](https://github.com/canonical/operator/blob/main/.github/workflows/example-charm-charmcraft-test.yaml). For the matching charm-side files, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm.


Since the example in the doc doesn't show how to discover modules, I'd move the earlier sentence here. Something like this:

Suggested change

For a complete workflow that discovers modules dynamically (no hard-coded matrix), see the Ops repository's [example-charm-charmcraft-test.yaml](https://github.com/canonical/operator/blob/main/.github/workflows/example-charm-charmcraft-test.yaml). For the matching charm-side files, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm.

It's also possible to discover modules dynamically (no hard-coded matrix), so that when you add a `test_*.py` module and corresponding `task.yaml` file, you automatically get a new CI job.

For an example, see the Ops repository's [example-charm-charmcraft-test.yaml workflow](https://github.com/canonical/operator/blob/main/.github/workflows/example-charm-charmcraft-test.yaml). This workflow runs integration tests for the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm.

tonyandrewmeyer added 10 commits April 21, 2026 13:27

ci: use charmcraft test for example charms.

b448397

Merge branch 'main' into craft-test-in-example-charms

73d6900

Fix the lxd issue.

a44231a

I wonder if Concierge should learn how to handle this? Why does Concierge normally work but isn't via spread?

Try with 22.04, I think the issue is the known one with lxd and netwo…

c7892ee

…rking on ubuntu-latest.

Revert "Try with 22.04, I think the issue is the known one with lxd a…

63abbcb

…nd networking on ubuntu-latest." This reverts commit c7892ee.

Try using the action to get lxd

65bb211

Attempt to fix the tests.

982b514

More test tweaks.

d635c93

Fix path issue.

957753f

Explain things in the docs.

3f52ce9

tonyandrewmeyer requested review from dwilding and james-garner-canonical April 22, 2026 08:05

tonyandrewmeyer marked this pull request as ready for review April 22, 2026 08:18

dwilding mentioned this pull request Apr 22, 2026

docs: align example charms with pre-release Charmcraft #2444

Open

james-garner-canonical reviewed Apr 23, 2026

View reviewed changes

Comment thread .../from-zero-to-hero-write-your-first-kubernetes-charm/integrate-your-charm-with-postgresql.md Outdated

Comment thread ...ial/from-zero-to-hero-write-your-first-kubernetes-charm/create-a-minimal-kubernetes-charm.md Outdated

Shrink scope, per review.

6d2708d

tonyandrewmeyer commented Apr 23, 2026

View reviewed changes

Comment thread ...ial/from-zero-to-hero-write-your-first-kubernetes-charm/create-a-minimal-kubernetes-charm.md Outdated

tonyandrewmeyer added 4 commits April 23, 2026 22:50

Apply suggestion from @tonyandrewmeyer

dad4507

Merge branch 'main' into craft-test-in-example-charms

41585b1

Fix revert.

11ed8ef

Tweak tip per review.

8327ea8

tonyandrewmeyer changed the title ~~ci: use charmcraft test for example charms~~ ci: use charmcraft test for example charm Apr 24, 2026

Apply suggestion from @tonyandrewmeyer

77af250

tonyandrewmeyer commented Apr 24, 2026

View reviewed changes

Comment thread .../from-zero-to-hero-write-your-first-kubernetes-charm/integrate-your-charm-with-postgresql.md Outdated

Apply suggestion from @tonyandrewmeyer

3099c4c

dwilding reviewed Apr 24, 2026

View reviewed changes

Comment thread .github/workflows/example-charm-charmcraft-test.yaml

dwilding reviewed Apr 24, 2026

View reviewed changes

	As your suite grows, split your integration tests across several `test_*.py` modules, grouped by feature. Tests within a module share a single `juju` fixture (and therefore a single Juju model), so keep tests that need to build on each other together. Across modules, each gets a fresh model, which makes modules independent of each other.
	As your suite grows, split your integration tests across several `test_*.py` modules, grouped by feature. Tests within a module share a single `juju` fixture (and therefore a single Juju model), so keep tests that need to build on each other together. Each module gets a fresh model, which makes modules independent of each other.

	- `test_<feature>.py` — one module per feature area (for example, `test_backups.py`, `test_tls.py`, `test_upgrade.py`, `test_scaling.py`).
	- `test_<feature>.py` — for example, `test_backups.py`, `test_tls.py`, `test_upgrade.py`, `test_scaling.py`.


		The main reason to split is parallel CI execution: each module can run as a separate job, so total wall-clock time is governed by the slowest module rather than the sum of all modules. `tox -e integration` still runs every module sequentially on a single machine; it's the CI matrix (see {ref}`set-up-ci-integration`) that turns module boundaries into parallel jobs. Adding a new `test_*.py` file then automatically adds a new CI job — no workflow changes needed.

		For real-world examples of module-per-feature splits, see:

	For real-world examples of module-per-feature splits, see:
	For real-world examples of split tests, see:

	For a minimal scaffold that you can lift into your own charm, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm in the Ops repository.
	For a minimal scaffold that you can use in your own charm, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm in the Ops repository.

-If you initialised your charm with `charmcraft init --profile test-machine` or `--profile test-kubernetes` (both currently experimental), your charm includes a `spread.yaml` and one `spread/integration/<module>/task.yaml` per test module. You can use `charmcraft test` in CI to run each module as its own matrix job, so total wall-clock time is bounded by the slowest module rather than the sum of all modules. Adding a new `test_*.py` module — along with its `task.yaml` — automatically adds a new CI job.
+If you initialised your charm with `charmcraft init --profile test-machine` or `--profile test-kubernetes` (both currently experimental), your charm includes extra testing machinery:
+- A [spread](https://github.com/canonical/spread) configuration file called `spread.yaml`.
+- One file `spread/integration/<module>/task.yaml` per test module.
+You can use `charmcraft test` in CI to run each module as its own matrix job, so total wall-clock time is bounded by the slowest module rather than the sum of all modules. Adding a new `test_*.py` module — along with its `task.yaml` — automatically adds a new CI job.

-For a complete workflow that discovers modules dynamically (no hard-coded matrix), see the Ops repository's [example-charm-charmcraft-test.yaml](https://github.com/canonical/operator/blob/main/.github/workflows/example-charm-charmcraft-test.yaml). For the matching charm-side files, see the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm.
+It's also possible to discover modules dynamically (no hard-coded matrix), so that when you add a `test_*.py` module and corresponding `task.yaml` file, you automatically get a new CI job.
+For an example, see the Ops repository's [example-charm-charmcraft-test.yaml workflow](https://github.com/canonical/operator/blob/main/.github/workflows/example-charm-charmcraft-test.yaml). This workflow runs integration tests for the [httpbin-demo](https://github.com/canonical/operator/tree/main/examples/httpbin-demo) example charm.

Conversation

tonyandrewmeyer commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

james-garner-canonical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonyandrewmeyer commented Apr 23, 2026

Uh oh!

Uh oh!

Uh oh!

dwilding left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tonyandrewmeyer commented Apr 21, 2026 •

edited

Loading