feat: Add integration tests for flake-checker #47

martin · 2026-09-12T11:58:08Z

martin commented

2026-09-12 11:58:08 +00:00

No description provided.

martin self-assigned this

2026-09-12 11:58:08 +00:00

martin added 1 commit

2026-09-12 11:58:09 +00:00

feat: Add integration tests for flake-checker

ci/woodpecker/pr/pr-review Pipeline was successful

Details

39ee93bce7

martin requested review from clank-bot

2026-09-12 11:58:09 +00:00

clank-bot commented

2026-09-12 11:58:59 +00:00

Review Summary

This PR adds a fish-based integration test runner that auto-discovers plugins/<name>/integration.yaml files, builds Nix OCI images, loads them into Docker or Podman, and executes pipelines via woodpecker-cli exec. The overall design is solid — auto-discovery eliminates a central registry, and the Docker/Podman dual-backend support is well thought out.

The documentation (AGENTS.md, README.md) and justfile recipe are well-written and consistent. The flake-checker integration test is a reasonable first example.

I found two correctness issues in the Podman branch of scripts/integration-test.fish and one minor robustness concern. The two bugs involve missing exit-status checks that would produce confusing error messages if Nix builds fail.

*🤖 Review by opencode (opencode/mimo-v2.5-free) — verdict: changes

# Review Summary This PR adds a fish-based integration test runner that auto-discovers `plugins/<name>/integration.yaml` files, builds Nix OCI images, loads them into Docker or Podman, and executes pipelines via `woodpecker-cli exec`. The overall design is solid — auto-discovery eliminates a central registry, and the Docker/Podman dual-backend support is well thought out. The documentation (AGENTS.md, README.md) and justfile recipe are well-written and consistent. The `flake-checker` integration test is a reasonable first example. I found two correctness issues in the Podman branch of `scripts/integration-test.fish` and one minor robustness concern. The two bugs involve missing exit-status checks that would produce confusing error messages if Nix builds fail. --- *🤖 Review by opencode (opencode/mimo-v2.5-free) — verdict: **changes**

clank-bot reviewed

2026-09-12 11:58:59 +00:00

clank-bot left a comment

Code review findings from opencode.

Code review findings from opencode.

scripts/integration-test.fish Outdated

					
				@ -0,0 +64,4 @@

				    set -l image "woodpecker-plugin-$plugin"

				    set -l image_version (nix eval --raw "$root#$plugin.version")

				    or fail "could not determine image version for $plugin"

[bug] Missing exit-status check on `nix build "$root#$plugin-image"` in the Podman branch. If this fails, `result` symlink won't exist, and `readlink -f result` will fail with a misleading error. Add `or fail "could not build image for $plugin"` after this line (same as line 63 already does for the Docker branch).

[bug] Missing exit-status check on `nix build "$root#$plugin-image"` in the Podman branch. If this fails, `result` symlink won't exist, and `readlink -f result` will fail with a misleading error. Add `or fail "could not build image for $plugin"` after this line (same as line 63 already does for the Docker branch).

martin marked this conversation as resolved

scripts/integration-test.fish Outdated

					
				@ -0,0 +65,4 @@

				    set -l image "woodpecker-plugin-$plugin"

				    set -l image_version (nix eval --raw "$root#$plugin.version")

				    or fail "could not determine image version for $plugin"

				    set -l versioned_image "$image:$image_version"

[bug] Missing exit-status check on the second `nix build` (copyToDockerDaemon script). If this fails, `skopeo_bin` will be empty and the `fail` check on line 71 catches it, but the error message won't indicate that the Nix build itself failed. Add `or fail "could not build image loader for $plugin"` here for clearer diagnostics.

[bug] Missing exit-status check on the second `nix build` (copyToDockerDaemon script). If this fails, `skopeo_bin` will be empty and the `fail` check on line 71 catches it, but the error message won't indicate that the Nix build itself failed. Add `or fail "could not build image loader for $plugin"` here for clearer diagnostics.

martin marked this conversation as resolved

scripts/integration-test.fish Outdated

					
				@ -0,0 +67,4 @@

				    or fail "could not determine image version for $plugin"

				    set -l versioned_image "$image:$image_version"

				    set -l image_ref "$image:integration"

				    printf '\n==> testing %s\n' $plugin

[style] The skopeo binary path extraction uses a regex on Nix store paths (`/nix/store/[^ ]*skopeo[^ ]*/bin`). This is fragile — the Nix store path format or the script output could change across Nix versions. Consider extracting the skopeo path more robustly (e.g., via `nix-build` with an explicit skopeo reference) or documenting this assumption.

[style] The skopeo binary path extraction uses a regex on Nix store paths (`/nix/store/[^ ]*skopeo[^ ]*/bin`). This is fragile — the Nix store path format or the script output could change across Nix versions. Consider extracting the skopeo path more robustly (e.g., via `nix-build` with an explicit skopeo reference) or documenting this assumption.

martin marked this conversation as resolved

martin force-pushed tests/integration from 39ee93bce7

ci/woodpecker/pr/pr-review Pipeline was successful

Details

to f4843394c4

ci/woodpecker/pr/pr-review Pipeline was successful

Details

2026-09-12 11:59:04 +00:00

Compare

clank-bot reviewed

2026-09-12 11:59:16 +00:00

clank-bot left a comment

Code review findings from opencode.

Code review findings from opencode.

scripts/integration-test.fish Outdated

					
				@ -0,0 +66,4 @@

				    set -l image_version (nix eval --raw "$root#$plugin.version")

				    or fail "could not determine image version for $plugin"

				    set -l versioned_image "$image:$image_version"

				    set -l image_ref "$image:integration"

[bug] Missing exit-status check on `nix build "$root#$plugin-image"` in the Podman branch (line 69). If this build fails, the `result` symlink won't exist and `readlink -f result` on line 71 will fail with a misleading error. Add `or fail "could not build image for $plugin"` after line 69, matching the pattern used in the Docker branch.

[bug] Missing exit-status check on `nix build "$root#$plugin-image"` in the Podman branch (line 69). If this build fails, the `result` symlink won't exist and `readlink -f result` on line 71 will fail with a misleading error. Add `or fail "could not build image for $plugin"` after line 69, matching the pattern used in the Docker branch.

martin marked this conversation as resolved

scripts/integration-test.fish Outdated

					
				@ -0,0 +69,4 @@

				    set -l image_ref "$image:integration"

				    printf '\n==> testing %s\n' $plugin

				    nix build "$root#$plugin-image"

[bug] Missing exit-status check on `nix build "$root#$plugin-image.copyToDockerDaemon"` (line 72). If this fails, `skopeo_bin` extraction will fail with a confusing error. Add `or fail "could not build image loader for $plugin"` after this line for clearer diagnostics.

[bug] Missing exit-status check on `nix build "$root#$plugin-image.copyToDockerDaemon"` (line 72). If this fails, `skopeo_bin` extraction will fail with a confusing error. Add `or fail "could not build image loader for $plugin"` after this line for clearer diagnostics.

martin marked this conversation as resolved

scripts/integration-test.fish Outdated

					
				@ -0,0 +70,4 @@

				    printf '\n==> testing %s\n' $plugin

				    nix build "$root#$plugin-image"

[style] The skopeo binary path extraction via regex (`/nix/store/[^ ]*skopeo[^ ]*/bin`) is fragile across Nix versions. Consider documenting this assumption or using a more robust extraction method (e.g., `nix-build` with an explicit skopeo reference).

[style] The skopeo binary path extraction via regex (`/nix/store/[^ ]*skopeo[^ ]*/bin`) is fragile across Nix versions. Consider documenting this assumption or using a more robust extraction method (e.g., `nix-build` with an explicit skopeo reference).

martin marked this conversation as resolved

martin force-pushed tests/integration from f4843394c4

ci/woodpecker/pr/pr-review Pipeline was successful

Details

to f6b12b86ac

ci/woodpecker/pr/pr-review Pipeline was successful

Details

2026-09-12 17:38:41 +00:00

Compare

clank-bot reviewed

2026-09-12 17:39:09 +00:00

clank-bot left a comment

Code review findings from opencode.

Code review findings from opencode.

scripts/integration-test.fish

[bug] Missing exit-status check on `nix build "$root#$plugin-image.copyToDockerDaemon"`. If this build fails, the `readlink -f` on line 242 will fail with a misleading error about the symlink not existing. Add `or fail "could not build image loader for $plugin"` after this line.

[bug] Missing exit-status check on `nix build "$root#$plugin-image.copyToDockerDaemon"`. If this build fails, the `readlink -f` on line 242 will fail with a misleading error about the symlink not existing. Add `or fail "could not build image loader for $plugin"` after this line.

martin marked this conversation as resolved

scripts/integration-test.fish

[style] The skopeo binary path extraction via `nix-store --query --references` assumes a specific Nix store path format. While functional, this is fragile across Nix versions. Consider documenting this assumption or using a more direct approach like `nix build nixpkgs#skopeo` to get skopeo separately.

[style] The skopeo binary path extraction via `nix-store --query --references` assumes a specific Nix store path format. While functional, this is fragile across Nix versions. Consider documenting this assumption or using a more direct approach like `nix build nixpkgs#skopeo` to get skopeo separately.

martin marked this conversation as resolved

martin added 1 commit

2026-09-12 17:44:46 +00:00

fix: More exit codes check and simplify skopeo

ci/woodpecker/pr/pr-review Pipeline was successful

Details

71546d4812

clank-bot reviewed

2026-09-12 17:45:18 +00:00

clank-bot left a comment

Code review findings from opencode.

Code review findings from opencode.

scripts/integration-test.fish

[bug] Missing exit-status check on `nix build "$root#$plugin-image"` in the Podman branch. If this build fails, `result` symlink won't exist, and `readlink -f result` on line 242 will fail with a misleading error. Add `or fail "could not build image for $plugin"` after this line, matching the pattern used in the Docker branch.

[bug] Missing exit-status check on `nix build "$root#$plugin-image"` in the Podman branch. If this build fails, `result` symlink won't exist, and `readlink -f result` on line 242 will fail with a misleading error. Add `or fail "could not build image for $plugin"` after this line, matching the pattern used in the Docker branch.

scripts/integration-test.fish

[bug] Missing exit-status check on `nix build "$root#$plugin-image.copyToDockerDaemon"`. If this build fails, the subsequent `readlink -f result` will fail with a misleading error about the symlink not existing. Add `or fail "could not build image loader for $plugin"` after this line for clearer diagnostics.

[bug] Missing exit-status check on `nix build "$root#$plugin-image.copyToDockerDaemon"`. If this build fails, the subsequent `readlink -f result` will fail with a misleading error about the symlink not existing. Add `or fail "could not build image loader for $plugin"` after this line for clearer diagnostics.

scripts/integration-test.fish

[performance] `readlink -f result` relies on the `result` symlink in the current working directory, which is fragile if the working directory is not the repository root. Consider using `readlink -f "$root/result"` for robustness, or explicitly `cd` to the expected location before this block.

[performance] `readlink -f result` relies on the `result` symlink in the current working directory, which is fragile if the working directory is not the repository root. Consider using `readlink -f "$root/result"` for robustness, or explicitly `cd` to the expected location before this block.