Kubernetes cluster using Nixidy and ArgoCD
  • Nix 85.8%
  • Just 14.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Martin Berg Alstad c08c9f4710
All checks were successful
ci/woodpecker/cron/flake-check Pipeline was successful
Add personal-services System catalog descriptor (#87)
feat(catalog): define personal-services System

Co-authored-by: OpenCode <opencode@anoma.ly>

Reviewed-on: #87
2026-11-21 22:13:37 +00:00
.woodpecker chore(deps): update docker.io/alpine docker tag to v3.24.4 (#84) 2026-11-16 20:04:27 +00:00
env chore(deps): lock file maintenance (#85) 2026-11-16 19:57:01 +00:00
lib feat: Init argo rollouts (#52) 2026-09-13 16:00:01 +00:00
manifests chore(deps): lock file maintenance (#85) 2026-11-16 19:57:01 +00:00
modules chore: document clusterRepo provenance 2026-11-15 21:16:48 +01:00
tests chore(deps): update dependency arnarg/nixidy to v0.23.1 (#61) 2026-10-18 17:34:54 +00:00
.gitignore Replace beads with Forgejo issues 2026-11-07 22:22:41 +00:00
.sops.yaml feat: Add cluster age key and document (#46) 2026-07-31 21:34:36 +00:00
AGENTS.md chore: retarget GitOps repo to Personal/cluster (#81) 2026-11-15 20:13:08 +00:00
catalog-info.yaml Add personal-services System catalog descriptor (#87) 2026-11-21 22:13:37 +00:00
flake.lock chore(deps): lock file maintenance (#85) 2026-11-16 19:57:01 +00:00
flake.nix Replace beads with Forgejo issues 2026-11-07 22:22:41 +00:00
justfile feat: Add cluster age key and document (#46) 2026-07-31 21:34:36 +00:00
README.md docs: fix stale Getting Started environment docs (#83) 2026-11-16 19:58:20 +00:00
renovate.json Migration: renovate-config (#64) 2026-10-25 17:58:25 +00:00
treefmt.nix Add statix checker to treefmt 2026-05-15 19:34:45 +00:00

Kubernetes Cluster - Nixidy + ArgoCD

GitOps-managed Kubernetes cluster using Nixidy for type-safe manifest generation and ArgoCD for continuous deployment.

Overview

This repository implements the "Rendered Manifests Pattern":

  • Nix configuration (env/{dev,infra,prod}/*.nix) defines all Kubernetes resources in a type-safe way
  • Nixidy generates plain YAML manifests and commits them to manifests/{dev,infra,prod}/
  • ArgoCD watches the manifest directories and automatically syncs changes to the cluster
  • App-of-apps pattern: ArgoCD manages itself and all other applications

Key Benefits: Type safety, reproducible builds, Git-based audit trail, declarative infrastructure.

Repository Structure

cluster/
├── flake.nix                   # Nix flake with dependencies and env configuration
├── flake.lock                  # Locked dependency versions
├── justfile                    # Common operations (run `just` to see all)
├── treefmt.nix                 # Formatter configuration
├── modules/
│   ├── common.nix              # Global networking options + nixidy defaults + port assertion
│   ├── templates.nix           # Reusable application templates (webApplication, route)
│   └── generators.nix          # CRD resource option generators (Traefik, Prometheus Operator)
├── lib/
│   └── default.nix             # lib.my.mkApplication, lib.my.addManagedByLabel, lib.my.checkServicePortUniqueness
├── tests/
│   ├── default.nix             # Unit tests (nix-unit, all systems)
│   └── integration.nix         # Integration tests (x86_64-linux only)
├── env/
│   ├── dev/                    # Dev application definitions (auto-discovered by import-tree)
│   │   ├── *.nix               # Application configs (argocd, traefik, etc.)
│   │   └── *.sops.yaml         # Encrypted secrets
│   ├── infra/                  # Infrastructure application definitions
│   │   ├── *.nix               # Infrastructure apps (monitoring, logging, etc.)
│   │   └── *.sops.yaml         # Encrypted secrets
│   └── prod/                   # Prod application definitions
│       └── *.nix
└── manifests/
    ├── dev/                    # Generated YAML manifests for dev (committed, do not edit)
    │   ├── apps-dev/           # ArgoCD Applications for this environment (the app-of-apps children)
    │   ├── argocd/             # Per-application manifests, one directory per app
    │   └── */                  # e.g. traefik/, toolbox/, martials.no/
    ├── infra/                  # Generated YAML manifests for infra (committed, do not edit)
    │   ├── apps-infra/         # ArgoCD Applications for this environment
    │   └── */                  # e.g. prometheus-stack/, argo-rollouts/
    └── prod/                   # Generated YAML manifests for prod (committed, do not edit)
        ├── apps-prod/          # ArgoCD Applications for this environment
        └── */                  # e.g. martials.no/, recurring-events/

Prerequisites

  • Nix with flakes enabled
  • Kubernetes cluster (k3s recommended)
  • SOPS with age keys for secrets management (optional)

Ingress & Routing

The cluster uses Traefik as the ingress controller, deployed via Helm chart.

Configuration

  • LoadBalancer IP: 192.168.10.228 (configured in modules/common.nix)
  • Domain (dev): *.dev.martials.no / Domain (infra): *.infra.martials.no / Domain (prod): *.martials.no (configured in modules/common.nix)
  • Entry Points:
    • web: HTTP (port 80)
    • websecure: HTTPS (port 443)

Routing Methods

Traefik IngressRoute (type-safe Nix resources preferred; YAML also supported):

resources.ingressRoutes.myapp-route.spec = {
  entryPoints = [ "web" ];
  routes = [
    {
      match = "Host(`myapp.dev.martials.no`)";
      kind = "Rule";
      services = [ { name = "myapp-svc"; port = 80; } ];
    }
  ];
};

See toolbox.nix for a complete example.

Customizing Networking

Edit modules/common.nix to configure:

  • networking.localIp: LoadBalancer IP address
  • networking.domain.root: Base domain
  • networking.subDomain: Subdomain for the environment (set automatically by mkEnv in flake.nix)

Blue-Green Rollouts

The cluster uses Argo Rollouts for blue-green deployments. The Argo Rollouts controller and its CRDs run in the infra environment and reconcile rollout resources in application environments such as dev and prod.

For a blue-green application, traffic is handled through two Services:

  • The active Service receives production traffic through the existing Traefik route.
  • The preview Service selects the new ReplicaSet during an update.

The new pods must pass their Kubernetes readiness probe before they are considered healthy. Automatic promotion is enabled, so a healthy preview is promoted to active without manual intervention. If the readiness probe fails, the previous version remains active and the update does not promote.

Deployment Flow

  1. Update the application image in its environment configuration.

  2. Build and validate the environment:

    just build dev
    just test
    just test-IT
    
  3. Generate the manifests and commit both the configuration and generated manifests:

    echo y | just switch dev
    git add env/ manifests/
    git commit -m "Update application image"
    git push
    
  4. ArgoCD synchronizes the change.

  5. Argo Rollouts creates the preview ReplicaSet, waits for Ready pods, and promotes it automatically when healthy.

Observing a Rollout

kubectl get rollouts -n dev
kubectl describe rollout toolbox -n dev
kubectl argo rollouts get rollout toolbox -n dev
kubectl get pods -n dev -l app.kubernetes.io/name=toolbox
kubectl get services toolbox-svc toolbox-preview-svc -n dev

The kubectl argo rollouts commands require the Argo Rollouts kubectl plugin.

Manual Recovery

Automatic promotion is the normal path, but an operator can intervene when needed:

kubectl argo rollouts promote toolbox -n dev
kubectl argo rollouts abort toolbox -n dev
kubectl argo rollouts undo toolbox -n dev

An abort keeps the previous version active. An undo rolls the rollout back to its previous revision. Git remains the source of truth, so persistent rollback should also be represented by reverting or changing the image in the environment configuration and regenerating the manifests.

See the Argo Rollouts blue-green documentation for strategy details.

Quick Start

# Clone the repository (skip if you already have a clone)
git clone ssh://forgejo@nidaros.secure.martials.no/Personal/cluster.git
cd cluster

# Enter development shell (provides nixidy, kubectl, sops)
nix develop

# View available commands
just

# Build manifests (preview, doesn't modify files)
just build dev

# Compare the build against the committed manifests (writes nothing)
just diff dev

# Generate manifests into manifests/dev/ (prompts for confirmation)
just switch dev

# Review changes and commit
git diff manifests/
git add env/ manifests/
git commit -m "Update configuration"
git push

Note: ArgoCD automatically deploys changes pushed to the repository. just switch {env} runs git add . itself before regenerating, so the generated files are staged for you. The canonical GitOps URL is https://code.martials.no/Personal/cluster.git.

Common Operations

Run just or just --list to see all available commands. Most common:

just build dev          # Preview generated manifests
just switch dev         # Generate and write manifests
just diff dev           # Compare current vs generated
just argocd-status      # Check ArgoCD application status
just pods               # View all pods
just sops-edit          # Edit encrypted secrets

See the justfile for the complete list of operations.

Development Workflow

Making Changes

  1. Edit configuration in env/dev/*.nix, env/infra/*.nix, or env/prod/*.nix
  2. Build and validate: just build {env}
  3. Review the generated diff without writing: just diff {env}
  4. Generate manifests: echo y | just switch {env}
  5. Review: git diff manifests/
  6. Commit and push
  7. ArgoCD auto-syncs within minutes

Adding a New Application

  1. Create env/{dev,infra,prod}/myapp.nix — it is automatically discovered by import-tree, no registration needed:
{ lib, config, ... }:

{
  applications = lib.my.mkApplication {
    name = "myapp";
    image = "${config.networking.registry}/myapp:v1.0.0";
    env = config.nixidy.env;
    internalPort = 8080;
  };
}
  1. Build, switch, and commit: just build dev → echo y | just switch dev → commit env/ + manifests/

See toolbox.nix for a complete example with ingress and metrics.

Adding a New Environment

The cluster currently runs three environments: dev, infra, and prod, defined in the envs attrset in flake.nix. Adding another one follows the same pattern; substitute your-env below.

1. Create the Environment Directory

mkdir -p env/your-env

2. Register the Environment in flake.nix

Add the name to the envs list:

[
  "dev"       # → subdomain "dev"      → dev.martials.no
  "infra"     # → subdomain "infra"    → infra.martials.no
  "prod"      # → subdomain null       → martials.no
  "your-env"  # → subdomain "your-env" → your-env.martials.no
]

mkEnv sets networking.subDomain to the environment name (or null for prod), derives domain.fqdn from it, sets nixidy.appOfApps.name to apps-{env}, and points nixidy.target.rootPath at ./manifests/{env}.

3. Add Your Applications

Create one file per application in env/your-env/; each file is discovered automatically by import-tree, so there is no registration step:

{ config, lib, ... }:

{
  applications = lib.my.mkApplication {
    name = "toolbox";
    inherit (config.nixidy) env;
    image = "code.martials.no/personal/toolbox:1.0.0";
  };
}

Ports must stay unique across all environments; the build asserts this globally in modules/common.nix.

4. (Optional) Adjust Global Networking

modules/common.nix holds the defaults shared by every environment: networking.domain.root, networking.localIp, networking.clusterRepo, and networking.registry. Change them there only when the whole cluster should change, not for a single environment.

5. Bootstrap the Environment (one-time)

just bootstrap {env} renders the root Application apps-{env} and pipes it straight into kubectl apply -f -:

# Validate the environment first
just build your-env

# Inspect the object that would be applied (no cluster changes)
nix run .#nixidy -- bootstrap .#your-env

# Apply it
just bootstrap your-env

# Confirm the root Application exists and starts syncing
kubectl get applications -n argocd

The rendered object is a single Application named apps-{env} in the argocd namespace:

  • source ./manifests/{env}/apps-{env} at revision main
  • syncPolicy.automated with prune and selfHeal enabled

It is the app-of-apps entrypoint and is not stored in Git, which is why it must be applied once per environment. Everything Argo CD manages afterwards — including that environment's child Applications — comes from manifests/{env}/apps-{env}.

Prerequisite: Argo CD must already be running in the cluster. The initial installation is out-of-band and is not created by just bootstrap; after that the argocd-dev Application manages Argo CD from manifests/dev/argocd.

6. Generate and Commit Manifests

# Write manifests into manifests/your-env/
echo y | just switch your-env

# Review what changed
just diff your-env
git diff manifests/

# Commit the Nix configuration and the generated manifests together
git add env/ manifests/ flake.nix
git commit -m "Add your-env environment"
git push

7. Verify ArgoCD Sync

just argocd-status                       # status of every Application
just argocd-describe apps-your-env       # the new root Application
just watch-apps                          # live status loop

Key Points:

  • Only dev, infra, and prod exist today; your-env above is a placeholder name.
  • just bootstrap {env} is a one-time, per-environment step that applies apps-{env}; that object is not tracked in Git.
  • After bootstrap, every deployment is just switch {env} → push → Argo CD auto-sync.
  • Each environment gets manifests/{env}/ containing an apps-{env}/ directory with its Applications and one directory per generated application.
  • Templates (modules/templates.nix) and generators (modules/generators.nix) are shared, so a new environment only needs application definitions.

Using Helm Charts

helm.releases.myapp = {
  chart = lib.helm.downloadHelmChart {
    repo = "https://charts.example.com";
    chart = "app";
    version = "1.0.0";
    chartHash = "";  # Leave empty; build error provides correct hash
  };

  values = {
    replicaCount = 2;
    # ... other values
  };
};

Managing Secrets with SOPS

Secrets are stored in git as SOPS-encrypted SopsSecret resources (env/**/*.sops.yaml). The sops-secrets-operator (cluster-wide, running in dev) decrypts them in-cluster and creates the actual Kubernetes Secret in the namespace declared by the SopsSecret (no targetNamespace support).

just sops-status             # Operator pods + SopsSecret status
just sops-edit [path]        # Edit an encrypted secrets file (default: env/dev/argocd-secrets.sops.yaml)
just sops-encrypt [path]     # Encrypt a file in place
just sops-decrypt [path]     # Decrypt to stdout (does not modify)
just sops-bootstrap-age-key <keyfile>  # Bootstrap/replace the operator age-key Secret
just hash-password <pwd>     # Generate bcrypt for ArgoCD admin

Adding a new secret:

  1. Copy an existing SopsSecret (e.g. env/dev/argocd-secrets.sops.yaml) as a template into env/<env>/<name>.sops.yaml; update metadata.name/namespace and secretTemplates as needed.
  2. Insert values: just sops-edit env/<env>/<name>.sops.yaml (saving encrypts the file)
  3. Wire into the application in the env module: add extraRawYamls = [ ./<name>.sops.yaml ]; to applications.<app> (see env/dev/argocd.nix).
  4. just fmt → just build <env> → just switch <env> → commit env/ + manifests/.
  5. Verify: just sops-status shows SUCCESS and the Secret exists in the target namespace.

Notes:

  • Files must end in .sops.yaml so the creation rule in .sops.yaml (encrypted_regex: ^(data|stringData)$) applies.
  • The namespace: field of the SopsSecret decides where the Secret is created — it must match the application's namespace.
  • Never hand-edit or re-format an encrypted file (breaks its MAC); always use just sops-edit / just sops-encrypt.
  • The decryption age key is mounted into the operator from the sops-age-key-file Secret in dev. The Secret must contain a single data key literally named key — the mount exposes it as the file /etc/sops-age-key-file/key that SOPS_AGE_KEY_FILE points at. If the data key is renamed, the operator silently fails decryption. Bootstrap/replace it with just sops-bootstrap-age-key <keyfile> (which always sets the data key name to key); verify with just sops-status. Never commit a private key; recipients are managed in .sops.yaml.

Resource Labels

Every generated Kubernetes resource receives the label app.kubernetes.io/managed-by via a nixidy object transform (modules/common.nix:47, lib/default.nix:37):

Source managed-by value Examples
Direct Nix resources (templates, manual resources) nixidy Deployments, Services, IngressRoutes, namespaces, ArgoCD Applications, CRDs
Helm chart resources Helm (preserved from chart) ArgoCD, Traefik, Prometheus stack, SOPS operator
CRDs from Helm charts nixidy Charts typically don't label CRDs, so nixidy fills the gap

The transform does not overwrite an existing managed-by label, so Helm-charted resources accurately report Helm as their manager. Resources defined directly in Nix get nixidy.

Documentation

  • AGENTS.md: Guidelines for AI agents and contributors
  • justfile: All available commands with descriptions

Architecture

  • Type Safety: Nixidy validates resources against Kubernetes schemas
  • Reproducibility: flake.lock pins all dependencies
  • GitOps: Single source of truth in Git, ArgoCD handles deployment
  • Declarative: Entire cluster state defined in Nix configuration
  • Secrets: SOPS encrypts secrets at rest in Git, decrypted in-cluster

Resources

License

Personal cluster configuration. Use as reference for your own setup.