Kubernetes Cluster - Nixidy + ArgoCD

GitOps-managed Kubernetes cluster using Nixidy for type-safe manifest generation and ArgoCD for continuous deployment.

Overview

This repository implements the "Rendered Manifests Pattern":

• Nix configuration (env/{dev,infra,prod}/*.nix) defines all Kubernetes resources in a type-safe way
• Nixidy generates plain YAML manifests and commits them to manifests/{dev,infra,prod}/
• ArgoCD watches the manifest directories and automatically syncs changes to the cluster
• App-of-apps pattern: ArgoCD manages itself and all other applications

Key Benefits: Type safety, reproducible builds, Git-based audit trail, declarative infrastructure.

Repository Structure

cluster/
├── flake.nix                   # Nix flake with dependencies and env configuration
├── flake.lock                  # Locked dependency versions
├── justfile                    # Common operations (run `just` to see all)
├── treefmt.nix                 # Formatter configuration
├── modules/
│   ├── common.nix              # Global networking options + nixidy defaults + port assertion
│   ├── templates.nix           # Reusable application templates (webApplication, route)
│   └── generators.nix          # CRD resource option generators (Traefik, Prometheus Operator)
├── lib/
│   └── default.nix             # lib.my.mkApplication, lib.my.addManagedByLabel, lib.my.checkServicePortUniqueness
├── tests/
│   ├── default.nix             # Unit tests (nix-unit, all systems)
│   └── integration.nix         # Integration tests (x86_64-linux only)
├── env/
│   ├── dev/                    # Dev application definitions (auto-discovered by import-tree)
│   │   ├── *.nix               # Application configs (argocd, traefik, etc.)
│   │   └── *.sops.yaml         # Encrypted secrets
│   ├── infra/                  # Infrastructure application definitions
│   │   ├── *.nix               # Infrastructure apps (monitoring, logging, etc.)
│   │   └── *.sops.yaml         # Encrypted secrets
│   └── prod/                   # Prod application definitions
│       └── *.nix
└── manifests/
    ├── dev/                    # Generated YAML manifests for dev (committed, do not edit)
    │   ├── apps/               # ArgoCD Application definitions
    │   └── */                  # Application manifests
    ├── infra/                  # Generated YAML manifests for infra (committed, do not edit)
    │   ├── apps/               # ArgoCD Application definitions
    │   └── */                  # Application manifests
    └── prod/                   # Generated YAML manifests for prod (committed, do not edit)
        ├── apps/               # ArgoCD Application definitions
        └── */                  # Application manifests

Prerequisites

• Nix with flakes enabled
• Kubernetes cluster (k3s recommended)
• SOPS with age keys for secrets management (optional)

Ingress & Routing

The cluster uses Traefik as the ingress controller, deployed via Helm chart.

Configuration

• LoadBalancer IP: 192.168.10.228 (configured in modules/common.nix)
• Domain (dev): *.dev.martials.no / Domain (infra): *.infra.martials.no / Domain (prod): *.martials.no (configured in modules/common.nix)
• Entry Points:
  • web: HTTP (port 80)
  • websecure: HTTPS (port 443)

Routing Methods

Traefik IngressRoute (type-safe Nix resources preferred; YAML also supported):

resources.ingressRoutes.myapp-route.spec = {
  entryPoints = [ "web" ];
  routes = [
    {
      match = "Host(`myapp.dev.martials.no`)";
      kind = "Rule";
      services = [ { name = "myapp-svc"; port = 80; } ];
    }
  ];
};

See toolbox.nix for a complete example.

Customizing Networking

Edit modules/common.nix to configure:

• networking.localIp: LoadBalancer IP address
• networking.domain.root: Base domain
• networking.subDomain: Subdomain for the environment (set automatically by mkEnv in flake.nix)

Blue-Green Rollouts

The cluster uses Argo Rollouts for blue-green deployments. The Argo Rollouts controller and its CRDs run in the infra environment and reconcile rollout resources in application environments such as dev and prod.

For a blue-green application, traffic is handled through two Services:

• The active Service receives production traffic through the existing Traefik route.
• The preview Service selects the new ReplicaSet during an update.

The new pods must pass their Kubernetes readiness probe before they are considered healthy. Automatic promotion is enabled, so a healthy preview is promoted to active without manual intervention. If the readiness probe fails, the previous version remains active and the update does not promote.

Deployment Flow

1. Update the application image in its environment configuration.

2. Build and validate the environment:

   just build dev
   just test
   just test-IT
   

3. Generate the manifests and commit both the configuration and generated manifests:

   echo y | just switch dev
   git add env/ manifests/
   git commit -m "Update application image"
   git push
   

4. ArgoCD synchronizes the change.

5. Argo Rollouts creates the preview ReplicaSet, waits for Ready pods, and promotes it automatically when healthy.

Observing a Rollout

kubectl get rollouts -n dev
kubectl describe rollout toolbox -n dev
kubectl argo rollouts get rollout toolbox -n dev
kubectl get pods -n dev -l app.kubernetes.io/name=toolbox
kubectl get services toolbox-svc toolbox-preview-svc -n dev

The kubectl argo rollouts commands require the Argo Rollouts kubectl plugin.

Manual Recovery

Automatic promotion is the normal path, but an operator can intervene when needed:

kubectl argo rollouts promote toolbox -n dev
kubectl argo rollouts abort toolbox -n dev
kubectl argo rollouts undo toolbox -n dev

An abort keeps the previous version active. An undo rolls the rollout back to its previous revision. Git remains the source of truth, so persistent rollback should also be represented by reverting or changing the image in the environment configuration and regenerating the manifests.

See the Argo Rollouts blue-green documentation for strategy details.

Quick Start

# Enter development shell (provides nixidy, kubectl, sops)
nix develop

# View available commands
just

# Build manifests (preview, doesn't modify files)
just build dev

# Generate manifests to manifests/dev/
just switch dev

# Review changes and commit
git diff manifests/
git add env/ manifests/
git commit -m "Update configuration"
git push

Note: ArgoCD automatically deploys changes pushed to the repository.

Common Operations

Run just or just --list to see all available commands. Most common:

just build dev          # Preview generated manifests
just switch dev         # Generate and write manifests
just diff dev           # Compare current vs generated
just argocd-status      # Check ArgoCD application status
just pods               # View all pods
just sops-edit          # Edit encrypted secrets

See the justfile for the complete list of operations.

Development Workflow

Making Changes

1. Edit configuration in env/dev/*.nix, env/infra/*.nix, or env/prod/*.nix
2. Build and validate: just build {env}
3. Generate manifests: echo y | just switch {env}
4. Review: git diff manifests/
5. Commit and push
6. ArgoCD auto-syncs within minutes

Adding a New Application

1. Create env/{dev,infra,prod}/myapp.nix — it is automatically discovered by import-tree, no registration needed:

{ lib, config, ... }:

{
  applications = lib.my.mkApplication {
    name = "myapp";
    image = "${config.networking.registry}/myapp:v1.0.0";
    env = config.nixidy.env;
    internalPort = 8080;
  };
}

2. Build, switch, and commit: just build dev → echo y | just switch dev → commit env/ + manifests/

See toolbox.nix for a complete example with ingress and metrics.

Adding a New Environment

To add a new environment (e.g., staging, testing), follow these steps:

1. Create Environment Directory

mkdir -p env/staging

2. Register Environment in flake.nix

Update the mkEnv function or the envs list in flake.nix to include your new environment. The mkEnv function automatically:

• Sets networking.subDomain to the environment name (or null for prod)
• Computes domain.fqdn as subDomain.domain.root (e.g., staging.martials.no)

Add your environment name to the environments list in the envs section:

[
  "dev"      # → subdomain "dev"   → dev.martials.no
  "infra"    # → subdomain "infra" → infra.martials.no
  "prod"     # → subdomain null    → martials.no
  "staging"  # → subdomain "staging" → staging.martials.no
]

3. Add Your Applications

Create application configuration files in the new environment directory. For example, create env/staging/toolbox.nix:

{ config, ... }:

{
  applications.toolbox = {
    namespace = "staging";
    templates.webApplication.toolbox = {
      image = "${config.networking.registry}/toolbox:1.0.0";
      internalPort = 8080;
    };
  };
}

Add additional .nix files for other applications as needed. Each file is automatically discovered by import-tree.

4. (Optional) Update modules/common.nix

If your environment needs different domain root or other networking settings, update modules/common.nix:

# In modules/common.nix, update domain.root or other networking config
networking.domain.root = "example.com";  # Override for all envs

The FQDN is automatically computed from networking.subDomain + domain.root. For the staging environment, the resulting FQDN would be staging.example.com.

5. Bootstrap the Environment

The bootstrap recipe deploys the initial ArgoCD manifests to your cluster:

# Validate configuration first
just build staging

# Bootstrap the environment (deploys ArgoCD and initial apps)
just bootstrap staging

# Verify deployment
kubectl get applications -n argocd

6. Generate and Commit Manifests

# Generate manifests to manifests/staging/
just switch staging

# Review changes
git diff manifests/

# Commit both .nix and manifests/
git add env/ manifests/
git commit -m "Add staging environment"
git push

7. Verify ArgoCD Sync

Once bootstrapped, ArgoCD automatically watches the manifests/staging/ directory and syncs applications:

# Watch application status
just watch-apps

# Or check specific app
just argocd-describe apps-staging

Key Points:

• The bootstrap recipe is the one-time setup that deploys ArgoCD and the app-of-apps
• After bootstrap, all subsequent deployments use just switch {env} → push → ArgoCD auto-syncs
• Each environment gets its own manifests/{env}/ directory
• Environment-specific configuration is stored in env/{env}/*.nix

Using Helm Charts

helm.releases.myapp = {
  chart = lib.helm.downloadHelmChart {
    repo = "https://charts.example.com";
    chart = "app";
    version = "1.0.0";
    chartHash = "";  # Leave empty; build error provides correct hash
  };

  values = {
    replicaCount = 2;
    # ... other values
  };
};

Managing Secrets with SOPS

Secrets are stored in git as SOPS-encrypted SopsSecret resources (env/**/*.sops.yaml). The sops-secrets-operator (cluster-wide, running in dev) decrypts them in-cluster and creates the actual Kubernetes Secret in the namespace declared by the SopsSecret (no targetNamespace support).

just sops-status             # Operator pods + SopsSecret status
just sops-edit [path]        # Edit an encrypted secrets file (default: env/dev/argocd-secrets.sops.yaml)
just sops-encrypt [path]     # Encrypt a file in place
just sops-decrypt [path]     # Decrypt to stdout (does not modify)
just sops-bootstrap-age-key <keyfile>  # Bootstrap/replace the operator age-key Secret
just hash-password <pwd>     # Generate bcrypt for ArgoCD admin

Adding a new secret:

1. Copy an existing SopsSecret (e.g. env/dev/argocd-secrets.sops.yaml) as a template into env/<env>/<name>.sops.yaml; update metadata.name/namespace and secretTemplates as needed.
2. Insert values: just sops-edit env/<env>/<name>.sops.yaml (saving encrypts the file)
3. Wire into the application in the env module: add extraRawYamls = [ ./<name>.sops.yaml ]; to applications.<app> (see env/dev/argocd.nix).
4. just fmt → just build <env> → just switch <env> → commit env/ + manifests/.
5. Verify: just sops-status shows SUCCESS and the Secret exists in the target namespace.

Notes:

• Files must end in .sops.yaml so the creation rule in .sops.yaml (encrypted_regex: ^(data|stringData)$) applies.
• The namespace: field of the SopsSecret decides where the Secret is created — it must match the application's namespace.
• Never hand-edit or re-format an encrypted file (breaks its MAC); always use just sops-edit / just sops-encrypt.
• The decryption age key is mounted into the operator from the sops-age-key-file Secret in dev. The Secret must contain a single data key literally named key — the mount exposes it as the file /etc/sops-age-key-file/key that SOPS_AGE_KEY_FILE points at. If the data key is renamed, the operator silently fails decryption. Bootstrap/replace it with just sops-bootstrap-age-key <keyfile> (which always sets the data key name to key); verify with just sops-status. Never commit a private key; recipients are managed in .sops.yaml.

Resource Labels

Every generated Kubernetes resource receives the label app.kubernetes.io/managed-by via a nixidy object transform (modules/common.nix:47, lib/default.nix:37):

Source	managed-by value	Examples
Direct Nix resources (templates, manual resources)	nixidy	Deployments, Services, IngressRoutes, namespaces, ArgoCD Applications, CRDs
Helm chart resources	Helm (preserved from chart)	ArgoCD, Traefik, Prometheus stack, SOPS operator
CRDs from Helm charts	nixidy	Charts typically don't label CRDs, so nixidy fills the gap

The transform does not overwrite an existing managed-by label, so Helm-charted resources accurately report Helm as their manager. Resources defined directly in Nix get nixidy.

Documentation

• AGENTS.md: Guidelines for AI agents and contributors
• justfile: All available commands with descriptions

Architecture

• Type Safety: Nixidy validates resources against Kubernetes schemas
• Reproducibility: flake.lock pins all dependencies
• GitOps: Single source of truth in Git, ArgoCD handles deployment
• Declarative: Entire cluster state defined in Nix configuration
• Secrets: SOPS encrypts secrets at rest in Git, decrypted in-cluster

Resources

• Nixidy Documentation - Official docs and options reference
• ArgoCD Documentation - GitOps deployment
• k3s Documentation - Lightweight Kubernetes
• SOPS - Secrets management

License

Personal cluster configuration. Use as reference for your own setup.