- Nix 85.8%
- Just 14.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
All checks were successful
ci/woodpecker/cron/flake-check Pipeline was successful
feat(catalog): define personal-services System Co-authored-by: OpenCode <opencode@anoma.ly> Reviewed-on: #87 |
||
| .woodpecker | ||
| env | ||
| lib | ||
| manifests | ||
| modules | ||
| tests | ||
| .gitignore | ||
| .sops.yaml | ||
| AGENTS.md | ||
| catalog-info.yaml | ||
| flake.lock | ||
| flake.nix | ||
| justfile | ||
| README.md | ||
| renovate.json | ||
| treefmt.nix | ||
Kubernetes Cluster - Nixidy + ArgoCD
GitOps-managed Kubernetes cluster using Nixidy for type-safe manifest generation and ArgoCD for continuous deployment.
Overview
This repository implements the "Rendered Manifests Pattern":
- Nix configuration (
env/{dev,infra,prod}/*.nix) defines all Kubernetes resources in a type-safe way - Nixidy generates plain YAML manifests and commits them to
manifests/{dev,infra,prod}/ - ArgoCD watches the manifest directories and automatically syncs changes to the cluster
- App-of-apps pattern: ArgoCD manages itself and all other applications
Key Benefits: Type safety, reproducible builds, Git-based audit trail, declarative infrastructure.
Repository Structure
cluster/
├── flake.nix # Nix flake with dependencies and env configuration
├── flake.lock # Locked dependency versions
├── justfile # Common operations (run `just` to see all)
├── treefmt.nix # Formatter configuration
├── modules/
│ ├── common.nix # Global networking options + nixidy defaults + port assertion
│ ├── templates.nix # Reusable application templates (webApplication, route)
│ └── generators.nix # CRD resource option generators (Traefik, Prometheus Operator)
├── lib/
│ └── default.nix # lib.my.mkApplication, lib.my.addManagedByLabel, lib.my.checkServicePortUniqueness
├── tests/
│ ├── default.nix # Unit tests (nix-unit, all systems)
│ └── integration.nix # Integration tests (x86_64-linux only)
├── env/
│ ├── dev/ # Dev application definitions (auto-discovered by import-tree)
│ │ ├── *.nix # Application configs (argocd, traefik, etc.)
│ │ └── *.sops.yaml # Encrypted secrets
│ ├── infra/ # Infrastructure application definitions
│ │ ├── *.nix # Infrastructure apps (monitoring, logging, etc.)
│ │ └── *.sops.yaml # Encrypted secrets
│ └── prod/ # Prod application definitions
│ └── *.nix
└── manifests/
├── dev/ # Generated YAML manifests for dev (committed, do not edit)
│ ├── apps-dev/ # ArgoCD Applications for this environment (the app-of-apps children)
│ ├── argocd/ # Per-application manifests, one directory per app
│ └── */ # e.g. traefik/, toolbox/, martials.no/
├── infra/ # Generated YAML manifests for infra (committed, do not edit)
│ ├── apps-infra/ # ArgoCD Applications for this environment
│ └── */ # e.g. prometheus-stack/, argo-rollouts/
└── prod/ # Generated YAML manifests for prod (committed, do not edit)
├── apps-prod/ # ArgoCD Applications for this environment
└── */ # e.g. martials.no/, recurring-events/
Prerequisites
- Nix with flakes enabled
- Kubernetes cluster (k3s recommended)
- SOPS with age keys for secrets management (optional)
Ingress & Routing
The cluster uses Traefik as the ingress controller, deployed via Helm chart.
Configuration
- LoadBalancer IP:
192.168.10.228(configured inmodules/common.nix) - Domain (dev):
*.dev.martials.no/ Domain (infra):*.infra.martials.no/ Domain (prod):*.martials.no(configured inmodules/common.nix) - Entry Points:
web: HTTP (port 80)websecure: HTTPS (port 443)
Routing Methods
Traefik IngressRoute (type-safe Nix resources preferred; YAML also supported):
resources.ingressRoutes.myapp-route.spec = {
entryPoints = [ "web" ];
routes = [
{
match = "Host(`myapp.dev.martials.no`)";
kind = "Rule";
services = [ { name = "myapp-svc"; port = 80; } ];
}
];
};
See toolbox.nix for a complete example.
Customizing Networking
Edit modules/common.nix to configure:
networking.localIp: LoadBalancer IP addressnetworking.domain.root: Base domainnetworking.subDomain: Subdomain for the environment (set automatically bymkEnvinflake.nix)
Blue-Green Rollouts
The cluster uses Argo Rollouts for blue-green deployments. The Argo Rollouts controller and its CRDs run in the infra environment and reconcile rollout resources in application environments such as dev and prod.
For a blue-green application, traffic is handled through two Services:
- The active Service receives production traffic through the existing Traefik route.
- The preview Service selects the new ReplicaSet during an update.
The new pods must pass their Kubernetes readiness probe before they are considered healthy. Automatic promotion is enabled, so a healthy preview is promoted to active without manual intervention. If the readiness probe fails, the previous version remains active and the update does not promote.
Deployment Flow
-
Update the application image in its environment configuration.
-
Build and validate the environment:
just build dev just test just test-IT -
Generate the manifests and commit both the configuration and generated manifests:
echo y | just switch dev git add env/ manifests/ git commit -m "Update application image" git push -
ArgoCD synchronizes the change.
-
Argo Rollouts creates the preview ReplicaSet, waits for Ready pods, and promotes it automatically when healthy.
Observing a Rollout
kubectl get rollouts -n dev
kubectl describe rollout toolbox -n dev
kubectl argo rollouts get rollout toolbox -n dev
kubectl get pods -n dev -l app.kubernetes.io/name=toolbox
kubectl get services toolbox-svc toolbox-preview-svc -n dev
The kubectl argo rollouts commands require the Argo Rollouts kubectl plugin.
Manual Recovery
Automatic promotion is the normal path, but an operator can intervene when needed:
kubectl argo rollouts promote toolbox -n dev
kubectl argo rollouts abort toolbox -n dev
kubectl argo rollouts undo toolbox -n dev
An abort keeps the previous version active. An undo rolls the rollout back to its previous revision. Git remains the source of truth, so persistent rollback should also be represented by reverting or changing the image in the environment configuration and regenerating the manifests.
See the Argo Rollouts blue-green documentation for strategy details.
Quick Start
# Clone the repository (skip if you already have a clone)
git clone ssh://forgejo@nidaros.secure.martials.no/Personal/cluster.git
cd cluster
# Enter development shell (provides nixidy, kubectl, sops)
nix develop
# View available commands
just
# Build manifests (preview, doesn't modify files)
just build dev
# Compare the build against the committed manifests (writes nothing)
just diff dev
# Generate manifests into manifests/dev/ (prompts for confirmation)
just switch dev
# Review changes and commit
git diff manifests/
git add env/ manifests/
git commit -m "Update configuration"
git push
Note: ArgoCD automatically deploys changes pushed to the repository. just switch {env} runs git add . itself before regenerating, so the generated files are staged for you. The canonical GitOps URL is https://code.martials.no/Personal/cluster.git.
Common Operations
Run just or just --list to see all available commands. Most common:
just build dev # Preview generated manifests
just switch dev # Generate and write manifests
just diff dev # Compare current vs generated
just argocd-status # Check ArgoCD application status
just pods # View all pods
just sops-edit # Edit encrypted secrets
See the justfile for the complete list of operations.
Development Workflow
Making Changes
- Edit configuration in
env/dev/*.nix,env/infra/*.nix, orenv/prod/*.nix - Build and validate:
just build {env} - Review the generated diff without writing:
just diff {env} - Generate manifests:
echo y | just switch {env} - Review:
git diff manifests/ - Commit and push
- ArgoCD auto-syncs within minutes
Adding a New Application
- Create
env/{dev,infra,prod}/myapp.nix— it is automatically discovered by import-tree, no registration needed:
{ lib, config, ... }:
{
applications = lib.my.mkApplication {
name = "myapp";
image = "${config.networking.registry}/myapp:v1.0.0";
env = config.nixidy.env;
internalPort = 8080;
};
}
- Build, switch, and commit:
just build dev→echo y | just switch dev→ commitenv/+manifests/
See toolbox.nix for a complete example with ingress and metrics.
Adding a New Environment
The cluster currently runs three environments: dev, infra, and prod, defined in the envs
attrset in flake.nix. Adding another one follows the same pattern; substitute your-env below.
1. Create the Environment Directory
mkdir -p env/your-env
2. Register the Environment in flake.nix
Add the name to the envs list:
[
"dev" # → subdomain "dev" → dev.martials.no
"infra" # → subdomain "infra" → infra.martials.no
"prod" # → subdomain null → martials.no
"your-env" # → subdomain "your-env" → your-env.martials.no
]
mkEnv sets networking.subDomain to the environment name (or null for prod), derives
domain.fqdn from it, sets nixidy.appOfApps.name to apps-{env}, and points
nixidy.target.rootPath at ./manifests/{env}.
3. Add Your Applications
Create one file per application in env/your-env/; each file is discovered automatically by
import-tree, so there is no registration step:
{ config, lib, ... }:
{
applications = lib.my.mkApplication {
name = "toolbox";
inherit (config.nixidy) env;
image = "code.martials.no/personal/toolbox:1.0.0";
};
}
Ports must stay unique across all environments; the build asserts this globally in
modules/common.nix.
4. (Optional) Adjust Global Networking
modules/common.nix holds the defaults shared by every environment: networking.domain.root,
networking.localIp, networking.clusterRepo, and networking.registry. Change them there only
when the whole cluster should change, not for a single environment.
5. Bootstrap the Environment (one-time)
just bootstrap {env} renders the root Application apps-{env} and pipes it straight into
kubectl apply -f -:
# Validate the environment first
just build your-env
# Inspect the object that would be applied (no cluster changes)
nix run .#nixidy -- bootstrap .#your-env
# Apply it
just bootstrap your-env
# Confirm the root Application exists and starts syncing
kubectl get applications -n argocd
The rendered object is a single Application named apps-{env} in the argocd namespace:
- source
./manifests/{env}/apps-{env}at revisionmain syncPolicy.automatedwithpruneandselfHealenabled
It is the app-of-apps entrypoint and is not stored in Git, which is why it must be applied once
per environment. Everything Argo CD manages afterwards — including that environment's child
Applications — comes from manifests/{env}/apps-{env}.
Prerequisite: Argo CD must already be running in the cluster. The initial installation is
out-of-band and is not created by just bootstrap; after that the argocd-dev Application manages
Argo CD from manifests/dev/argocd.
6. Generate and Commit Manifests
# Write manifests into manifests/your-env/
echo y | just switch your-env
# Review what changed
just diff your-env
git diff manifests/
# Commit the Nix configuration and the generated manifests together
git add env/ manifests/ flake.nix
git commit -m "Add your-env environment"
git push
7. Verify ArgoCD Sync
just argocd-status # status of every Application
just argocd-describe apps-your-env # the new root Application
just watch-apps # live status loop
Key Points:
- Only
dev,infra, andprodexist today;your-envabove is a placeholder name. just bootstrap {env}is a one-time, per-environment step that appliesapps-{env}; that object is not tracked in Git.- After bootstrap, every deployment is
just switch {env}→ push → Argo CD auto-sync. - Each environment gets
manifests/{env}/containing anapps-{env}/directory with its Applications and one directory per generated application. - Templates (
modules/templates.nix) and generators (modules/generators.nix) are shared, so a new environment only needs application definitions.
Using Helm Charts
helm.releases.myapp = {
chart = lib.helm.downloadHelmChart {
repo = "https://charts.example.com";
chart = "app";
version = "1.0.0";
chartHash = ""; # Leave empty; build error provides correct hash
};
values = {
replicaCount = 2;
# ... other values
};
};
Managing Secrets with SOPS
Secrets are stored in git as SOPS-encrypted SopsSecret resources (env/**/*.sops.yaml). The
sops-secrets-operator (cluster-wide, running in dev) decrypts them in-cluster and creates the actual
Kubernetes Secret in the namespace declared by the SopsSecret (no targetNamespace support).
just sops-status # Operator pods + SopsSecret status
just sops-edit [path] # Edit an encrypted secrets file (default: env/dev/argocd-secrets.sops.yaml)
just sops-encrypt [path] # Encrypt a file in place
just sops-decrypt [path] # Decrypt to stdout (does not modify)
just sops-bootstrap-age-key <keyfile> # Bootstrap/replace the operator age-key Secret
just hash-password <pwd> # Generate bcrypt for ArgoCD admin
Adding a new secret:
- Copy an existing
SopsSecret(e.g.env/dev/argocd-secrets.sops.yaml) as a template intoenv/<env>/<name>.sops.yaml; updatemetadata.name/namespaceandsecretTemplatesas needed. - Insert values:
just sops-edit env/<env>/<name>.sops.yaml(saving encrypts the file) - Wire into the application in the env module: add
extraRawYamls = [ ./<name>.sops.yaml ];toapplications.<app>(seeenv/dev/argocd.nix). just fmt→just build <env>→just switch <env>→ commitenv/+manifests/.- Verify:
just sops-statusshowsSUCCESSand theSecretexists in the target namespace.
Notes:
- Files must end in
.sops.yamlso the creation rule in.sops.yaml(encrypted_regex: ^(data|stringData)$) applies. - The
namespace:field of theSopsSecretdecides where theSecretis created — it must match the application's namespace. - Never hand-edit or re-format an encrypted file (breaks its MAC); always use
just sops-edit/just sops-encrypt. - The decryption age key is mounted into the operator from the
sops-age-key-fileSecret indev. The Secret must contain a single data key literally namedkey— the mount exposes it as the file/etc/sops-age-key-file/keythatSOPS_AGE_KEY_FILEpoints at. If the data key is renamed, the operator silently fails decryption. Bootstrap/replace it withjust sops-bootstrap-age-key <keyfile>(which always sets the data key name tokey); verify withjust sops-status. Never commit a private key; recipients are managed in.sops.yaml.
Resource Labels
Every generated Kubernetes resource receives the label app.kubernetes.io/managed-by via a nixidy object transform (modules/common.nix:47, lib/default.nix:37):
| Source | managed-by value |
Examples |
|---|---|---|
| Direct Nix resources (templates, manual resources) | nixidy |
Deployments, Services, IngressRoutes, namespaces, ArgoCD Applications, CRDs |
| Helm chart resources | Helm (preserved from chart) |
ArgoCD, Traefik, Prometheus stack, SOPS operator |
| CRDs from Helm charts | nixidy |
Charts typically don't label CRDs, so nixidy fills the gap |
The transform does not overwrite an existing managed-by label, so Helm-charted resources accurately report Helm as their manager. Resources defined directly in Nix get nixidy.
Documentation
- AGENTS.md: Guidelines for AI agents and contributors
- justfile: All available commands with descriptions
Architecture
- Type Safety: Nixidy validates resources against Kubernetes schemas
- Reproducibility:
flake.lockpins all dependencies - GitOps: Single source of truth in Git, ArgoCD handles deployment
- Declarative: Entire cluster state defined in Nix configuration
- Secrets: SOPS encrypts secrets at rest in Git, decrypted in-cluster
Resources
- Nixidy Documentation - Official docs and options reference
- ArgoCD Documentation - GitOps deployment
- k3s Documentation - Lightweight Kubernetes
- SOPS - Secrets management
License
Personal cluster configuration. Use as reference for your own setup.