T14: Vault Kubernetes auth backend + per-app VSO roles #29

New issue

Open

opened 2026-08-12 15:02:23 +00:00 by ginjiruu · 0 comments

ginjiruu commented

2026-08-12 15:02:23 +00:00

Owner

T14: Vault Kubernetes auth backend + per-app VSO roles

token_reviewer_jwt decision (revised 2026-08-12, per #38 Golden Path; gateway updated 2026-09): Vault authenticates to every cluster's apiserver through the k8s-hub (Cilium Gateway API, authz-bridge) with the vault Kanidm SA API token as token_reviewer_jwt — exchanged per request (audience = the cluster's apiserver client, e.g. minicluster), the same token for every cluster, including the cluster Vault runs on. Vault is decoupled from the cluster it runs on; if Vault moves, its connection config is unchanged. The pod SA tokens presented at login still travel in the TokenReview body and are validated by the apiserver itself.

Implements the next step of the ADR-0004 read path: Vault's Kubernetes auth method mounted per cluster, with per-app VaultAuth roles so the Vault Secrets Operator can authenticate as each app's own ServiceAccount.

Deliverables

vault/kubernetes-auth.tf — a vault_auth_backend (type=kubernetes, path kubernetes/<cluster>) + vault_kubernetes_auth_backend_config per cluster in var.kubernetes_auth. The single token_reviewer_jwt is the vault Kanidm SA API token (from #35; sensitive, via var.kubernetes_token_reviewer_jwts/varsFrom — never code); kubernetes_host points at the k8s-hub apiserver route (T13 #28, https://<cluster>.k8s.animeteamspeak.moe). The route terminates with a Let's Encrypt cert (public CA, system trust), so kubernetes_ca_cert is optional. Defaults empty, so the vault-config auto-apply on mini is a no-op until bootstrapped.
Per-app Kubernetes auth roles in vault/modules/app_secrets for every configured cluster backend:
- <app>-vso → token policy <app>_reader, bound to SA <app>-vso in the app namespace (VSO read path)
- <app>-esopush → token policy <app>_generated-secret-manager, bound to SA <app>-esopush in flux-system (ESO PushSecret, generated tier)
- <app>-break-glass → token policy <app>_root, bound to SA <app>-break-glass in flux-system (emergency admin)
<app>_reader policy — read/list over the app's own generated/ + manual/ paths (ADR-0004 A-reader); the app's runtime read path through VSO.
Per-app ServiceAccounts in k8s/modules/tenant: <app>-vso in the app namespace; <app>-esopush and <app>-break-glass in flux-system.
Bug fix: vault/modules/app_secrets identity groups referenced policy names with a trailing -policy that don't exist (policies are ${app}_${capability}). Dropped the suffix.

Bootstrap (prerequisites before enabling)

Vault calls each cluster's TokenReview API as a Kanidm principal through the gateway (Golden Path, #38):

#35: create the vault Kanidm SA and grant its groups the groups_name scope map on the per-cluster OAuth2 clients (minicluster/portablecluster).
#11 (flux-clusters): apiserver OIDC flags live, so the gateway-forwarded token (audience = the cluster's apiserver client, groups_name claim) authenticates.
Per cluster: bind the vault SA's capability group to system:auth-delegator (tokenreviews.create) — one ClusterRoleBinding keyed by the group, not a per-cluster SA.
Config: kubernetes_host = https://<cluster>.k8s.animeteamspeak.moe (k8s-hub route), kubernetes_ca_cert = optional (Let's Encrypt, public trust), token_reviewer_jwt = the vault SA API token — the same token for every cluster, exchanged per request by the authz-bridge.

There is no per-cluster vault-auth SA and no 'omit for the in-cluster case' — the path is uniform even for the cluster Vault runs on. Vault stays decoupled from its host cluster: the connection config is per target cluster, not per deployment location.

Bootstrap values for mini

kubernetes_host: https://mini.k8s.animeteamspeak.moe (k8s-hub apiserver route, T13 #28)
kubernetes_ca_cert: not required — the k8s-hub route terminates with a Let's Encrypt cert (public trust); mini's apiserver CA is not needed either
token_reviewer_jwt: the vault Kanidm SA API token (#35) — same value for mini, portable, remote

Out of scope

VSO operator install per cluster (shipped separately in lab/clusters PR #1525)
Per-app VaultAuth/VaultStaticSecret CRs in the app repos
Wiring tofu-controller sync for the k8s/ root (T10, #25)

## T14: Vault Kubernetes auth backend + per-app VSO roles > `token_reviewer_jwt` decision (revised 2026-08-12, per **#38 Golden Path**; gateway updated 2026-09): Vault authenticates to every cluster's apiserver through the **k8s-hub** (Cilium Gateway API, authz-bridge) with the **`vault` Kanidm SA API token** as `token_reviewer_jwt` — exchanged per request (audience = the cluster's apiserver client, e.g. `minicluster`), the **same token for every cluster, including the cluster Vault runs on**. Vault is decoupled from the cluster it runs on; if Vault moves, its connection config is unchanged. The pod SA tokens presented at login still travel in the TokenReview body and are validated by the apiserver itself. Implements the next step of the ADR-0004 read path: Vault's Kubernetes auth method mounted per cluster, with per-app `VaultAuth` roles so the Vault Secrets Operator can authenticate as each app's own ServiceAccount. ## Deliverables 1. **`vault/kubernetes-auth.tf`** — a `vault_auth_backend` (`type=kubernetes`, path `kubernetes/<cluster>`) + `vault_kubernetes_auth_backend_config` per cluster in `var.kubernetes_auth`. The single `token_reviewer_jwt` is the `vault` Kanidm SA API token (from #35; sensitive, via `var.kubernetes_token_reviewer_jwts`/varsFrom — never code); `kubernetes_host` points at the k8s-hub apiserver route (T13 #28, `https://<cluster>.k8s.animeteamspeak.moe`). The route terminates with a Let's Encrypt cert (public CA, system trust), so `kubernetes_ca_cert` is optional. Defaults empty, so the vault-config auto-apply on mini is a no-op until bootstrapped. 2. **Per-app Kubernetes auth roles** in `vault/modules/app_secrets` for every configured cluster backend: - `<app>-vso` → token policy `<app>_reader`, bound to SA `<app>-vso` in the app namespace (VSO read path) - `<app>-esopush` → token policy `<app>_generated-secret-manager`, bound to SA `<app>-esopush` in `flux-system` (ESO PushSecret, generated tier) - `<app>-break-glass` → token policy `<app>_root`, bound to SA `<app>-break-glass` in `flux-system` (emergency admin) 3. **`<app>_reader` policy** — read/list over the app's own `generated/` + `manual/` paths (ADR-0004 `A-reader`); the app's runtime read path through VSO. 4. **Per-app ServiceAccounts** in `k8s/modules/tenant`: `<app>-vso` in the app namespace; `<app>-esopush` and `<app>-break-glass` in `flux-system`. 5. **Bug fix**: `vault/modules/app_secrets` identity groups referenced policy names with a trailing `-policy` that don't exist (policies are `${app}_${capability}`). Dropped the suffix. ## Bootstrap (prerequisites before enabling) Vault calls each cluster's **TokenReview** API as a Kanidm principal through the gateway (Golden Path, #38): - **#35**: create the `vault` Kanidm SA and grant its groups the `groups_name` scope map on the per-cluster OAuth2 clients (`minicluster`/`portablecluster`). - **#11 (flux-clusters)**: apiserver OIDC flags live, so the gateway-forwarded token (audience = the cluster's apiserver client, `groups_name` claim) authenticates. - **Per cluster**: bind the `vault` SA's capability group to `system:auth-delegator` (`tokenreviews.create`) — one ClusterRoleBinding keyed by the group, not a per-cluster SA. - **Config**: `kubernetes_host` = `https://<cluster>.k8s.animeteamspeak.moe` (k8s-hub route), `kubernetes_ca_cert` = optional (Let's Encrypt, public trust), `token_reviewer_jwt` = the `vault` SA API token — **the same token for every cluster**, exchanged per request by the authz-bridge. There is **no per-cluster `vault-auth` SA and no 'omit for the in-cluster case'** — the path is uniform even for the cluster Vault runs on. Vault stays decoupled from its host cluster: the connection config is per *target* cluster, not per *deployment location*. ## Bootstrap values for mini - `kubernetes_host`: `https://mini.k8s.animeteamspeak.moe` (k8s-hub apiserver route, T13 #28) - `kubernetes_ca_cert`: not required — the k8s-hub route terminates with a **Let's Encrypt** cert (public trust); mini's apiserver CA is not needed either - `token_reviewer_jwt`: the `vault` Kanidm SA API token (#35) — same value for mini, portable, remote ## Out of scope - VSO operator install per cluster (shipped separately in `lab/clusters` PR #1525) - Per-app `VaultAuth`/`VaultStaticSecret` CRs in the app repos - Wiring `tofu-controller` sync for the `k8s/` root (T10, #25)

ginjiruu referenced this issue from a pull request that will close it,

2026-08-12 15:03:03 +00:00

T14: Vault Kubernetes auth backend + per-app VSO roles #30

ginjiruu referenced this issue

2026-08-12 15:51:01 +00:00

Research: Sveltos for multi-cluster manifest delivery (firewalled cluster) #31

ginjiruu referenced this issue

2026-08-12 16:01:24 +00:00

Multi-cluster catalog: cluster registry + per-app deployment targets #32

ginjiruu referenced this issue

2026-08-12 16:01:47 +00:00

Multi-cluster catalog: cluster registry + per-app deployment targets #33