Research: short-lived id_token refresh for static cluster credentials #38

New issue

Open

opened 2026-08-12 16:51:36 +00:00 by ginjiruu · 0 comments

ginjiruu commented

2026-08-12 16:51:36 +00:00

Owner

Research: short-lived id_token refresh for static cluster credentials — DECIDED

Status: DECIDED (2026-08-12). The original question — "keep the OIDC bearer credential fresh for static stores" — is superseded: the Golden Path stores nothing, so nothing needs refreshing. The mechanism is a TLS-terminating gateway performing the RFC 8693 token exchange per request — originally agentgateway, migrated to the Cilium Gateway API k8s-hub authz-bridge in 2026-09. Full spike and evidence are below; the blocker is resolved (#39 closed) and the remaining deliverable is Gate 2.

The original problem

Kanidm id_tokens are short-lived, but the credential stores the cluster tooling uses are static — Argo CD cluster secrets, Sveltos's stored kubeconfig Secrets, and Vault's Kubernetes auth token_reviewer_jwt. The chain #35/#36/#37/#34 exists so Kanidm is the central identity store for all actors (humans, agents, machines), and every tool that trusts Kanidm authenticates as a Kanidm service account (ADR-0006 machine-as-principal).

Note: Sveltos' built-in tokenRequestRenewalOption was explored and still not valid — it renews k8s ServiceAccount tokens via the remote TokenRequest API rather than minting Kanidm tokens, so it keeps consumers on k8s-native SA auth. Under the Golden Path it is simply never needed.

Candidate mechanisms (evaluated)

Exec credential plugin in the stored kubeconfig — re-examined: Argo CD cluster secrets do support execProviderConfig (argoproj/argo-cd#4600) and Sveltos supports exec providers (self-refreshing the stored Secret). Superseded — a kubeconfig carrying a static token pointed at the k8s-hub needs no exec at all.
Refresh CronJob (per-cluster refresh → Vault KV → VSO distribution) — the originally-leaned recommendation. Superseded — a gateway performs the exchange per request; there is no write path to maintain, no token to store, no crown-jewel refresher to secure, no token_reviewer_jwt single-writer (ignore_changes) hazard.
IaC-provider-driven refresh — fragile, gated on human approvePlan / reconcile cadence, does not cover Sveltos. Rejected.
Proxy/sidecar in the data path — CHOSEN, in the form of a gateway per-request token exchange (agentgateway backendAuth.oauthTokenExchange; now the k8s-hub authz-bridge on Cilium).

DECISION — Golden Path: per-request token exchange at the gateway (agentgateway → Cilium k8s-hub, 2026-09)

Every actor (human or machine) holds a single Kanidm identity; no stored short-lived tokens exist anywhere. For machines this is a Kanidm service-account API token — the only long-lived credential, held by the actor itself, never replicated. All access to Kanidm-trusting lab services flows through the k8s-hub, a TLS-terminating Gateway API (Cilium) proxy whose authz-bridge performs the RFC 8693 exchange per request:

request:  Authorization: Bearer <kanidm SA API token>
exchange: POST https://auth.animeteamspeak.moe/oauth2/token
            grant_type=urn:ietf:params:oauth:grant-type:token-exchange
            client_id=<cluster apiserver client-id>    (public client, no secret; e.g. minicluster)
            subject_token=<SA API token>
            subject_token_type=urn:ietf:params:oauth:token-type:access_token
            audience=<cluster apiserver client-id>
            scope=openid groups_name
forward:  Authorization: Bearer <fresh token> → backend

The gateway caches the exchanged token only until Kanidm's expires_in (900s). The stale window is gone — a fresh token is minted on demand; the subject token is never rotated or written into consumer stores.

Per-consumer shape:

Consumer	Config	vs today
Argo CD cluster secret (#37)	`server: https://<cluster>.k8s.animeteamspeak.moe`, `config.bearerToken: <argocd-<cluster> SA API token>`, no CA (Let's Encrypt, public trust)	No TLS client certs, no exec, no refresh
Sveltos kubeconfig (#36)	`server: https://<cluster>.k8s.animeteamspeak.moe`, `token: <sveltos-<cluster> SA API token>`	No exec, no refresh, no Secret rewrite
Vault kubernetes auth (#29)	`token_reviewer_jwt` is the `vault` Kanidm SA API token; TokenReview calls go through the k8s-hub apiserver route (exchanged per request, audience = the cluster's apiserver client)	Same path as Argo/Sveltos — even in-cluster: Vault is decoupled from the cluster it runs on and its config is unchanged if it moves. The pod's SA token still travels in the TokenReview body and is validated by the apiserver itself
kubectl (humans, #28)	unchanged — kubelogin exec plugin	Not affected

agentgateway config essentials (v1.4.1) — historical; the gateway migrated to the Cilium Gateway API k8s-hub authz-bridge in 2026-09 (per-cluster OauthClient/TokenExchangePolicy/OidcPolicy in the oauth.animeteamspeak.moe/providers.animeteamspeak.moe groups, scopes openid groups_name, exchanged id_token forwarded, Let's Encrypt TLS):

Token endpoint: host: auth.animeteamspeak.moe:443, path: /oauth2/token, policies: { backendTLS: {} } is required — TLS does not auto-enable on :443 despite the docs (without it the gateway connects plaintext and the ingress 404s).
clientAuth: { clientId: <service>, method: clientSecretPost } — public client, no secret, matches kanidm-login.py.
audiences: [<service-client-id>], scopes: [openid, groups]. The gateway forwards the exchanged access_token (RFC 9068 at+jwt) upstream.

Validated facts (live, kanidm/server 1.10.4 + agentgateway v1.4.1):

Kanidm exchange response: access_token (at+jwt, aud=<client>), id_token, refresh_token, expires_in: 900, token_type: Bearer.
Public secret-less clients work (clientSecretPost).
The groups claim requires the client's scope map to grant groups to the SA's groups (ADR-0005). SAs whose groups lack it get access_denied (e.g. kanidmclaimtesting in argocd_users: available scopes {argocd_role, email, openid}).
BLOCKER — RESOLVED (#39 closed 2026-08-12): Kanidm returned issued_token_type: "AccessToken" (bare enum name) instead of the RFC 8693 URN; agentgateway strictly validated and rejected it. Fixed with patched Kanidm 1.11.0-itt-fix on portable; the direct exchange now returns the URN and the agentgateway E2E returns 200. Not upstreamed.

Deliverables (status)

Recommended mechanism: per-request RFC 8693 token exchange at the gateway (agentgateway; now the Cilium k8s-hub authz-bridge) — one mechanism, all consumers, nothing stored.
Token TTL / refresh cadence: none — minted per request, cached to Kanidm expires_in (900s). No stale window.
Feeding into #36 / #37: target state = point at the k8s-hub with the SA API token. Vault token_reviewer_jwt: the vault Kanidm SA API token through the gateway, same path even in-cluster (see #29).
Blocker: Kanidm issued_token_type conformance — #39 closed (patched Kanidm 1.11.0-itt-fix, exchange verified 200).
Gate 2 (k8s path): apiserver OIDC flags live (flux-clusters #11 / #1522) + a real proxy test that the apiserver accepts the forwarded token and watch/exec streams survive.

Linked

#35 (Kanidm per-cluster clients + SAs — the credential source)
#36 (Sveltos OIDC switch), #37 (Argo CD OIDC switch)
#29 (T14 Vault Kubernetes auth — token_reviewer_jwt resolved here)
#39 (Kanidm conformance blocker — closed, not upstreamed)
#28 (T13 connectivity — the k8s-hub is the concrete gateway)
flux-clusters #11 / #1522 (apiserver OIDC flags)

> ## Research: short-lived id_token refresh for static cluster credentials — **DECIDED** > > Status: **DECIDED (2026-08-12)**. The original question — "keep the OIDC bearer credential fresh for static stores" — is **superseded**: the Golden Path stores nothing, so nothing needs refreshing. The mechanism is **a TLS-terminating gateway performing the RFC 8693 token exchange per request** — originally agentgateway, migrated to the **Cilium Gateway API k8s-hub authz-bridge in 2026-09**. Full spike and evidence are below; the blocker is resolved (#39 closed) and the remaining deliverable is Gate 2. ## The original problem Kanidm id_tokens are short-lived, but the credential stores the cluster tooling uses are **static** — Argo CD cluster secrets, Sveltos's stored kubeconfig Secrets, and Vault's Kubernetes auth `token_reviewer_jwt`. The chain #35/#36/#37/#34 exists so Kanidm is the central identity store for all actors (humans, agents, machines), and every tool that trusts Kanidm authenticates as a Kanidm service account (ADR-0006 machine-as-principal). > Note: Sveltos' built-in `tokenRequestRenewalOption` was explored and **still not valid** — it renews k8s ServiceAccount tokens via the remote TokenRequest API rather than minting Kanidm tokens, so it keeps consumers on k8s-native SA auth. Under the Golden Path it is simply never needed. ## Candidate mechanisms (evaluated) 1. **Exec credential plugin in the stored kubeconfig** — re-examined: Argo CD cluster secrets **do** support `execProviderConfig` (argoproj/argo-cd#4600) and Sveltos supports exec providers (self-refreshing the stored Secret). **Superseded** — a kubeconfig carrying a static token pointed at the k8s-hub needs no exec at all. 2. **Refresh CronJob** (per-cluster refresh → Vault KV → VSO distribution) — the originally-leaned recommendation. **Superseded** — a gateway performs the exchange per request; there is no write path to maintain, no token to store, no crown-jewel refresher to secure, no `token_reviewer_jwt` single-writer (`ignore_changes`) hazard. 3. **IaC-provider-driven refresh** — fragile, gated on human approvePlan / reconcile cadence, does not cover Sveltos. **Rejected.** 4. **Proxy/sidecar in the data path** — **CHOSEN**, in the form of a gateway per-request token exchange (agentgateway `backendAuth.oauthTokenExchange`; now the k8s-hub authz-bridge on Cilium). ## DECISION — Golden Path: per-request token exchange at the gateway (agentgateway → Cilium k8s-hub, 2026-09) **Every actor (human or machine) holds a single Kanidm identity; no stored short-lived tokens exist anywhere.** For machines this is a Kanidm service-account **API token** — the only long-lived credential, held by the actor itself, never replicated. All access to Kanidm-trusting lab services flows through the k8s-hub, a TLS-terminating Gateway API (Cilium) proxy whose authz-bridge performs the RFC 8693 exchange **per request**: ``` request: Authorization: Bearer <kanidm SA API token> exchange: POST https://auth.animeteamspeak.moe/oauth2/token grant_type=urn:ietf:params:oauth:grant-type:token-exchange client_id=<cluster apiserver client-id> (public client, no secret; e.g. minicluster) subject_token=<SA API token> subject_token_type=urn:ietf:params:oauth:token-type:access_token audience=<cluster apiserver client-id> scope=openid groups_name forward: Authorization: Bearer <fresh token> → backend ``` The gateway caches the exchanged token only until Kanidm's `expires_in` (900s). The stale window is gone — a fresh token is minted on demand; the subject token is never rotated or written into consumer stores. **Per-consumer shape:** | Consumer | Config | vs today | |---|---|---| | Argo CD cluster secret (#37) | `server: https://<cluster>.k8s.animeteamspeak.moe`, `config.bearerToken: <argocd-<cluster> SA API token>`, no CA (Let's Encrypt, public trust) | No TLS client certs, no exec, no refresh | | Sveltos kubeconfig (#36) | `server: https://<cluster>.k8s.animeteamspeak.moe`, `token: <sveltos-<cluster> SA API token>` | No exec, no refresh, no Secret rewrite | | Vault kubernetes auth (#29) | `token_reviewer_jwt` is the **`vault` Kanidm SA API token**; TokenReview calls go through the k8s-hub apiserver route (exchanged per request, audience = the cluster's apiserver client) | Same path as Argo/Sveltos — even in-cluster: Vault is decoupled from the cluster it runs on and its config is unchanged if it moves. The pod's SA token still travels in the TokenReview body and is validated by the apiserver itself | | kubectl (humans, #28) | unchanged — kubelogin exec plugin | Not affected | **agentgateway config essentials (v1.4.1)** — *historical; the gateway migrated to the Cilium Gateway API k8s-hub authz-bridge in 2026-09 (per-cluster `OauthClient`/`TokenExchangePolicy`/`OidcPolicy` in the `oauth.animeteamspeak.moe`/`providers.animeteamspeak.moe` groups, scopes `openid groups_name`, exchanged **id_token** forwarded, Let's Encrypt TLS)*: - Token endpoint: `host: auth.animeteamspeak.moe:443`, `path: /oauth2/token`, **`policies: { backendTLS: {} }` is required** — TLS does not auto-enable on :443 despite the docs (without it the gateway connects plaintext and the ingress 404s). - `clientAuth: { clientId: <service>, method: clientSecretPost }` — public client, no secret, matches `kanidm-login.py`. - `audiences: [<service-client-id>]`, `scopes: [openid, groups]`. The gateway forwards the exchanged **`access_token`** (RFC 9068 `at+jwt`) upstream. **Validated facts (live, kanidm/server 1.10.4 + agentgateway v1.4.1):** - Kanidm exchange response: `access_token` (`at+jwt`, `aud=<client>`), `id_token`, `refresh_token`, `expires_in: 900`, `token_type: Bearer`. - Public secret-less clients work (`clientSecretPost`). - The `groups` claim requires the client's **scope map** to grant `groups` to the SA's groups (ADR-0005). SAs whose groups lack it get `access_denied` (e.g. `kanidmclaimtesting` in `argocd_users`: available scopes `{argocd_role, email, openid}`). - **BLOCKER — RESOLVED (#39 closed 2026-08-12)**: Kanidm returned `issued_token_type: "AccessToken"` (bare enum name) instead of the RFC 8693 URN; agentgateway strictly validated and rejected it. Fixed with patched Kanidm `1.11.0-itt-fix` on portable; the direct exchange now returns the URN and the agentgateway E2E returns 200. Not upstreamed. ## Deliverables (status) - [x] Recommended mechanism: **per-request RFC 8693 token exchange at the gateway** (agentgateway; now the Cilium k8s-hub authz-bridge) — one mechanism, all consumers, nothing stored. - [x] Token TTL / refresh cadence: **none** — minted per request, cached to Kanidm `expires_in` (900s). No stale window. - [x] Feeding into #36 / #37: target state = point at the k8s-hub with the SA API token. Vault `token_reviewer_jwt`: the `vault` Kanidm SA API token through the gateway, same path even in-cluster (see #29). - [x] **Blocker**: Kanidm `issued_token_type` conformance — **#39 closed** (patched Kanidm `1.11.0-itt-fix`, exchange verified 200). - [ ] Gate 2 (k8s path): apiserver OIDC flags live (flux-clusters #11 / #1522) + a real proxy test that the apiserver accepts the forwarded token and watch/exec streams survive. ## Linked - #35 (Kanidm per-cluster clients + SAs — the credential source) - #36 (Sveltos OIDC switch), #37 (Argo CD OIDC switch) - #29 (T14 Vault Kubernetes auth — `token_reviewer_jwt` resolved here) - #39 (Kanidm conformance blocker — **closed**, not upstreamed) - #28 (T13 connectivity — the k8s-hub is the concrete gateway) - flux-clusters #11 / #1522 (apiserver OIDC flags)

ginjiruu referenced this issue

2026-08-12 16:51:47 +00:00

Migrate Sveltos cluster auth from static SA to Kanidm SA + OIDC #36

ginjiruu referenced this issue

2026-08-12 16:51:56 +00:00

Argo CD: in-place migration of cluster auth from static SA to Kanidm SA + OIDC #37

ginjiruu referenced this issue

2026-08-12 16:52:13 +00:00

Kanidm: provision per-cluster OAuth2 clients + machine SAs for cluster tooling #35

ginjiruu referenced this issue

2026-08-12 18:24:13 +00:00

Kanidm RFC 8693 conformance: issued_token_type serialization breaks agentgateway token exchange #39