Argo CD: in-place migration of cluster auth from static SA to Kanidm SA + OIDC #37

New issue

Open

opened 2026-08-12 16:50:32 +00:00 by ginjiruu · 0 comments

ginjiruu commented

2026-08-12 16:50:32 +00:00

Owner

Argo CD: in-place migration of cluster auth from static SA to Kanidm SA + OIDC

Mechanism resolved by #38 (Golden Path): no refresh machinery. Argo CD cluster secrets point at the k8s-hub TLS backend (Cilium Gateway API, Let's Encrypt) and carry the Kanidm SA API token as a static bearerToken; the authz-bridge performs the RFC 8693 token exchange per request. #39 resolved (2026-08-12) — exchange verified end-to-end; no conformance blocker remains. (2026-09: the gateway migrated from agentgateway to the Cilium Gateway API k8s-hub.)

Current state (verified)

Two Argo CD cluster secrets on portable: cluster-10.15.1.170-1476597603 (mini) and cluster-mail.animeteamspeak.moe-4231318278 (remote).
Both use tlsClientConfig (caData + certData + keyData) — i.e. TLS client certs from a bootstrap SA. No bearer token today.

Target state

Each Argo CD cluster secret authenticates as a Kanidm SA via an OIDC id_token, resolved to RBAC through the apiserver's OIDC groups_name claim.
Credential delivery (Golden Path, #38): the secret's config is
```
{
  "bearerToken": "<argocd-<cluster> SA API token>"
}
```
with server: https://<cluster>.k8s.animeteamspeak.moe (k8s-hub route, matching the argocd_cluster server/name in argocd/apps.tf). No tlsClientConfig needed — the route terminates with a Let's Encrypt cert (public trust). The authz-bridge exchanges the SA API token per request and forwards a fresh token to the real apiserver. No exec, no refresh, no stored short-lived token.
The argocd_cluster resources declared in iac (argocd/apps.tf, register_clusters, T15) supply server/name; the credential side lives in the cluster secrets.

Migration path (in place, per cluster)

#35: create the argocd-<cluster> Kanidm SA (the per-cluster OAuth2 client, e.g. minicluster, is provisioned in kanidm/oauth2.tf).
Stand up the k8s-hub apiserver routes (authz-bridge ExternalAuth + per-cluster OauthClient/TokenExchangePolicy, public client, audience = the cluster's apiserver client, scopes openid groups_name) — live in flux-clusters generic-templates/k8s-hub (exchange first verified under agentgateway, #39 resolved 2026-08-12).
Bind the cluster-admin-equivalent ClusterRole to the SA's OIDC identity on the target cluster (via the capability groups, ADR-0006 model).
Write a new cluster secret for that cluster: server → k8s-hub route (https://<cluster>.k8s.animeteamspeak.moe), bearerToken → the SA API token, no CA (Let's Encrypt, public trust). Argo CD picks it up and reconciles against the new identity.
Verify apps sync and health stay green; confirm the apiserver sees the Kanidm identity (not system:admin).
Delete the old TLS-cert cluster secret + the bootstrap SA/CRB.

Open questions

None deferred to #38 (resolved). Remaining Gate-2 item: apiserver OIDC flags live (flux-clusters #11/#1522) and a live proxy test that watch/exec streams survive the gateway — tracked in #38's Gate-2 deliverable.

Acceptance criteria

mini + remote Argo CD cluster secrets authenticate as Kanidm SAs (OIDC) through the k8s-hub
Apps reconcile and health checks stay green during the switch
Static bootstrap SAs/CRBs removed; access revoked by removing the Kanidm SA
No stored short-lived token anywhere (SA API token is the only secret, in the cluster secret, exchanged per request)

Depends on / linked

#35 (Kanidm per-cluster clients + SAs)
#38 (Golden Path decision — mechanism) and #39 (Kanidm conformance blocker — resolved)
flux-clusters #11 / #1522 (apiserver OIDC flags live)
T13 #28 (kubeconfig/credential distribution; cluster registration)
T15 #32 (argocd_cluster resources, register_clusters flag)
#36 (Sveltos parallel migration — shares #35/#38/#39)

## Argo CD: in-place migration of cluster auth from static SA to Kanidm SA + OIDC > Mechanism resolved by **#38 (Golden Path)**: no refresh machinery. Argo CD cluster secrets point at the k8s-hub TLS backend (Cilium Gateway API, Let's Encrypt) and carry the Kanidm SA API token as a static `bearerToken`; the authz-bridge performs the RFC 8693 token exchange per request. **#39 resolved (2026-08-12)** — exchange verified end-to-end; no conformance blocker remains. *(2026-09: the gateway migrated from agentgateway to the Cilium Gateway API k8s-hub.)* ## Current state (verified) - Two Argo CD cluster secrets on portable: `cluster-10.15.1.170-1476597603` (mini) and `cluster-mail.animeteamspeak.moe-4231318278` (remote). - Both use `tlsClientConfig` (caData + certData + keyData) — i.e. TLS client certs from a bootstrap SA. No bearer token today. ## Target state - Each Argo CD cluster secret authenticates as a Kanidm SA via an OIDC id_token, resolved to RBAC through the apiserver's OIDC `groups_name` claim. - **Credential delivery (Golden Path, #38):** the secret's `config` is ```json { "bearerToken": "<argocd-<cluster> SA API token>" } ``` with `server: https://<cluster>.k8s.animeteamspeak.moe` (k8s-hub route, matching the `argocd_cluster` `server`/`name` in `argocd/apps.tf`). No `tlsClientConfig` needed — the route terminates with a Let's Encrypt cert (public trust). The authz-bridge exchanges the SA API token per request and forwards a fresh token to the real apiserver. **No exec, no refresh, no stored short-lived token.** - The `argocd_cluster` resources declared in iac (`argocd/apps.tf`, `register_clusters`, T15) supply `server`/`name`; the credential side lives in the cluster secrets. ## Migration path (in place, per cluster) 1. #35: create the `argocd-<cluster>` Kanidm SA (the per-cluster OAuth2 client, e.g. `minicluster`, is provisioned in `kanidm/oauth2.tf`). 2. Stand up the k8s-hub apiserver routes (authz-bridge `ExternalAuth` + per-cluster `OauthClient`/`TokenExchangePolicy`, public client, audience = the cluster's apiserver client, scopes `openid groups_name`) — live in flux-clusters `generic-templates/k8s-hub` (exchange first verified under agentgateway, **#39 resolved 2026-08-12**). 3. Bind the cluster-admin-equivalent ClusterRole to the SA's OIDC identity on the target cluster (via the capability groups, ADR-0006 model). 4. Write a **new** cluster secret for that cluster: `server` → k8s-hub route (`https://<cluster>.k8s.animeteamspeak.moe`), `bearerToken` → the SA API token, no CA (Let's Encrypt, public trust). Argo CD picks it up and reconciles against the new identity. 5. Verify apps sync and health stay green; confirm the apiserver sees the Kanidm identity (not `system:admin`). 6. Delete the old TLS-cert cluster secret + the bootstrap SA/CRB. ## Open questions - None deferred to #38 (resolved). Remaining Gate-2 item: apiserver OIDC flags live (flux-clusters #11/#1522) and a live proxy test that watch/exec streams survive the gateway — tracked in #38's Gate-2 deliverable. ## Acceptance criteria - [ ] mini + remote Argo CD cluster secrets authenticate as Kanidm SAs (OIDC) through the k8s-hub - [ ] Apps reconcile and health checks stay green during the switch - [ ] Static bootstrap SAs/CRBs removed; access revoked by removing the Kanidm SA - [ ] No stored short-lived token anywhere (SA API token is the only secret, in the cluster secret, exchanged per request) ## Depends on / linked - #35 (Kanidm per-cluster clients + SAs) - #38 (Golden Path decision — mechanism) and **#39** (Kanidm conformance blocker — **resolved**) - flux-clusters #11 / #1522 (apiserver OIDC flags live) - T13 #28 (kubeconfig/credential distribution; cluster registration) - T15 #32 (argocd_cluster resources, register_clusters flag) - #36 (Sveltos parallel migration — shares #35/#38/#39)

ginjiruu referenced this issue

2026-08-12 16:51:36 +00:00

Research: short-lived id_token refresh for static cluster credentials #38

ginjiruu referenced this issue

2026-08-12 16:51:47 +00:00

Migrate Sveltos cluster auth from static SA to Kanidm SA + OIDC #36

ginjiruu referenced this issue

2026-08-12 16:52:13 +00:00

Kanidm: provision per-cluster OAuth2 clients + machine SAs for cluster tooling #35

ginjiruu referenced this issue

2026-08-12 18:24:13 +00:00

Kanidm RFC 8693 conformance: issued_token_type serialization breaks agentgateway token exchange #39