Migrate Sveltos cluster auth from static SA to Kanidm SA + OIDC

ginjiruu commented

2026-08-12 16:50:17 +00:00

Owner

Migrate Sveltos cluster auth from static SA to Kanidm SA + OIDC

Mechanism resolved by #38 (Golden Path): no refresh machinery. Sveltos kubeconfigs point at the agentgateway TLS backend and carry the Kanidm SA API token as a static token; agentgateway performs the RFC 8693 token exchange per request. #39 resolved (2026-08-12) — exchange verified end-to-end; no conformance blocker remains.

Follow-on to #34 (Sveltos bootstrap with a static cluster-admin SA). Once the k8s apiserver OIDC flags (flux-clusters #11/#1522) and the Kanidm kubernetes client (#35) exist, switch Sveltos's connection to each fleet cluster to a Kanidm service account authenticating via OIDC — same RFC 8693 flow Argo CD and tofu-controller already use — so there are no static long-lived SA credentials on the fleet.

Note: keeping Sveltos on k8s ServiceAccount auth with its built-in tokenRequestRenewalOption was explored and not deemed valid — it renews k8s SA tokens rather than Kanidm tokens. The decision is that all cluster tooling authenticates as Kanidm service accounts (ADR-0006 machine-as-principal). Under the Golden Path this option is simply never needed.

Current state (after #34)

Sveltos connects to each cluster with a cluster-admin SA kubeconfig created by sveltosctl generate kubeconfig --create, stored as a <cluster>-sveltos-kubeconfig Secret in the management cluster (mini).
This is the same pattern Argo CD's current cluster secrets use (TLS client certs from a static bootstrap SA).

Target state

Each fleet cluster's <cluster>-sveltos-kubeconfig Secret holds a kubeconfig whose auth is a Kanidm OIDC identity (sveltos-<cluster> SA via #35), resolved to RBAC through the apiserver's OIDC groups claim.

Credential delivery (Golden Path, #38): the kubeconfig carries

users:
- name: sveltos-<cluster>
  user:
    token: <sveltos-<cluster> SA API token>   # the only secret; per-actor
clusters:
- name: <cluster>
  cluster:
    server: https://<agentgateway apiserver route>
    certificate-authority-data: <agentgateway CA>

agentgateway exchanges the SA API token per request and forwards a fresh token to the real apiserver. No exec credential plugin, no expirationTimestamp, no Secret rewrite loop.

Migration path (in place, per cluster)

#35: create the sveltos-<cluster> Kanidm SA + kubernetes client.
Stand up the agentgateway apiserver routes (backendAuth.oauthTokenExchange, backendTLS: {}, public client, audiences: [kubernetes], scope: openid groups) — exchange verified (#39 resolved 2026-08-12).
Bind the required ClusterRole(s) to the SA's OIDC identity on the cluster (via the capability groups, ADR-0006 model).
Rewrite the <cluster>-sveltos-kubeconfig Secret to the Golden-Path form above.
Verify: sveltosctl show addons still green, ClusterProfiles reconcile, drift detection works.
Delete the bootstrap SA + ClusterRoleBinding.

Acceptance criteria

No static long-lived SA credentials remain in fleet-cluster kubeconfigs (the only secret is the per-actor Kanidm SA API token)
Sveltos reconciles through OIDC-issued identity; revocation = removing the Kanidm SA (no cluster-state cleanup)
No refresh machinery: no exec provider, no stored short-lived token

Depends on / linked

#38 (Golden Path decision — mechanism) and #39 (Kanidm conformance blocker — resolved)
#34 (Sveltos bootstrap) — the base to migrate
#35 (Kanidm kubernetes client + SAs) — the credential source
flux-clusters #11 / #1522 (apiserver OIDC flags live)
Argo CD's parallel migration (#37) shares the same #35/#38/#39 prerequisites

## Migrate Sveltos cluster auth from static SA to Kanidm SA + OIDC > Mechanism resolved by **#38 (Golden Path)**: no refresh machinery. Sveltos kubeconfigs point at the agentgateway TLS backend and carry the Kanidm SA API token as a static `token`; agentgateway performs the RFC 8693 token exchange per request. **#39 resolved (2026-08-12)** — exchange verified end-to-end; no conformance blocker remains. Follow-on to #34 (Sveltos bootstrap with a static cluster-admin SA). Once the k8s apiserver OIDC flags (flux-clusters #11/#1522) and the Kanidm `kubernetes` client (#35) exist, switch Sveltos's connection to each fleet cluster to a Kanidm service account authenticating via OIDC — same RFC 8693 flow Argo CD and tofu-controller already use — so there are no static long-lived SA credentials on the fleet. > Note: keeping Sveltos on k8s ServiceAccount auth with its built-in `tokenRequestRenewalOption` was explored and **not deemed valid** — it renews k8s SA tokens rather than Kanidm tokens. The decision is that all cluster tooling authenticates as Kanidm service accounts (ADR-0006 machine-as-principal). Under the Golden Path this option is simply never needed. ## Current state (after #34) - Sveltos connects to each cluster with a cluster-admin SA kubeconfig created by `sveltosctl generate kubeconfig --create`, stored as a `<cluster>-sveltos-kubeconfig` Secret in the management cluster (mini). - This is the same pattern Argo CD's current cluster secrets use (TLS client certs from a static bootstrap SA). ## Target state - Each fleet cluster's `<cluster>-sveltos-kubeconfig` Secret holds a kubeconfig whose auth is a Kanidm OIDC identity (`sveltos-<cluster>` SA via #35), resolved to RBAC through the apiserver's OIDC `groups` claim. - **Credential delivery (Golden Path, #38):** the kubeconfig carries ```yaml users: - name: sveltos-<cluster> user: token: <sveltos-<cluster> SA API token> # the only secret; per-actor clusters: - name: <cluster> cluster: server: https://<agentgateway apiserver route> certificate-authority-data: <agentgateway CA> ``` agentgateway exchanges the SA API token per request and forwards a fresh token to the real apiserver. **No exec credential plugin, no `expirationTimestamp`, no Secret rewrite loop.** ## Migration path (in place, per cluster) 1. #35: create the `sveltos-<cluster>` Kanidm SA + `kubernetes` client. 2. Stand up the agentgateway apiserver routes (`backendAuth.oauthTokenExchange`, `backendTLS: {}`, public client, `audiences: [kubernetes]`, `scope: openid groups`) — exchange verified (**#39 resolved 2026-08-12**). 3. Bind the required ClusterRole(s) to the SA's OIDC identity on the cluster (via the capability groups, ADR-0006 model). 4. Rewrite the `<cluster>-sveltos-kubeconfig` Secret to the Golden-Path form above. 5. Verify: `sveltosctl show addons` still green, ClusterProfiles reconcile, drift detection works. 6. Delete the bootstrap SA + ClusterRoleBinding. ## Acceptance criteria - [ ] No static long-lived SA credentials remain in fleet-cluster kubeconfigs (the only secret is the per-actor Kanidm SA API token) - [ ] Sveltos reconciles through OIDC-issued identity; revocation = removing the Kanidm SA (no cluster-state cleanup) - [ ] No refresh machinery: no exec provider, no stored short-lived token ## Depends on / linked - #38 (Golden Path decision — mechanism) and **#39** (Kanidm conformance blocker — **resolved**) - #34 (Sveltos bootstrap) — the base to migrate - #35 (Kanidm `kubernetes` client + SAs) — the credential source - flux-clusters #11 / #1522 (apiserver OIDC flags live) - Argo CD's parallel migration (#37) shares the same #35/#38/#39 prerequisites

ginjiruu referenced this issue

2026-08-12 16:50:32 +00:00

Argo CD: in-place migration of cluster auth from static SA to Kanidm SA + OIDC #37

ginjiruu referenced this issue

2026-08-12 16:51:36 +00:00

Research: short-lived id_token refresh for static cluster credentials #38

ginjiruu referenced this issue

2026-08-12 18:24:13 +00:00

Kanidm RFC 8693 conformance: issued_token_type serialization breaks agentgateway token exchange #39

ginjiruu referenced this issue

2026-08-12 18:25:41 +00:00

Kanidm: provision per-cluster OAuth2 clients + machine SAs for cluster tooling #35