Kanidm RFC 8693 conformance: issued_token_type serialization breaks agentgateway token exchange #39

New issue

Closed

opened 2026-08-12 18:24:13 +00:00 by ginjiruu · 1 comment

ginjiruu commented 2026-08-12 18:24:13 +00:00

Owner

Copy link

Kanidm RFC 8693 conformance: issued_token_type serialization breaks agentgateway token exchange

Decision: tracked internally only — do NOT report upstream to the Kanidm project. This is a deliberate call (2026-08-12): the fix is a ~1-line change on either side and we may choose to patch our own agentgateway build instead of waiting on an upstream release. If we later change our mind, the upstream report is a pre-written, ready-to-file issue (see "Upstream filing packet" at the bottom).

Status

Open / blocking the Golden Path. This is the single hard blocker for the agentgateway per-request token-exchange architecture (#38). All three cluster-tooling consumers (#36 Sveltos, #37 Argo CD) and every future Kanidm-trusting service (homebox, sonarr, …) depend on it.

Why it matters

The Golden Path (#38) replaces all short-lived-token refresh machinery with an agentgateway that performs the RFC 8693 token exchange per request:

POST https://auth.animeteamspeak.moe/oauth2/token
  grant_type=urn:ietf:params:oauth:grant-type:token-exchange
  client_id=<service-client-id>                    (public client, no secret)
  subject_token=<kanidm SA API token>
  subject_token_type=urn:ietf:params:oauth:token-type:access_token
  audience=<service-client-id>
  scope=openid groups

The gateway then forwards the exchanged token upstream. This works end-to-end against a conformant authorization server, but fails against Kanidm because Kanidm's exchange response advertises a non-standard issued_token_type.

Reproduction (verified live 2026-08-12)

Environment: kanidm/server 1.10.4 (deployed on portable), agentgateway standalone v1.4.1.

1. Create a Kanidm service account, add it to a group with a scope-map entry for the argocd OAuth2 client, generate an API token.
2. Run the RFC 8693 exchange with scope=openid:

   curl -s "https://auth.animeteamspeak.moe/oauth2/token" \
     --data-urlencode grant_type=urn:ietf:params:oauth:grant-type:token-exchange \
     --data-urlencode client_id=argocd \
     --data-urlencode subject_token="$SA_TOKEN" \
     --data-urlencode subject_token_type=urn:ietf:params:oauth:token-type:access_token \
     --data-urlencode audience=argocd \
     --data-urlencode scope=openid
   

   Response (sensitive fields elided):

   {
     "token_type": "Bearer",
     "issued_token_type": "AccessToken",
     "expires_in": 900,
     "scope": "openid",
     "access_token": "…",
     "id_token": "…",
     "refresh_token": "…"
   }
   

3. Point agentgateway at this token endpoint (with policies.backendTLS: {} — see config notes below) and send a request through it. It fails:

   backend authentication failed: token exchange returned unusable issued_token_type: AccessToken
   

The exchange also returns id_token alongside access_token, but agentgateway forwards the access_token field only.

Root cause

RFC 8693 (§3.1) requires issued_token_type to be a URI, and the standard for an access token is the URN urn:ietf:params:oauth:token-type:access_token.

• Kanidm emits the bare enum name instead of the URN. kanidm/proto/src/oauth2.rs:

  #[derive(Serialize, Deserialize, Debug, PartialEq)]
  pub enum IssuedTokenType {
      AccessToken,   // serializes as the literal string "AccessToken"
      RefreshToken,
      IdToken,
      Saml1,
      Saml2,
  }
  

  Confirmed present in tag v1.10.4 and unfixed on kanidm main (checked 2026-08-12). requested_token_type does not influence it: the response is "AccessToken" whether the field is omitted or explicitly urn:ietf:params:oauth:token-type:access_token; requesting an id_token/jwt URN returns invalid_request.
• agentgateway strictly validates the field and only accepts the registered URNs. crates/agentgateway/src/http/auth/oauth/mod.rs OAuthTokenType::from_urn has no AccessToken branch, so Kanidm's value parses to None and the exchange is rejected with unusable issued_token_type: AccessToken. Confirmed in both v1.4.1 and agentgateway main. Even with requested_token_type unset, agentgateway expects access_token (empirically verified: a mock returning a different-but-valid URN was rejected with expected urn:ietf:params:oauth:token-type:access_token).

Agentgateway config notes (from the same spike, for anyone re-running it)

• TLS does not auto-enable on port 443 despite the docs. The token-endpoint host must set policies: { backendTLS: {} }, or the gateway connects plaintext to :443 and the ingress returns 404.
• Public (secret-less) clients work: clientAuth: { clientId: argocd, method: clientSecretPost } with no clientSecret — matches how kanidm-login.py authenticates.
• scope: openid groups is rejected with access_denied for a principal whose groups have no groups scope-map entry on the client (ADR-0005 gap, separate from this bug). available_scopes in the Kanidm server log showed {argocd_role, email, openid}.

Impact

• Blocker: no consumer (Sveltos kubeconfig, Argo CD cluster secret, future homebox/sonarr routes) can go through agentgateway until resolved.
• The k8s apiserver OIDC path additionally needs the token's aud to be the kubernetes client and flux-clusters #11/#1522 live — those are tracked separately.
• Workaround with no gateway: Argo CD exec-plugin (execProviderConfig) or Sveltos exec-provider still work today, because a custom wrapper can read the id_token/access_token from the raw Kanidm response. The gateway path needs the fix.

Proposed fixes (evaluate when implementing)

1. Patch Kanidm (preferred for ecosystem correctness): implement Serialize/Deserialize for IssuedTokenType that maps the variants to/from the RFC 8693 URNs (urn:ietf:params:oauth:token-type:{access_token,refresh_token,id_token,saml1,saml2}). ~1-line change. Would fix every OIDC client, not just agentgateway.
2. Patch agentgateway: make the issued_token_type validation tolerant — treat "AccessToken" as the access-token URN, or skip the check when no requestedTokenType is configured. Smaller surface; fixes only our gateways.
3. Do nothing / pin: accept the gap and keep exec-plugin + a small wrapper that parses the raw response. Least desirable — keeps the refresh-era complexity the Golden Path removes.

Acceptance criteria (when the blocker is lifted)

• curl exchange against Kanidm returns issued_token_type: urn:ietf:params:oauth:token-type:access_token (or agentgateway accepts the response).
• A request through agentgateway backendAuth.oauthTokenExchange (token endpoint = Kanidm, backendTLS: {}, public client, scope: openid groups) forwards a fresh token upstream (200).
• The forwarded token carries aud=<service-client-id> and the groups claim when the SA's groups have the scope map.

Upstream filing packet

If we ever decide to report upstream, this is the material. Expected format: kanidm/kanidm issue under Server, labelled oauth2, with the reproduction in the "Reproduction" section above and a pointer to proto/src/oauth2.rs IssuedTokenType. Include the note that requested_token_type is ignored by the server when serializing the response.

Linked

• #38 (Golden Path decision — this is its hard blocker)
• #36 (Sveltos OIDC migration — gated on this)
• #37 (Argo CD OIDC migration — gated on this)
• #35 (Kanidm kubernetes client + machine SAs — the credential source)
• #11 / flux-clusters #1522 (apiserver OIDC flags — needed for the k8s audience path)

RESOLUTION — CLOSED (2026-08-12)

Fixed server-side and verified. Kanidm 1.11.0-itt-fix (harbor.animeteamspeak.moe/library/kanidm-server:1.11.0-itt-fix, deployed to portable) implements the IssuedTokenType URN serialization fix. Re-ran the full agentgateway E2E against it:

1. Direct exchange now returns issued_token_type: "urn:ietf:params:oauth:token-type:access_token" (was the bare "AccessToken"). Kanidm logs show requested_token_type: Some("urn:ietf:params:oauth:token-type:access_token") with status_code: 200.
2. agentgateway E2E (gateway container → real Kanidm, backendTLS: {}, public client, audiences: [argocd], scope: openid): the SA API token was exchanged and a fresh at+jwt forwarded upstream, HTTP 200 (previously backend authentication failed: unusable issued_token_type: AccessToken).

Remaining (not blockers for this issue):

• groups scope-map gap (ADR-0005): SAs whose groups lack a groups scope-map entry on the client still get access_denied for scope=openid groups. argocd_users currently exposes {argocd_role, email, openid}. Tracked as a provisioning requirement in #35.
• Gate 2 (k8s path): apiserver OIDC flags live (flux-clusters #11/#1522) + real proxy/stream test. Tracked in #38/#28.

Decision (unchanged): not filed upstream. The lab is self-hosting the patched build; the "Upstream filing packet" above remains available if that changes.

## Kanidm RFC 8693 conformance: `issued_token_type` serialization breaks agentgateway token exchange > **Decision: tracked internally only — do NOT report upstream to the Kanidm project.** This is a deliberate call (2026-08-12): the fix is a ~1-line change on either side and we may choose to patch our own agentgateway build instead of waiting on an upstream release. If we later change our mind, the upstream report is a pre-written, ready-to-file issue (see "Upstream filing packet" at the bottom). ## Status **Open / blocking the Golden Path.** This is the single hard blocker for the agentgateway per-request token-exchange architecture (#38). All three cluster-tooling consumers (#36 Sveltos, #37 Argo CD) and every future Kanidm-trusting service (homebox, sonarr, …) depend on it. ## Why it matters The Golden Path (#38) replaces all short-lived-token refresh machinery with an agentgateway that performs the RFC 8693 token exchange **per request**: ``` POST https://auth.animeteamspeak.moe/oauth2/token grant_type=urn:ietf:params:oauth:grant-type:token-exchange client_id=<service-client-id> (public client, no secret) subject_token=<kanidm SA API token> subject_token_type=urn:ietf:params:oauth:token-type:access_token audience=<service-client-id> scope=openid groups ``` The gateway then forwards the exchanged token upstream. This works end-to-end against a conformant authorization server, but **fails against Kanidm** because Kanidm's exchange response advertises a non-standard `issued_token_type`. ## Reproduction (verified live 2026-08-12) Environment: kanidm/server **1.10.4** (deployed on portable), agentgateway standalone **v1.4.1**. 1. Create a Kanidm service account, add it to a group with a scope-map entry for the `argocd` OAuth2 client, generate an API token. 2. Run the RFC 8693 exchange with `scope=openid`: ```bash curl -s "https://auth.animeteamspeak.moe/oauth2/token" \ --data-urlencode grant_type=urn:ietf:params:oauth:grant-type:token-exchange \ --data-urlencode client_id=argocd \ --data-urlencode subject_token="$SA_TOKEN" \ --data-urlencode subject_token_type=urn:ietf:params:oauth:token-type:access_token \ --data-urlencode audience=argocd \ --data-urlencode scope=openid ``` Response (sensitive fields elided): ```json { "token_type": "Bearer", "issued_token_type": "AccessToken", "expires_in": 900, "scope": "openid", "access_token": "…", "id_token": "…", "refresh_token": "…" } ``` 3. Point agentgateway at this token endpoint (with `policies.backendTLS: {}` — see config notes below) and send a request through it. It fails: ``` backend authentication failed: token exchange returned unusable issued_token_type: AccessToken ``` The exchange also returns `id_token` alongside `access_token`, but agentgateway forwards the `access_token` field only. ## Root cause RFC 8693 (§3.1) requires `issued_token_type` to be a URI, and the standard for an access token is the URN `urn:ietf:params:oauth:token-type:access_token`. - **Kanidm** emits the bare enum name instead of the URN. `kanidm/proto/src/oauth2.rs`: ```rust #[derive(Serialize, Deserialize, Debug, PartialEq)] pub enum IssuedTokenType { AccessToken, // serializes as the literal string "AccessToken" RefreshToken, IdToken, Saml1, Saml2, } ``` Confirmed present in tag v1.10.4 and **unfixed on kanidm main** (checked 2026-08-12). `requested_token_type` does not influence it: the response is `"AccessToken"` whether the field is omitted or explicitly `urn:ietf:params:oauth:token-type:access_token`; requesting an id_token/jwt URN returns `invalid_request`. - **agentgateway** strictly validates the field and only accepts the registered URNs. `crates/agentgateway/src/http/auth/oauth/mod.rs` `OAuthTokenType::from_urn` has no `AccessToken` branch, so Kanidm's value parses to `None` and the exchange is rejected with `unusable issued_token_type: AccessToken`. Confirmed in both v1.4.1 and agentgateway main. Even with `requested_token_type` unset, agentgateway expects `access_token` (empirically verified: a mock returning a different-but-valid URN was rejected with `expected urn:ietf:params:oauth:token-type:access_token`). ## Agentgateway config notes (from the same spike, for anyone re-running it) - **TLS does not auto-enable on port 443** despite the docs. The token-endpoint `host` must set `policies: { backendTLS: {} }`, or the gateway connects plaintext to :443 and the ingress returns 404. - Public (secret-less) clients work: `clientAuth: { clientId: argocd, method: clientSecretPost }` with no `clientSecret` — matches how `kanidm-login.py` authenticates. - `scope: openid groups` is rejected with `access_denied` for a principal whose groups have no `groups` scope-map entry on the client (ADR-0005 gap, separate from this bug). `available_scopes` in the Kanidm server log showed `{argocd_role, email, openid}`. ## Impact - **Blocker**: no consumer (Sveltos kubeconfig, Argo CD cluster secret, future homebox/sonarr routes) can go through agentgateway until resolved. - The k8s apiserver OIDC path additionally needs the token's `aud` to be the `kubernetes` client and flux-clusters #11/#1522 live — those are tracked separately. - Workaround with **no gateway**: Argo CD exec-plugin (`execProviderConfig`) or Sveltos exec-provider still work today, because a custom wrapper can read the `id_token`/`access_token` from the raw Kanidm response. The gateway path needs the fix. ## Proposed fixes (evaluate when implementing) 1. **Patch Kanidm** (preferred for ecosystem correctness): implement `Serialize`/`Deserialize` for `IssuedTokenType` that maps the variants to/from the RFC 8693 URNs (`urn:ietf:params:oauth:token-type:{access_token,refresh_token,id_token,saml1,saml2}`). ~1-line change. Would fix every OIDC client, not just agentgateway. 2. **Patch agentgateway**: make the `issued_token_type` validation tolerant — treat `"AccessToken"` as the access-token URN, or skip the check when no `requestedTokenType` is configured. Smaller surface; fixes only our gateways. 3. **Do nothing / pin**: accept the gap and keep exec-plugin + a small wrapper that parses the raw response. Least desirable — keeps the refresh-era complexity the Golden Path removes. ## Acceptance criteria (when the blocker is lifted) - [ ] `curl` exchange against Kanidm returns `issued_token_type: urn:ietf:params:oauth:token-type:access_token` (or agentgateway accepts the response). - [ ] A request through agentgateway `backendAuth.oauthTokenExchange` (token endpoint = Kanidm, `backendTLS: {}`, public client, `scope: openid groups`) forwards a fresh token upstream (200). - [ ] The forwarded token carries `aud=<service-client-id>` and the `groups` claim when the SA's groups have the scope map. ## Upstream filing packet If we ever decide to report upstream, this is the material. Expected format: kanidm/kanidm issue under `Server`, labelled `oauth2`, with the reproduction in the "Reproduction" section above and a pointer to `proto/src/oauth2.rs` `IssuedTokenType`. Include the note that `requested_token_type` is ignored by the server when serializing the response. ## Linked - #38 (Golden Path decision — this is its hard blocker) - #36 (Sveltos OIDC migration — gated on this) - #37 (Argo CD OIDC migration — gated on this) - #35 (Kanidm `kubernetes` client + machine SAs — the credential source) - #11 / flux-clusters #1522 (apiserver OIDC flags — needed for the k8s audience path) ## RESOLUTION — CLOSED (2026-08-12) **Fixed server-side and verified.** Kanidm `1.11.0-itt-fix` (`harbor.animeteamspeak.moe/library/kanidm-server:1.11.0-itt-fix`, deployed to portable) implements the `IssuedTokenType` URN serialization fix. Re-ran the full agentgateway E2E against it: 1. **Direct exchange** now returns `issued_token_type: "urn:ietf:params:oauth:token-type:access_token"` (was the bare `"AccessToken"`). Kanidm logs show `requested_token_type: Some("urn:ietf:params:oauth:token-type:access_token")` with `status_code: 200`. 2. **agentgateway E2E** (gateway container → real Kanidm, `backendTLS: {}`, public client, `audiences: [argocd]`, `scope: openid`): the SA API token was exchanged and a fresh `at+jwt` forwarded upstream, **HTTP 200** (previously `backend authentication failed: unusable issued_token_type: AccessToken`). **Remaining (not blockers for this issue):** - `groups` scope-map gap (ADR-0005): SAs whose groups lack a `groups` scope-map entry on the client still get `access_denied` for `scope=openid groups`. `argocd_users` currently exposes `{argocd_role, email, openid}`. Tracked as a provisioning requirement in #35. - Gate 2 (k8s path): apiserver OIDC flags live (flux-clusters #11/#1522) + real proxy/stream test. Tracked in #38/#28. **Decision (unchanged):** not filed upstream. The lab is self-hosting the patched build; the "Upstream filing packet" above remains available if that changes.

ginjiruu referenced this issue 2026-08-12 18:24:33 +00:00

Research: short-lived id_token refresh for static cluster credentials #38

ginjiruu referenced this issue 2026-08-12 18:25:40 +00:00

Argo CD: in-place migration of cluster auth from static SA to Kanidm SA + OIDC #37

ginjiruu referenced this issue 2026-08-12 18:25:40 +00:00

Migrate Sveltos cluster auth from static SA to Kanidm SA + OIDC #36

ginjiruu referenced this issue 2026-08-12 18:25:41 +00:00

Kanidm: provision per-cluster OAuth2 clients + machine SAs for cluster tooling #35

ginjiruu referenced this issue 2026-08-12 18:25:41 +00:00

T13: kubeconfig distribution & connectivity architecture (kubectl access) #28

ginjiruu commented 2026-08-13 01:39:50 +00:00

Author

Owner

Copy link

RESOLVED: Kanidm 1.11.0-itt-fix returns the RFC 8693 issued_token_type URN; agentgateway E2E passes (HTTP 200, fresh at+jwt forwarded). Remaining: groups scope-map gap (#35) and Gate 2 apiserver OIDC (#38/#28). Not filed upstream.

RESOLVED: Kanidm 1.11.0-itt-fix returns the RFC 8693 issued_token_type URN; agentgateway E2E passes (HTTP 200, fresh at+jwt forwarded). Remaining: groups scope-map gap (#35) and Gate 2 apiserver OIDC (#38/#28). Not filed upstream.

ginjiruu closed this issue 2026-08-13 01:39:51 +00:00

ginjiruu referenced this issue 2026-08-13 01:55:23 +00:00

T10: tofu-controller sync for provider roots (flux-clusters) #25

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

master

renovate/configure

t15-multicluster-catalog

t14-vault-k8s-auth

No results found.

Labels

Clear labels   

needs-info

Waiting on reporter for more information

  

needs-triage

Maintainer needs to evaluate this issue

  

ready-for-agent

Fully specified, ready for an AFK agent

  

ready-for-human

Requires human implementation

  

wontfix

Will not be actioned

No labels

needs-info

needs-triage

ready-for-agent

ready-for-human

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lab/iac#39

No description provided.