lab/authz-bridge
1
0
Fork 0
Code Issues 9 Pull requests 6 Projects Releases Packages Wiki Activity Actions 12

OIDC expired-session 4xx loop: bound the Check-path refresh to the 200ms ext_authz budget and fail to a 302 #60

New issue
Closed
opened 2026-09-20 18:34:23 +00:00 by ginjiruu · 2 comments
ginjiruu commented 2026-09-20 18:34:23 +00:00
Owner
Copy link

Problem

After an OIDC Session has expired, a browser holding the (stale) Session cookie is
greeted by repeated 4xx (gateway fail-closed 403, empty body, server: envoy)
on the protected endpoint instead of the 302 re-login. Refreshing does not help;
the only recovery observed was clearing the domain cookies and re-authorizing.

Reported live: admin.media.animeteamspeak.moe (App media/privatemedia, bridge on
the mini cluster, Kanidm on portable), 2026-09-20 ~16:00–16:14 UTC. The same
pattern occurred at 21:55 (Sep 19) and 06:16 (Sep 20) — recurring, not a one-off.

Evidence (correlated bridge + IdP logs)

Bridge (mini, authz-bridge-system, UTC):

Time Line
16:00:00 Session id_token failed validation — oidc: token is expired (Token Expiry: 2026-09-20 06:31:31 UTC)
16:00:01 The Session refresh failed — Post "https://auth.animeteamspeak.moe/oauth2/token": context canceled
16:00:03 The Session refresh failed — context canceled
16:00:04 The Session refresh failed — oauth2: "invalid_grant"
16:14:16 Renewed the Session
16:14:18–24 The Session refresh failed ×5 — oauth2: "invalid_grant"

Kanidm (portable, kanidm/kanidm-0; note kubectl --timestamps prints EDT, −04:00):

Time (UTC) POST /oauth2/token
16:00:01.151 200, 181ms (the refresh succeeded server-side)
16:00:03.333 400, 201ms
16:00:04.688 400, 61ms
16:14:16.779 200, 93ms
16:14:18.9–16:14:24.4 400 ×5 (32–68ms) — the same already-rotated refresh token

Two facts fall out of the correlation:

  1. The 4xx is the gateway's fail-closed 403, not the bridge's. The refresh
    round-trip (mini → portable gateway → kanidm nginx sidecar → kanidm; Kanidm alone
    181–201ms) exceeds the gateway's hardcoded 200ms ext_authz gRPC timeout; Envoy
    cancels the gRPC call (context canceled) and returns its own 403
    (failure_mode_allow=false). The bridge's 302 never reaches the browser.
    (Same root cause as #53: the GRPC branch of Cilium's GEP-1494 translation never
    sets Timeout — cilium/cilium#48354, no API knob.)
  2. Kanidm's refresh token is at-most-once. A refresh that succeeds server-side
    while the bridge's context is canceled at ~200ms loses the OK + re-issued
    Set-Cookie; the browser keeps presenting the now-dead refresh token, so every
    subsequent request is a refresh → 400 (invalid_grant) or a 403 when the
    round-trip is slow → the stuck 4xx loop. (Kanidm goes further than rotation: on
    refresh-token reuse it cancels the whole session — see the design doc in
    References. That is why clearing the cookie, not a retry, is the only recovery.)
    Clearing the cookie removes the dead token from the picture: no cookie →
    immediate 302 (no IdP I/O) → clean login.

Root cause (bridge-side)

internal/serve/oidc.go — the lazy refresh (ADR 0008) is the only OIDC Check
path that does IdP I/O, and it does so unbounded on the gateway's request context:

  • Check (line ~211): s.renew(ctx, app, refreshToken) — no deadline against the
    200ms budget.
  • renew (lines ~329–363): runs the refresh grant on the gateway's context, so a
    slow grant is canceled mid-flight and the bridge's response never lands in budget.

Because the grant is unbounded, the Check response is not guaranteed to be delivered
before the gateway's 200ms timer fires; when it isn't, Envoy returns its own 403 and
the browser is stuck (fact 2 above).

Fix

Bound the refresh grant to the gateway's budget and fail to a 302.

Run the refresh grant with its own deadline of 190ms — under the 200ms ext_authz
timeout. Colocation keeps the mini→portable network to ~5ms, leaving ~185ms for
Kanidm's server-side processing (measured 93–181ms), so a healthy refresh still
completes in budget. On deadline, treat it as a failed refresh → relogin (302).

The Check response is then always delivered in-budget: the client gets a proper 302
(or a renewed OK) instead of a gateway 403, and the stuck 4xx loop becomes a
re-login. The rotation trap is avoided structurally: the browser always receives
either the renewed OK (with the new token) or a 302 (re-login) — it never retries
with the stale, now-dead token, so Kanidm's reuse detection is never triggered.

Consider making the deadline configurable per-App (e.g. on OidcSession), defaulting
to 190ms.

Out of scope

  • Caching the renewal (per-replica token cache keyed by the old refresh token).
    Considered and dropped. It would recover the session transparently on the next
    request, but:

    • (security) it inverts the at-most-once guarantee. An OIDC refresh token is
      rotated and dead after one use. A cache mapping the old (dead) refresh token →
      the new (live) token would serve a live token to a stolen stale refresh token
      on a cache hit, bypassing Kanidm's reuse detection (which cancels the whole
      session). Without the cache, that same stolen dead token is rejected by Kanidm.
      This is a security weakening, not just added complexity. (Contrast the Exchange
      flow's tokenCache, which is safe because its key — the service-account API
      token — is a live, non-rotated credential, not a dead one.)
    • (redundancy) it is largely redundant with the 302 — the browser re-logs in
      rather than retrying the stale cookie, so the cache rarely hits;
    • (coherence) it has a multi-replica coherence gap — a retry landing on a
      different replica misses the cache and re-presents the dead token, triggering
      Kanidm's session-wide reuse cancellation;
    • (complexity) cross-replica cache management is a disproportionate complication
      for this fix.

    Revisit only if the occasional re-login on a slow refresh proves to be a UX problem.

  • Connection warming / re-warming — tried before; caused more issues than it
    solved. Not part of this fix.

  • The gateway-side timeout — #53 (cilium/cilium#48354) remains the upstream fix;
    this issue makes the bridge correct under the 200ms constraint.

Verification

  • Unit tests: a refresh that completes within the deadline → OK + re-issued Session
    cookie (no 403); a refresh that sleeps past the deadline → 302 (token-endpoint
    double that sleeps past the deadline).
  • Live: stale Session cookie against admin.media.animeteamspeak.moe (X-Request-Id
    recipe in docs/agents/deploy.md); expect a 302 (or a renewed OK) instead of a
    4xx loop; bridge logs show the new deadline line.

References

## Problem After an OIDC Session has expired, a browser holding the (stale) Session cookie is greeted by repeated **4xx** (gateway fail-closed 403, empty body, `server: envoy`) on the protected endpoint instead of the 302 re-login. Refreshing does not help; the only recovery observed was **clearing the domain cookies** and re-authorizing. Reported live: `admin.media.animeteamspeak.moe` (App `media/privatemedia`, bridge on the `mini` cluster, Kanidm on `portable`), 2026-09-20 ~16:00–16:14 UTC. The same pattern occurred at 21:55 (Sep 19) and 06:16 (Sep 20) — recurring, not a one-off. ## Evidence (correlated bridge + IdP logs) Bridge (mini, `authz-bridge-system`, UTC): | Time | Line | |---|---| | 16:00:00 | `Session id_token failed validation` — `oidc: token is expired (Token Expiry: 2026-09-20 06:31:31 UTC)` | | 16:00:01 | `The Session refresh failed` — `Post "https://auth.animeteamspeak.moe/oauth2/token": context canceled` | | 16:00:03 | `The Session refresh failed` — `context canceled` | | 16:00:04 | `The Session refresh failed` — `oauth2: "invalid_grant"` | | 16:14:16 | `Renewed the Session` | | 16:14:18–24 | `The Session refresh failed` ×5 — `oauth2: "invalid_grant"` | Kanidm (portable, `kanidm/kanidm-0`; note `kubectl --timestamps` prints EDT, −04:00): | Time (UTC) | POST /oauth2/token | |---|---| | 16:00:01.151 | **200, 181ms** (the refresh *succeeded* server-side) | | 16:00:03.333 | 400, 201ms | | 16:00:04.688 | 400, 61ms | | 16:14:16.779 | **200, 93ms** | | 16:14:18.9–16:14:24.4 | **400 ×5** (32–68ms) — the same already-rotated refresh token | Two facts fall out of the correlation: 1. **The 4xx is the gateway's fail-closed 403, not the bridge's.** The refresh round-trip (mini → portable gateway → kanidm nginx sidecar → kanidm; Kanidm alone 181–201ms) exceeds the gateway's hardcoded **200ms ext_authz gRPC timeout**; Envoy cancels the gRPC call (`context canceled`) and returns its own 403 (`failure_mode_allow=false`). The bridge's 302 never reaches the browser. (Same root cause as #53: the GRPC branch of Cilium's GEP-1494 translation never sets `Timeout` — cilium/cilium#48354, no API knob.) 2. **Kanidm's refresh token is at-most-once.** A refresh that *succeeds server-side* while the bridge's context is canceled at ~200ms loses the OK + re-issued `Set-Cookie`; the browser keeps presenting the now-dead refresh token, so every subsequent request is a refresh → 400 (`invalid_grant`) or a 403 when the round-trip is slow → the stuck 4xx loop. (Kanidm goes further than rotation: on refresh-token **reuse** it cancels the whole session — see the design doc in References. That is why clearing the cookie, not a retry, is the only recovery.) Clearing the cookie removes the dead token from the picture: no cookie → immediate 302 (no IdP I/O) → clean login. ## Root cause (bridge-side) `internal/serve/oidc.go` — the lazy refresh (ADR 0008) is the **only** OIDC Check path that does IdP I/O, and it does so unbounded on the gateway's request context: - `Check` (line ~211): `s.renew(ctx, app, refreshToken)` — no deadline against the 200ms budget. - `renew` (lines ~329–363): runs the refresh grant on the gateway's context, so a slow grant is canceled mid-flight and the bridge's response never lands in budget. Because the grant is unbounded, the Check response is not guaranteed to be delivered before the gateway's 200ms timer fires; when it isn't, Envoy returns its own 403 and the browser is stuck (fact 2 above). ## Fix **Bound the refresh grant to the gateway's budget and fail to a 302.** Run the refresh grant with its own deadline of **190ms** — under the 200ms ext_authz timeout. Colocation keeps the mini→portable network to ~5ms, leaving ~185ms for Kanidm's server-side processing (measured 93–181ms), so a healthy refresh still completes in budget. On deadline, treat it as a failed refresh → `relogin` (302). The Check response is then always delivered in-budget: the client gets a proper 302 (or a renewed OK) instead of a gateway 403, and the stuck 4xx loop becomes a re-login. The rotation trap is avoided structurally: the browser always receives either the renewed OK (with the new token) or a 302 (re-login) — it never retries with the stale, now-dead token, so Kanidm's reuse detection is never triggered. Consider making the deadline configurable per-App (e.g. on `OidcSession`), defaulting to 190ms. ## Out of scope - **Caching the renewal (per-replica token cache keyed by the old refresh token).** Considered and dropped. It would recover the session transparently on the next request, but: - **(security) it inverts the at-most-once guarantee.** An OIDC refresh token is rotated and *dead* after one use. A cache mapping the old (dead) refresh token → the new (live) token would serve a live token to a **stolen stale refresh token** on a cache hit, **bypassing Kanidm's reuse detection** (which cancels the whole session). Without the cache, that same stolen dead token is rejected by Kanidm. This is a security weakening, not just added complexity. (Contrast the Exchange flow's `tokenCache`, which is safe because its key — the service-account API token — is a *live, non-rotated* credential, not a dead one.) - **(redundancy)** it is largely redundant with the 302 — the browser re-logs in rather than retrying the stale cookie, so the cache rarely hits; - **(coherence)** it has a multi-replica coherence gap — a retry landing on a different replica misses the cache and re-presents the dead token, triggering Kanidm's session-wide reuse cancellation; - **(complexity)** cross-replica cache management is a disproportionate complication for this fix. Revisit only if the occasional re-login on a slow refresh proves to be a UX problem. - **Connection warming / re-warming** — tried before; caused more issues than it solved. Not part of this fix. - **The gateway-side timeout** — #53 (cilium/cilium#48354) remains the upstream fix; this issue makes the bridge correct under the 200ms constraint. ## Verification - Unit tests: a refresh that completes within the deadline → OK + re-issued Session cookie (no 403); a refresh that sleeps past the deadline → 302 (token-endpoint double that sleeps past the deadline). - Live: stale Session cookie against `admin.media.animeteamspeak.moe` (X-Request-Id recipe in `docs/agents/deploy.md`); expect a 302 (or a renewed OK) instead of a 4xx loop; bridge logs show the new deadline line. ## References - ADR 0008 (lazy refresh on the Check path), ADR 0011 (fail-closed status model) - #53 (same 200ms root cause, Exchange flow), cilium/cilium#48354 - Kanidm refresh-token design (at-most-once + reuse detection cancels the session): https://kanidm.github.io/kanidm/stable/developers/designs/oauth2_refresh_tokens.html
ginjiruu changed title from OIDC expired-session 4xx loop: bound the Check-path refresh to the 200ms ext_authz budget and cache the renewal to OIDC expired-session 4xx loop: bound the Check-path refresh to the 200ms ext_authz budget and fail to a 302 2026-09-20 18:48:34 +00:00
ginjiruu commented 2026-09-20 19:46:38 +00:00
Author
Owner
Copy link

Implemented and deployed

Fix (authz-bridge 4f0316d): the refresh grant on the Check path now runs under its own deadline — OidcSession.RefreshDeadline, default 190ms, under the 200ms ext_authz budget — and a deadline is treated as a failed refresh → relogin (302). The value is deliberately not clamped so it can track the gateway's timeout if it changes (hazard documented in the API field comment + ADR 0008 addendum).

Image: harbor.animeteamspeak.moe/library/authz-bridge:v0.1.0-4-g01eb540@sha256:509dd435… (clean tag, built from the committed tree).

Deployed to both clusters:

  • portable — skaffold, 3/3 replicas rolled out.
  • mini — via flux-clusters c452f0f1 (image pin + the refreshDeadline CRD field), rollout confirmed; admin.media.animeteamspeak.moe baseline (no session → 302) verified against the new bridge.

Verification

Unit tests (internal/serve/oidc_refresh_test.go, token-endpoint double with a delay param):

  • grant sleeps 300ms vs the 190ms default → 302 (grant attempted once);
  • grant sleeps 100ms vs a configured 50ms deadline → 302 (per-App value bounds the grant);
  • grant sleeps 300ms vs a configured 2s deadline → OK + re-issued Session cookie (within-deadline refresh).

Live (portable, httpbin-auth): forged a stale Session cookie (random-key id_token with the App's real issuer/audience but expired exp, dummy refresh token, encoded with the bridge's real cookie key) and sent it through the gateway:

  • client received the bridge's 302 re-login in-budget — no gateway 403, no 4xx loop;
  • bridge logs show the full refresh path: Session id_token failed validation (signature) → The Session refresh failed (Kanidm rejected the dummy token) → 302;
  • the grant round-trip completed well under the deadline (both lines in the same second), so the deadline branch correctly did not fire on a healthy-speed grant.

Not exercised live (by construction, not by omission):

  • the deadline branch itself — needs a grant slower than 190ms; covered by the unit tests above;
  • the renewed-OK path with a real session — needs a genuine login and the id_token approaching expiry. The next natural occurrence will now log either Renewed the Session (OK + re-issued cookie) or the new The Session refresh exceeded its deadline line, instead of context canceled + a gateway 403.

The rotation trap is avoided structurally as designed: the browser always receives either the renewed OK (new token) or a 302 (re-login) — it never retries with the stale, now-dead refresh token, so Kanidm's reuse detection is never triggered.

## Implemented and deployed **Fix** (authz-bridge `4f0316d`): the refresh grant on the Check path now runs under its own deadline — `OidcSession.RefreshDeadline`, default **190ms**, under the 200ms ext_authz budget — and a deadline is treated as a failed refresh → `relogin` (302). The value is deliberately not clamped so it can track the gateway's timeout if it changes (hazard documented in the API field comment + ADR 0008 addendum). **Image**: `harbor.animeteamspeak.moe/library/authz-bridge:v0.1.0-4-g01eb540@sha256:509dd435…` (clean tag, built from the committed tree). **Deployed to both clusters**: - `portable` — skaffold, 3/3 replicas rolled out. - `mini` — via flux-clusters `c452f0f1` (image pin + the `refreshDeadline` CRD field), rollout confirmed; `admin.media.animeteamspeak.moe` baseline (no session → 302) verified against the new bridge. ## Verification **Unit tests** (`internal/serve/oidc_refresh_test.go`, token-endpoint double with a delay param): - grant sleeps 300ms vs the 190ms default → **302** (grant attempted once); - grant sleeps 100ms vs a configured 50ms deadline → **302** (per-App value bounds the grant); - grant sleeps 300ms vs a configured 2s deadline → **OK + re-issued Session cookie** (within-deadline refresh). **Live (portable, `httpbin-auth`)**: forged a stale Session cookie (random-key `id_token` with the App's real issuer/audience but expired `exp`, dummy refresh token, encoded with the bridge's real cookie key) and sent it through the gateway: - client received the bridge's **302 re-login in-budget — no gateway 403, no 4xx loop**; - bridge logs show the full refresh path: `Session id_token failed validation` (signature) → `The Session refresh failed` (Kanidm rejected the dummy token) → 302; - the grant round-trip completed well under the deadline (both lines in the same second), so the deadline branch correctly did not fire on a healthy-speed grant. **Not exercised live** (by construction, not by omission): - the deadline branch itself — needs a grant slower than 190ms; covered by the unit tests above; - the renewed-OK path with a *real* session — needs a genuine login and the `id_token` approaching expiry. The next natural occurrence will now log either `Renewed the Session` (OK + re-issued cookie) or the new `The Session refresh exceeded its deadline` line, instead of `context canceled` + a gateway 403. The rotation trap is avoided structurally as designed: the browser always receives either the renewed OK (new token) or a 302 (re-login) — it never retries with the stale, now-dead refresh token, so Kanidm's reuse detection is never triggered.
ginjiruu commented 2026-09-21 14:45:33 +00:00
Author
Owner
Copy link

Fixed in 4f0316d (shipped in v0.1.2). The lazy refresh (ADR 0008) now runs the refresh_token grant under its own deadline - OidcSession.RefreshDeadline, default 190ms, under the gateway's 200ms ext_authz budget (refreshDeadlineOf, internal/serve/oidc.go). A deadline is treated as a failed refresh, so the Check path returns a 302 relogin delivered in-budget. The browser always receives either the renewed OK (with the re-issued cookie) or a 302, so it never retries with the now-dead (rotated) refresh token and Kanidm's at-most-once reuse detection is never triggered. Unit tests cover within-deadline -> OK + re-issued cookie and past-deadline -> 302. Out-of-scope items (caching the renewal, connection warming, the gateway-side timeout) remain as documented; the gateway-side timeout is tracked by #53 (cilium/cilium#48354).

Fixed in 4f0316d (shipped in v0.1.2). The lazy refresh (ADR 0008) now runs the refresh_token grant under its own deadline - OidcSession.RefreshDeadline, default 190ms, under the gateway's 200ms ext_authz budget (refreshDeadlineOf, internal/serve/oidc.go). A deadline is treated as a failed refresh, so the Check path returns a 302 relogin delivered in-budget. The browser always receives either the renewed OK (with the re-issued cookie) or a 302, so it never retries with the now-dead (rotated) refresh token and Kanidm's at-most-once reuse detection is never triggered. Unit tests cover within-deadline -> OK + re-issued cookie and past-deadline -> 302. Out-of-scope items (caching the renewal, connection warming, the gateway-side timeout) remain as documented; the gateway-side timeout is tracked by #53 (cilium/cilium#48354).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
ci/repo-declared-scaffold
fix/45-exchange-forward-id-token
fix/44-oidc-bearer-tokens
t36-pkce-nonce
t34-oidc-trio
t33-securecookie
t32-public-client
t37-e2e
t35-exchange-hstern
feat/oidc-logout
feat/oidc-callback
feat/oidc-lazy-refresh
feat/services-referencedgrant
feat/exchange-token-cache
feat/exchange-check
feat/oidc-steady-state-check
feat/end-session-endpoint
feat/build-surface
feat/replica-mirror
feat/leader-freshness
feat/registry-builder
renovate/configure
feat/idp-resolve
v0.1.2-ci-test
v0.1.1
v0.1.0
Labels
Clear labels   
needs-info

   
needs-triage

   
ready-for-agent

   
ready-for-human

   
wontfix
No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lab/authz-bridge#60
No description provided.