First-request 403: cold exchange exceeds the gateway's 200ms ext_authz timeout (cilium#48354) #53

New issue

Open

opened 2026-09-16 14:40:22 +00:00 by ginjiruu · 0 comments

ginjiruu commented

2026-09-16 14:40:22 +00:00

Owner

Problem

The first RFC 8693 token exchange to a freshly-Ready replica is dropped: the
client gets a 403 (empty body, server: envoy) at ~230ms. This is the residual
cold-start denial that #51's readiness gate + retry reduced but did not
eliminate. The user's position is firm: zero dropped requests, not "fewer."

Repro (live, portable cluster): deploy a fresh pod, wait for Ready, then send the
first exchange request with the service-account token. It 403s; the second request
(cache hit) is 200 in ~80ms. Consistent across fresh pods (229/238/239ms).

Root cause (measured + source-confirmed)

The gateway's ext_authz call is hard-limited to 200ms, and a cold exchange is
~330ms. The 200ms is not configurable anywhere we control.

The 200ms is Envoy's own default. source/extensions/filters/http/ext_authz/ config.h has static constexpr uint64_t DefaultTimeout = 200; (ms), applied when
envoy.config.core.v3.GrpcService.timeout is unset. It is a per-request Envoy
async-client timer on the gRPC Check RPC — not a cluster/route/gRPC-protocol
deadline.
Cilium never sets it. operator/pkg/model/translation/envoy_http_connection_manager.go,
buildExtAuthzHTTPFilter(): the GRPC branch builds extauthzv3.ExtAuthz without
a Timeout, while the HTTP branch sets Timeout: 10s. The asymmetry is an
oversight, not a design choice.
No API knob. The Gateway API ExternalAuth filter (v1.6.1) has no timeout
field — not under externalAuth, .grpc, or .http. Rule-level
timeouts.request/backendRequest are upper bounds (15s default) and don't extend
the 200ms ext_authz timer.
On timeout Envoy returns 403 + empty body (failure_mode_allow defaults false,
status_on_error unset → Forbidden). That matches the observed response exactly.
~230ms = 200ms + ~30ms overhead.
The bridge is not producing the 403. Every Check path returns a nil gRPC error
(status carried in the CheckResponse); the bridge returns 503 on exchange failure,
403 only for unknown host. The empty-body server: envoy 403 is the gateway's
fail-closed, not a bridge denial.

Why the cold exchange is ~330ms (kanidm logs + source)

The dominant cost is inside kanidm, not the connection. From kanidm's structured
timing logs:

check_oauth2_token_exchange_service_account:  112ms (cold)  vs  7ms (warm)

The cold cost is kanidm's DB read of the service account (internal_search_uuid
on the SAT's account_id) — a cold page/block-cache read that cools back down over
time (re-cooled within ~35min between measurements). Token issuance adds a constant
~43–87ms. So cold ≈ 200ms server-side + ~30ms network ≈ 230ms; warm ≈ 50ms
server-side ≈ 100ms total.

Why the existing fixes don't cover this

#51 readiness gate — correct and necessary (no traffic before the Snapshot is
servable), but the pod becomes Ready before the IdP's server-side state is warm.
Readiness can guarantee config is loaded; it cannot guarantee the IdP is fast.
TLS warm pre-fetch (this session's Warmer) — removes the ~27ms handshake, but
the ~300ms is IdP server-side processing, which no bridge-side connection warming
can touch.
#51 WithoutCancel retry — warms the per-replica cache (and the IdP state) for
the next request, but completes into a dead stream: the gateway already sent the
403 at 200ms. No in-bridge retry can save the in-flight request.

Workarounds that do NOT work

Patching the generated CiliumEnvoyConfig to add a timeout: the operator
reconciles the Gateway-owned CEC and reverts manual edits in <1s (controller owner
reference). Confirmed both in source and in the upstream issue below.
A throwaway warm-up exchange: an invalid SAT fails JWT signature verification
before the service-account DB read, so it never warms the account's block — and the
bridge doesn't hold a valid SAT to pre-exchange anyway.

Upstream

cilium/cilium#48354 (open CFP,
no assignee/milestone/PR) describes this exact bug for the OIDC code-exchange case.
Our case is RFC 8693 token exchange against Kanidm — a second independent data
point for the same "synchronous outbound IdP call inside Check()" GEP-1494 use case.
The issue's proposed minimal fix is to set GrpcService.Timeout (10s, matching HTTP
mode) in the GRPC branch of buildExtAuthzHTTPFilter().

Fix options (open — needs a decision)

A. Minimal Cilium PR — set GrpcService.Timeout: 10s in the GRPC branch
(matches HTTP mode). Unblocks us after a Cilium upgrade; small and easy to review.
The "configurable" follow-up (Gateway API field or Cilium annotation) is larger.
B. Redirect slow path (bridge-side, no Cilium change) — on a cache miss, Check
returns OK+302 (within 200ms) to a bridge HTTP exchange endpoint; the slow exchange
happens there (15s route timeout, no ext_authz limit); the client follows the
redirect and gets the resource. Zero 403s. Mirrors the OIDC flow's existing redirect
pattern. Complexity: multi-replica cache coherence (a retry could land on a different
cold replica) — needs a shared cache or a bounded redirect.
C. Make kanidm faster (IdP-side) — tune kanidm storage so the service-account
read is always warm (~7ms), making even a cold exchange ~100ms total. Outside this
repo.
D. Client-side retry — the client retries on 403; the second attempt hits the
warm cache. Rejected: the client still sees a 403, which is unacceptable.

Current state

The 4-gate readiness (Bootstrapped + Servable + listeners bound + IdP connections
warmed) and the TLS Warmer are implemented and live on portable (working tree,
uncommitted). The first cold exchange still 403s for the reasons above

## Problem The **first** RFC 8693 token exchange to a freshly-Ready replica is dropped: the client gets a **403** (empty body, `server: envoy`) at ~230ms. This is the residual cold-start denial that #51's readiness gate + retry reduced but did **not** eliminate. The user's position is firm: **zero dropped requests**, not "fewer." Repro (live, `portable` cluster): deploy a fresh pod, wait for Ready, then send the first exchange request with the service-account token. It 403s; the second request (cache hit) is 200 in ~80ms. Consistent across fresh pods (229/238/239ms). ## Root cause (measured + source-confirmed) The gateway's ext_authz call is hard-limited to **200ms**, and a cold exchange is **~330ms**. The 200ms is not configurable anywhere we control. - **The 200ms is Envoy's own default.** `source/extensions/filters/http/ext_authz/ config.h` has `static constexpr uint64_t DefaultTimeout = 200;` (ms), applied when `envoy.config.core.v3.GrpcService.timeout` is unset. It is a per-request Envoy async-client timer on the gRPC `Check` RPC — not a cluster/route/gRPC-protocol deadline. - **Cilium never sets it.** `operator/pkg/model/translation/envoy_http_connection_manager.go`, `buildExtAuthzHTTPFilter()`: the **GRPC** branch builds `extauthzv3.ExtAuthz` without a `Timeout`, while the **HTTP** branch sets `Timeout: 10s`. The asymmetry is an oversight, not a design choice. - **No API knob.** The Gateway API `ExternalAuth` filter (v1.6.1) has **no `timeout` field** — not under `externalAuth`, `.grpc`, or `.http`. Rule-level `timeouts.request`/`backendRequest` are upper bounds (15s default) and don't extend the 200ms ext_authz timer. - **On timeout** Envoy returns 403 + empty body (`failure_mode_allow` defaults false, `status_on_error` unset → `Forbidden`). That matches the observed response exactly. ~230ms = 200ms + ~30ms overhead. - **The bridge is not producing the 403.** Every Check path returns a nil gRPC error (status carried in the `CheckResponse`); the bridge returns 503 on exchange failure, 403 only for unknown host. The empty-body `server: envoy` 403 is the gateway's fail-closed, not a bridge denial. ### Why the cold exchange is ~330ms (kanidm logs + source) The dominant cost is **inside kanidm**, not the connection. From kanidm's structured timing logs: ``` check_oauth2_token_exchange_service_account: 112ms (cold) vs 7ms (warm) ``` The cold cost is kanidm's **DB read of the service account** (`internal_search_uuid` on the SAT's `account_id`) — a cold page/block-cache read that cools back down over time (re-cooled within ~35min between measurements). Token issuance adds a constant ~43–87ms. So cold ≈ 200ms server-side + ~30ms network ≈ 230ms; warm ≈ 50ms server-side ≈ 100ms total. ## Why the existing fixes don't cover this - **#51 readiness gate** — correct and necessary (no traffic before the Snapshot is servable), but the pod becomes Ready before the **IdP's server-side state** is warm. Readiness can guarantee config is loaded; it cannot guarantee the IdP is fast. - **TLS warm pre-fetch** (this session's `Warmer`) — removes the ~27ms handshake, but the ~300ms is IdP server-side processing, which no bridge-side connection warming can touch. - **#51 `WithoutCancel` retry** — warms the per-replica cache (and the IdP state) for the *next* request, but completes *into a dead stream*: the gateway already sent the 403 at 200ms. **No in-bridge retry can save the in-flight request.** ## Workarounds that do NOT work - **Patching the generated `CiliumEnvoyConfig`** to add a `timeout`: the operator reconciles the Gateway-owned CEC and reverts manual edits in <1s (controller owner reference). Confirmed both in source and in the upstream issue below. - **A throwaway warm-up exchange**: an invalid SAT fails JWT signature verification *before* the service-account DB read, so it never warms the account's block — and the bridge doesn't hold a valid SAT to pre-exchange anyway. ## Upstream **[cilium/cilium#48354](https://github.com/cilium/cilium/issues/48354)** (open CFP, no assignee/milestone/PR) describes this exact bug for the OIDC *code*-exchange case. Our case is **RFC 8693 token exchange** against Kanidm — a second independent data point for the same "synchronous outbound IdP call inside `Check()`" GEP-1494 use case. The issue's proposed minimal fix is to set `GrpcService.Timeout` (10s, matching HTTP mode) in the GRPC branch of `buildExtAuthzHTTPFilter()`. ## Fix options (open — needs a decision) - **A. Minimal Cilium PR** — set `GrpcService.Timeout: 10s` in the GRPC branch (matches HTTP mode). Unblocks us after a Cilium upgrade; small and easy to review. The "configurable" follow-up (Gateway API field or Cilium annotation) is larger. - **B. Redirect slow path (bridge-side, no Cilium change)** — on a cache miss, Check returns OK+302 (within 200ms) to a bridge HTTP exchange endpoint; the slow exchange happens there (15s route timeout, no ext_authz limit); the client follows the redirect and gets the resource. Zero 403s. Mirrors the OIDC flow's existing redirect pattern. Complexity: multi-replica cache coherence (a retry could land on a different cold replica) — needs a shared cache or a bounded redirect. - **C. Make kanidm faster (IdP-side)** — tune kanidm storage so the service-account read is always warm (~7ms), making even a cold exchange ~100ms total. Outside this repo. - **D. Client-side retry** — the client retries on 403; the second attempt hits the warm cache. Rejected: the client still sees a 403, which is unacceptable. ## Current state The 4-gate readiness (`Bootstrapped` + `Servable` + listeners bound + IdP connections warmed) and the TLS `Warmer` are implemented and live on `portable` (working tree, uncommitted). The first cold exchange still 403s for the reasons above

ginjiruu changed title from ~~cilium grpc timeout~~ to First-request 403: cold exchange exceeds the gateway's 200ms ext_authz timeout (cilium#48354)

2026-09-16 14:46:06 +00:00

ginjiruu referenced this issue

2026-09-20 18:34:23 +00:00

OIDC expired-session 4xx loop: bound the Check-path refresh to the 200ms ext_authz budget and fail to a 302 #60

ginjiruu referenced this issue

2026-09-20 19:05:09 +00:00

Exchange tokenCache hardening: digest key, single-flight, subject-exp-capped TTL, zeroized value #61

ginjiruu referenced this issue

2026-09-21 14:45:33 +00:00

OIDC expired-session 4xx loop: bound the Check-path refresh to the 200ms ext_authz budget and fail to a 302 #60