Cold-start denials: gate readiness on the Snapshot + retry the exchange on a dropped request #51

New issue

Closed

opened 2026-09-16 13:12:31 +00:00 by ginjiruu · 0 comments

ginjiruu commented

2026-09-16 13:12:31 +00:00

Owner

Problem

Two classes of failures observed live on the portable cluster:

Cold-start / readiness: a fresh replica receives traffic before it can
serve it. The default readyz was healthz.Ping (always ready), so during
the window before the per-replica Snapshot is populated (the mirror reading
the managed Registry Secret), requests fail with "unknown host" (403) or
"Cookie key is not ready" (503). The same shows up as "a new leader can't
serve requests" — the pod is ready (gets traffic) before its Snapshot is
built.
Cold token exchange: the first RFC 8693 exchange on a replica is slow
(TLS/connection setup to the IdP token endpoint). The gateway's ext_authz
timeout fires first, canceling the gRPC context; the bridge logs
The token exchange failed ... context canceled and the client gets a 403.
The aborted exchange does not reliably warm the per-replica token cache, so
the next request can be cold again.

Plan

Gate readyz on the Snapshot: the manager's readiness check returns
not-ready until the per-replica Snapshot holds at least one App and the
Cookie codec (Snapshot.Ready()). A replica then never receives traffic
before it can serve. The mirror is not leader-gated (it reads the managed
Registry Secret directly), so a fresh replica becomes ready from the existing
Registry without waiting to be elected leader (no leader/readiness
deadlock).
Retry the exchange on a dropped request: when the token exchange fails
because the gRPC request context was canceled (the gateway dropped it), retry
once with a fresh, non-cancelable context (context.WithoutCancel, values
preserved so the requestID stays on the logs). The result is cached, so the
next request is served fast instead of re-hitting the cold path.
Log the flow's V(1) lines by default: add --zap-log-level=2 to the
manager manifest so per-request Check/callback/exchange decisions are visible
without a temporary patch.

(Adapted from the original ginjiruu/authz-bridge edge-case handling — its
"gate readyz on the config store" and "retry with a fresh context when the
request context is canceled" patterns — without adopting its overall design.)

## Problem Two classes of failures observed live on the portable cluster: 1. **Cold-start / readiness**: a fresh replica receives traffic before it can serve it. The default `readyz` was `healthz.Ping` (always ready), so during the window before the per-replica Snapshot is populated (the mirror reading the managed Registry Secret), requests fail with "unknown host" (403) or "Cookie key is not ready" (503). The same shows up as "a new leader can't serve requests" — the pod is ready (gets traffic) before its Snapshot is built. 2. **Cold token exchange**: the first RFC 8693 exchange on a replica is slow (TLS/connection setup to the IdP token endpoint). The gateway's ext_authz timeout fires first, canceling the gRPC context; the bridge logs `The token exchange failed ... context canceled` and the client gets a 403. The aborted exchange does not reliably warm the per-replica token cache, so the next request can be cold again. ## Plan - **Gate `readyz` on the Snapshot**: the manager's readiness check returns not-ready until the per-replica Snapshot holds at least one App **and** the Cookie codec (`Snapshot.Ready()`). A replica then never receives traffic before it can serve. The mirror is not leader-gated (it reads the managed Registry Secret directly), so a fresh replica becomes ready from the existing Registry **without waiting to be elected leader** (no leader/readiness deadlock). - **Retry the exchange on a dropped request**: when the token exchange fails because the gRPC request context was canceled (the gateway dropped it), retry once with a fresh, non-cancelable context (`context.WithoutCancel`, values preserved so the `requestID` stays on the logs). The result is cached, so the next request is served fast instead of re-hitting the cold path. - **Log the flow's `V(1)` lines by default**: add `--zap-log-level=2` to the manager manifest so per-request Check/callback/exchange decisions are visible without a temporary patch. (Adapted from the original `ginjiruu/authz-bridge` edge-case handling — its "gate readyz on the config store" and "retry with a fresh context when the request context is canceled" patterns — without adopting its overall design.)

ginjiruu referenced this issue from a pull request that will close it,

2026-09-16 13:12:45 +00:00

fix: gate readiness on the Snapshot + retry the exchange on a dropped request #52

ginjiruu closed this issue

2026-09-16 13:13:10 +00:00

ginjiruu referenced this issue

2026-09-16 14:40:22 +00:00

First-request 403: cold exchange exceeds the gateway's 200ms ext_authz timeout (cilium#48354) #53