lab/authz-bridge
1
0
Fork 0
Code Issues 9 Pull requests 6 Projects Releases Packages Wiki Activity Actions 12

fix: gate readiness on the Snapshot + retry the exchange on a dropped request #52

Open
ginjiruu wants to merge 0 commits from feat/readiness-gate-and-cold-start into master
pull from: feat/readiness-gate-and-cold-start
merge into: lab:master
Conversation 0 Commits - Files changed -
ginjiruu commented 2026-09-16 13:12:45 +00:00
Owner
Copy link

Closes #51

What

  • Gate readyz on the Snapshot (cmd/main.go, internal/snapshot/snapshot.go): the manager's readiness check now returns not-ready until the per-replica Snapshot holds at least one App and the Cookie codec (Snapshot.Ready()). A replica never receives traffic before it can serve — no cold-start "unknown host" (403) / "Cookie key is not ready" (503) denials. The mirror is not leader-gated (it reads the managed Registry Secret directly), so a fresh replica becomes ready from the existing Registry without waiting to be elected leader.
  • Retry the exchange on a dropped request (internal/serve/exchange.go): when the token exchange fails because the gRPC request context was canceled (the gateway's ext_authz timeout on a slow cold exchange), retry once with a fresh, non-cancelable context (context.WithoutCancel, values preserved so the requestID stays on the logs). The result is cached, so the next request is fast instead of re-hitting the cold path.
  • Log the flow's V(1) lines by default (config/manager/manager.yaml): --zap-log-level=2 in the manifest, so per-request decisions are visible without a temporary patch.

Tests

  • TestSnapshotReady: Ready() requires both an App set and the Cookie codec (all four combinations).
  • TestExchangeRetriesWithFreshContextWhenCanceled: a canceled request context still yields an OK response (the fresh-context retry serves it and reaches the token endpoint).

Live verification (portable cluster)

  • Deployed; the pod became ready at 13:10:57 (Snapshot populated, apps:2, from the existing Registry) — 25s before it became leader (13:11:22). Confirms readiness no longer waits on leader election.
  • V(1) flow lines are visible by default (no patch): {"msg":"Bearer token failed validation","requestID":"defaultlevel-…",…}.
  • Cold exchange: attempts 1-2 were cold (gateway canceled → context canceled), the bridge logged Retrying the token exchange with a fresh context and warmed the cache; attempt 3 was a fast Served the Exchange Check from the token cache (200).
Closes #51 ## What - **Gate `readyz` on the Snapshot** (`cmd/main.go`, `internal/snapshot/snapshot.go`): the manager's readiness check now returns not-ready until the per-replica Snapshot holds at least one App **and** the Cookie codec (`Snapshot.Ready()`). A replica never receives traffic before it can serve — no cold-start "unknown host" (403) / "Cookie key is not ready" (503) denials. The mirror is not leader-gated (it reads the managed Registry Secret directly), so a fresh replica becomes ready from the existing Registry **without waiting to be elected leader**. - **Retry the exchange on a dropped request** (`internal/serve/exchange.go`): when the token exchange fails because the gRPC request context was canceled (the gateway's ext_authz timeout on a slow cold exchange), retry once with a fresh, non-cancelable context (`context.WithoutCancel`, values preserved so the `requestID` stays on the logs). The result is cached, so the next request is fast instead of re-hitting the cold path. - **Log the flow's `V(1)` lines by default** (`config/manager/manager.yaml`): `--zap-log-level=2` in the manifest, so per-request decisions are visible without a temporary patch. ## Tests - `TestSnapshotReady`: `Ready()` requires both an App set and the Cookie codec (all four combinations). - `TestExchangeRetriesWithFreshContextWhenCanceled`: a canceled request context still yields an OK response (the fresh-context retry serves it and reaches the token endpoint). ## Live verification (portable cluster) - Deployed; the pod became ready at **13:10:57** (Snapshot populated, apps:2, from the existing Registry) — **25s before** it became leader (13:11:22). Confirms readiness no longer waits on leader election. - V(1) flow lines are visible by default (no patch): `{"msg":"Bearer token failed validation","requestID":"defaultlevel-…",…}`. - Cold exchange: attempts 1-2 were cold (gateway canceled → `context canceled`), the bridge logged `Retrying the token exchange with a fresh context` and warmed the cache; attempt 3 was a fast `Served the Exchange Check from the token cache` (200).
fix: gate readiness on the Snapshot and retry the exchange on a dropped request
Some checks failed
Lint / Run on Ubuntu (pull_request) Has been cancelled
Details
E2E Tests / Run on Ubuntu (pull_request) Has been cancelled
Details
Tests / Run on Ubuntu (pull_request) Has been cancelled
Details
a903f37ac5 
Gate the manager's readyz on the per-replica Snapshot being populated
(at least one App and the Cookie codec), so a replica never receives
traffic before it can serve it — no cold-start 'unknown host' / 'Cookie
key not ready' denials. The mirror is not leader-gated (it reads the
managed Registry Secret directly), so a fresh replica becomes ready from
the existing Registry without waiting to be elected leader.

Retry the RFC 8693 token exchange once with a fresh, non-cancelable
context when the gRPC request context is canceled (the gateway's ext_authz
timeout on a slow cold exchange), so a dropped-but-servable request still
completes and warms the per-replica token cache.

Also log the flow-serving V(1) lines by default (--zap-log-level=2 in the
manager manifest).
Some checks failed
Lint / Run on Ubuntu (pull_request) Has been cancelled
Details
E2E Tests / Run on Ubuntu (pull_request) Has been cancelled
Details
Tests / Run on Ubuntu (pull_request) Has been cancelled
Details
This pull request is broken due to missing fork information.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/readiness-gate-and-cold-start:feat/readiness-gate-and-cold-start
git switch feat/readiness-gate-and-cold-start

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff feat/readiness-gate-and-cold-start
git switch feat/readiness-gate-and-cold-start
git rebase master
git switch master
git merge --ff-only feat/readiness-gate-and-cold-start
git switch feat/readiness-gate-and-cold-start
git rebase master
git switch master
git merge --no-ff feat/readiness-gate-and-cold-start
git switch master
git merge --squash feat/readiness-gate-and-cold-start
git switch master
git merge --ff-only feat/readiness-gate-and-cold-start
git switch master
git merge feat/readiness-gate-and-cold-start
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
needs-info

   
needs-triage

   
ready-for-agent

   
ready-for-human

   
wontfix
No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lab/authz-bridge!52
No description provided.