authz-bridge: readiness gate + leader-election coupling blocks HA scaling #7

New issue

Open

opened 2026-08-28 16:57:21 +00:00 by ginjiruu · 0 comments

ginjiruu commented

2026-08-28 16:57:21 +00:00

Owner

Background

Added a readiness gate so the bridge does not serve traffic before its config store is loaded (fixes cold-start 403 'unknown host' via /readyz returning 503 until apps are present). See internal/check readiness wiring in cmd/main.go and internal/configstore.

Temporary fix applied

Removed --leader-elect from config/manager/manager.yaml and the running deployments. This was required because:

The config store is populated by the AuthzApp controller, which runs ONLY in the leader.
The readiness gate (ready iff len(store.All()) > 0) therefore can only be satisfied by the leader.
During a rolling update the new pod cannot win the lease until the old leader releases it; the old pod is not scaled down until the new pod is Ready; the new pod never loads the store; never Ready. -> rollout exceeded its progress deadline (deadlock).

With a single replica this is fine and rolling upgrades are clean.

Scaling concerns to resolve later (do not ignore before scaling to >1 replica)

No leader election = no HA leader failover for the manager. Re-enabling leader election for >1 replica reintroduces the deadlock above unless the store sync is decoupled from leadership.
Readiness gate is app-count coarse: len(store.All()) > 0 passes as soon as ANY app loads. With multiple apps, a request for a not-yet-reconciled host can still 403 'unknown host' before that specific app is loaded. Consider gating on the controller's initial-sync-complete signal (per pod) instead of store contents.
If scaling to >1 replica with leader election, only the leader populates the store; non-leader pods would be perpetually not-ready under the current gate.

Recommended future direction

Decouple the store-sync from the leader role so every replica can populate its store and satisfy readiness independently (e.g. a per-pod controller 'initial sync done' flag), so the readiness gate works with leader election + HA. Revisit leader-elect policy for the intended replica count.

Status

Blocking only for >1 replica scaling. Current single-replica deployment is healthy; rolling upgrades verified clean (all requests redirect, no cold-start 403).

## Background Added a readiness gate so the bridge does not serve traffic before its config store is loaded (fixes cold-start 403 'unknown host' via /readyz returning 503 until apps are present). See internal/check readiness wiring in cmd/main.go and internal/configstore. ## Temporary fix applied Removed --leader-elect from config/manager/manager.yaml and the running deployments. This was required because: - The config store is populated by the AuthzApp controller, which runs ONLY in the leader. - The readiness gate (ready iff len(store.All()) > 0) therefore can only be satisfied by the leader. - During a rolling update the new pod cannot win the lease until the old leader releases it; the old pod is not scaled down until the new pod is Ready; the new pod never loads the store; never Ready. -> rollout exceeded its progress deadline (deadlock). With a single replica this is fine and rolling upgrades are clean. ## Scaling concerns to resolve later (do not ignore before scaling to >1 replica) 1. No leader election = no HA leader failover for the manager. Re-enabling leader election for >1 replica reintroduces the deadlock above unless the store sync is decoupled from leadership. 2. Readiness gate is app-count coarse: len(store.All()) > 0 passes as soon as ANY app loads. With multiple apps, a request for a not-yet-reconciled host can still 403 'unknown host' before that specific app is loaded. Consider gating on the controller's initial-sync-complete signal (per pod) instead of store contents. 3. If scaling to >1 replica with leader election, only the leader populates the store; non-leader pods would be perpetually not-ready under the current gate. ## Recommended future direction Decouple the store-sync from the leader role so every replica can populate its store and satisfy readiness independently (e.g. a per-pod controller 'initial sync done' flag), so the readiness gate works with leader election + HA. Revisit leader-elect policy for the intended replica count. ## Status Blocking only for >1 replica scaling. Current single-replica deployment is healthy; rolling upgrades verified clean (all requests redirect, no cold-start 403).

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ginjiruu/authz-bridge#7

No description provided.

Rows
Columns