lab/iac
1
0
Fork 0
Code Issues 11 Pull requests 3 Projects Releases Packages Wiki Activity Actions

T13: kubeconfig distribution & connectivity architecture (kubectl access) #28

New issue
Open
opened 2026-08-12 02:28:41 +00:00 by ginjiruu · 1 comment
ginjiruu commented 2026-08-12 02:28:41 +00:00
Owner
Copy link

T13: kubeconfig distribution & connectivity architecture (kubectl access)

Golden Path intersection (#38): for machine/agent access, the connectivity model below is realized concretely by the k8s-hub — a TLS-terminating Gateway API (Cilium) proxy in front of each apiserver that performs the RFC 8693 token exchange per request via the authz-bridge (kubeconfigs carry only a static Kanidm SA API token; the route terminates with a Let's Encrypt cert, public trust). Human kubectl access is unchanged (exec credential plugin / kubelogin). See #38 (decided) — #39 (conformance blocker) is resolved. (2026-09: the gateway migrated from agentgateway to the Cilium Gateway API k8s-hub — the mechanism is unchanged.)

Context

Users need kubectl access to clusters. Two problems: how users learn which clusters exist (discovery), and how bytes reach the apiserver (connectivity). This ticket records the chosen architecture — the design is still being hashed out, so it will be refined as implementation questions get answered.

Architecture

Connectivity

  • Gateway: apiserver behind a TLS-terminating gateway; users reach https://<cluster>.k8s.animeteamspeak.moe over normal HTTPS. No VPN — external runners can't rely on kernel modules (WireGuard).
    • Machine/agent traffic (Golden Path, #38): the gateway is the k8s-hub (Cilium Gateway API, gateway/cilium-default on portable) with the authz-bridge ExternalAuth flow — it terminates TLS (Let's Encrypt) and exchanges the bearer's Kanidm SA API token per request (audience = the cluster's apiserver client, e.g. minicluster; scopes openid groups_name; the exchanged id_token is forwarded), so the apiserver sees a fresh token and the kubeconfig holds no short-lived credential. This replaces the earlier "TLS passthrough / HTTPRoute" sketch for the machine path.
    • Human traffic: a plain TLS-terminating route (the k8s-hub base route is pure passthrough) — kubelogin exec does its own exchange client-side.
  • Firewalled clusters: a reverse tunnel (frp or rathole) — a small outbound-only client inside the firewalled network dials the relay on mini; kubectl traffic passes through as a byte pipe. No client-side kernel modules; only our own network dials out. DNS (<cluster>.k8s.animeteamspeak.moe) points at the relay/gateway.

AuthN/Z

  • Native apiserver OIDC against kanidm (the "k8s apiserver issue" — the k8s apiserver OIDC flags are tracked in flux-clusters #11 / iac #11).
  • Humans: kubectl uses an exec credential plugin (kubelogin / kanidm-login.py-style) — kubeconfigs carry no static credentials; the token is minted at runtime from kanidm.
  • Machines/agents (Golden Path, #38): kubeconfigs carry the actor's Kanidm SA API token (the single long-lived per-actor credential) + the k8s-hub server URL (Let's Encrypt cert, public trust); the authz-bridge exchanges it per request. Identity stays per-actor: apiserver sees the Kanidm principal, RBAC is driven by the capability groups (group-<app>-<capability>). Machines = humans: kanidm service accounts + RFC 8693 token exchange.

Discovery / distribution

  • One kubeconfig per cluster (exec block + server URL only, no static credentials) for humans — safe in git.
  • Distribution via Vault KV (ACL-gated) + kubeswitch pointed at it (discovery + fetch in one switch command). Portal API is the endgame, not now.

Where this lives

This spans repos — most belongs in its own repo (kubeconfig distribution + tunnel orchestration), while the apiserver flags side is flux-clusters #11. The k8s-hub data plane lives on portable (managed via flux-clusters generic-templates/k8s-hub) tracked via #38 (#39 resolved). (Originally agentgateway on mini; migrated to the Cilium Gateway API in 2026-09.)

Open questions (to answer while implementing)

  • agentgateway placement/TLS certs for the apiserver routes (depends on #39 unblocking) — #39 resolved (2026-08-12): the exchange is verified; placement/certs is now implementation work (Gate-2 in #38).
  • Whether human and machine traffic share one gateway listener or get separate routes.
## T13: kubeconfig distribution & connectivity architecture (kubectl access) > **Golden Path intersection (#38):** for **machine/agent access**, the connectivity model below is realized concretely by the **k8s-hub** — a TLS-terminating Gateway API (Cilium) proxy in front of each apiserver that performs the RFC 8693 token exchange per request via the authz-bridge (kubeconfigs carry only a static Kanidm SA API token; the route terminates with a Let's Encrypt cert, public trust). **Human kubectl access is unchanged** (exec credential plugin / kubelogin). See #38 (decided) — #39 (conformance blocker) is **resolved**. *(2026-09: the gateway migrated from agentgateway to the Cilium Gateway API k8s-hub — the mechanism is unchanged.)* ## Context Users need kubectl access to clusters. Two problems: how users learn which clusters exist (discovery), and how bytes reach the apiserver (connectivity). This ticket records the chosen architecture — the design is still being hashed out, so it will be refined as implementation questions get answered. ## Architecture ### Connectivity - **Gateway**: apiserver behind a TLS-terminating gateway; users reach `https://<cluster>.k8s.animeteamspeak.moe` over normal HTTPS. No VPN — external runners can't rely on kernel modules (WireGuard). - **Machine/agent traffic (Golden Path, #38):** the gateway is the **k8s-hub** (Cilium Gateway API, `gateway/cilium-default` on portable) with the **authz-bridge** `ExternalAuth` flow — it terminates TLS (Let's Encrypt) and exchanges the bearer's Kanidm SA API token per request (audience = the cluster's apiserver client, e.g. `minicluster`; scopes `openid groups_name`; the exchanged id_token is forwarded), so the apiserver sees a fresh token and the kubeconfig holds no short-lived credential. This replaces the earlier "TLS passthrough / HTTPRoute" sketch for the machine path. - **Human traffic:** a plain TLS-terminating route (the k8s-hub base route is pure passthrough) — kubelogin exec does its own exchange client-side. - **Firewalled clusters**: a reverse tunnel (frp or rathole) — a small outbound-only client inside the firewalled network dials the relay on mini; kubectl traffic passes through as a byte pipe. No client-side kernel modules; only our own network dials out. DNS (`<cluster>.k8s.animeteamspeak.moe`) points at the relay/gateway. ### AuthN/Z - **Native apiserver OIDC against kanidm** (the "k8s apiserver issue" — the k8s apiserver OIDC flags are tracked in flux-clusters #11 / iac #11). - **Humans**: kubectl uses an **exec credential plugin** (kubelogin / `kanidm-login.py`-style) — kubeconfigs carry **no static credentials**; the token is minted at runtime from kanidm. - **Machines/agents (Golden Path, #38):** kubeconfigs carry the actor's **Kanidm SA API token** (the single long-lived per-actor credential) + the k8s-hub server URL (Let's Encrypt cert, public trust); the authz-bridge exchanges it per request. Identity stays per-actor: apiserver sees the Kanidm principal, RBAC is driven by the capability groups (`group-<app>-<capability>`). Machines = humans: kanidm service accounts + RFC 8693 token exchange. ### Discovery / distribution - One kubeconfig per cluster (exec block + server URL only, no static credentials) for humans — safe in git. - Distribution via **Vault KV** (ACL-gated) + **kubeswitch** pointed at it (discovery + fetch in one `switch` command). Portal API is the endgame, not now. ## Where this lives This spans repos — most belongs in its **own repo** (kubeconfig distribution + tunnel orchestration), while the apiserver flags side is flux-clusters #11. The k8s-hub data plane lives on portable (managed via flux-clusters `generic-templates/k8s-hub`) tracked via #38 (#39 resolved). *(Originally agentgateway on mini; migrated to the Cilium Gateway API in 2026-09.)* ## Open questions (to answer while implementing) - ~~agentgateway placement/TLS certs for the apiserver routes (depends on #39 unblocking)~~ — **#39 resolved (2026-08-12)**: the exchange is verified; placement/certs is now implementation work (Gate-2 in #38). - Whether human and machine traffic share one gateway listener or get separate routes.
ginjiruu commented 2026-08-13 19:30:02 +00:00
Author
Owner
Copy link

Cross-link from the edge-ingress research (flux-clusters #1527): the portable↔mini connectivity path is now being built with Cilium ClusterMesh (flux-clusters #1528), which complements — not replaces — the reverse-tunnel plan here.

Key overlap: with the mesh up, an agentgateway running in portable (edge) can reach mini's apiserver token-exchange routes (RFC 8693, per lab/iac#38) as a normal local Service backend via Global Services, no new VIPs needed. That collapses the "which cluster holds the apiserver agentgateway" question: the edge agentgateway in portable can front the mini apiserver directly.

Two caveats to carry forward into #28's design:

  • The mesh does not help firewalled clusters that can't hold BGP/LB connectivity — the frp/rathole reverse-tunnel path stays for those.
  • Cilium Egress Gateway is not compatible with Cluster Mesh (gateway node must be same-cluster as the pods). Any fixed-SNAT egress design for meshed clusters must be per-cluster or via a central external gateway.
Cross-link from the edge-ingress research (flux-clusters **#1527**): the portable↔mini connectivity path is now being built with **Cilium ClusterMesh** (flux-clusters **#1528**), which complements — not replaces — the reverse-tunnel plan here. Key overlap: with the mesh up, an agentgateway running in portable (edge) can reach mini's apiserver token-exchange routes (RFC 8693, per lab/iac#38) as a normal local Service backend via Global Services, no new VIPs needed. That collapses the "which cluster holds the apiserver agentgateway" question: the edge agentgateway in portable can front the mini apiserver directly. Two caveats to carry forward into #28's design: - The mesh does **not** help firewalled clusters that can't hold BGP/LB connectivity — the frp/rathole reverse-tunnel path stays for those. - **Cilium Egress Gateway is not compatible with Cluster Mesh** (gateway node must be same-cluster as the pods). Any fixed-SNAT egress design for meshed clusters must be per-cluster or via a central external gateway.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
renovate/configure
t15-multicluster-catalog
t14-vault-k8s-auth
No results found.
Labels
Clear labels   
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
ready-for-agent
Fully specified, ready for an AFK agent

  
ready-for-human
Requires human implementation

  
wontfix
Will not be actioned

No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
wontfix
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lab/iac#28
No description provided.