Research: Sveltos for multi-cluster manifest delivery (firewalled cluster)

ginjiruu commented

2026-08-12 15:51:01 +00:00

Owner

Sveltos research — multi-cluster manifest delivery for the firewalled cluster

Research ticket. Captures the exploration of Sveltos (https://projectsveltos.io) as the delivery mechanism for tofu-rendered k8s manifests to the fleet, and the licensing finding that pulls mode off the table.

Context

The iac k8s/ root currently uses the hashicorp/kubernetes provider, which applies against a live apiserver — and that root is unwired to any cluster today (T10, #25). We explored two ways to reach clusters without per-cluster tofu runners:

Tofu render backend: hashicorp/kubernetes has no render backend (verified against the 2.38 schema — every resource applies live). But yamlencode() + local_file, or templatefile(), renders valid multi-doc YAML with zero cluster connectivity (empirically verified). So tofu can produce manifest bundles cluster-agnostically.
Sveltos to distribute those rendered manifests to the fleet by label match, including a firewalled cluster — via pull mode (the managed cluster's agent dials out to the management cluster).

Finding: pull mode is license-gated beyond 2 clusters

The pull-mode registration doc carries a hard constraint:

Test the Pull Mode with up to two managed clusters for free. Need more than two clusters? Contact us at support@projectsveltos.io to explore license options based on your needs!

We have five clusters (localhost, mini, mx2, portable, remote) plus an incoming firewalled cluster — well past two. Pull mode is out of scope. Any firewalled-cluster solution must therefore solve the firewall problem directly (Sveltos push mode cannot reach a cluster it cannot connect to), i.e. the T13 reverse-tunnel work (#28) is a prerequisite regardless.

What still holds for Sveltos

Sveltos push mode remains a candidate for the reachable fleet: a Sveltos controller on mini (management cluster), ClusterProfile selecting clusters by label, policyRefs reading rendered manifests from ConfigMaps/Secrets/OCI in the management cluster. Free, no per-cluster inbound needed for reachable clusters.
Tofu render + Sveltos is the synthesis: the iac k8s/ root renders per-app manifest bundles (no cluster), the tofu runner writes them into a ConfigMap on mini, a ClusterProfile distributes them. No kubeconfig distribution, no per-cluster tofu runner.

Open questions to research

Sveltos push mode architecture on mini: does the add-on controller need to reach each cluster's apiserver (yes — same network reachability as tofu would), and how does it register clusters (kubeconfig secret vs in-cluster)?
Sveltos pull mode license: is the 2-cluster cap per cluster profile / per deployment / per Sveltos version? Any OSS/community path (e.g. run two management clusters, or pull mode without the "managed cluster" quota)?
Does Sveltos support pulling rendered tofu output from an OCI registry (oci:// policyRef, deploymentType: Remote) — would that let the tofu runner push manifests to a registry instead of a ConfigMap?
Sveltos vs Flux multi-cluster (already used: Kustomization per cluster) for the same job — does Sveltos add enough (label-based targeting, templating, drift detection) to justify a new controller?
How rendered tofu output coexists with the existing flux-clusters manifests (per-cluster dirs) — migration/ownership boundaries.
Sveltos RBAC multi-tenancy (ClusterProfile vs Profile) mapped to the catalog capability groups (ADR-0006/0007).

T13 #28: kubeconfig distribution & connectivity architecture — the firewall prerequisite for the firewalled cluster
T10 #25: tofu-controller sync for provider roots — the k8s/ root is not wired
VSO deployment #1525 (flux-clusters) / T14 #29 (iac)

## Sveltos research — multi-cluster manifest delivery for the firewalled cluster Research ticket. Captures the exploration of Sveltos (https://projectsveltos.io) as the delivery mechanism for tofu-rendered k8s manifests to the fleet, and the licensing finding that pulls mode off the table. ## Context The iac `k8s/` root currently uses the `hashicorp/kubernetes` provider, which applies against a live apiserver — and that root is unwired to any cluster today (T10, #25). We explored two ways to reach clusters without per-cluster tofu runners: 1. **Tofu render backend**: `hashicorp/kubernetes` has **no render backend** (verified against the 2.38 schema — every resource applies live). But `yamlencode()` + `local_file`, or `templatefile()`, renders valid multi-doc YAML with **zero cluster connectivity** (empirically verified). So tofu can produce manifest bundles cluster-agnostically. 2. **Sveltos** to distribute those rendered manifests to the fleet by label match, including a firewalled cluster — via **pull mode** (the managed cluster's agent dials *out* to the management cluster). ## Finding: pull mode is license-gated beyond 2 clusters The pull-mode registration doc carries a hard constraint: > *Test the Pull Mode with up to two managed clusters for free. Need more than two clusters? Contact us at support@projectsveltos.io to explore license options based on your needs!* We have **five** clusters (`localhost`, `mini`, `mx2`, `portable`, `remote`) plus an incoming firewalled cluster — well past two. **Pull mode is out of scope.** Any firewalled-cluster solution must therefore solve the firewall problem directly (Sveltos push mode cannot reach a cluster it cannot connect to), i.e. the T13 reverse-tunnel work (#28) is a prerequisite regardless. ## What still holds for Sveltos - **Sveltos push mode** remains a candidate for the reachable fleet: a Sveltos controller on mini (management cluster), `ClusterProfile` selecting clusters by label, `policyRefs` reading rendered manifests from ConfigMaps/Secrets/OCI in the management cluster. Free, no per-cluster inbound needed for reachable clusters. - **Tofu render + Sveltos** is the synthesis: the iac `k8s/` root renders per-app manifest bundles (no cluster), the tofu runner writes them into a ConfigMap on mini, a `ClusterProfile` distributes them. No kubeconfig distribution, no per-cluster tofu runner. ## Open questions to research - Sveltos **push mode** architecture on mini: does the add-on controller need to reach each cluster's apiserver (yes — same network reachability as tofu would), and how does it register clusters (kubeconfig secret vs in-cluster)? - Sveltos **pull mode** license: is the 2-cluster cap per cluster profile / per deployment / per Sveltos version? Any OSS/community path (e.g. run two management clusters, or pull mode without the "managed cluster" quota)? - Does Sveltos support pulling rendered tofu output from an **OCI registry** (`oci://` policyRef, `deploymentType: Remote`) — would that let the tofu runner push manifests to a registry instead of a ConfigMap? - Sveltos vs **Flux multi-cluster** (already used: Kustomization per cluster) for the same job — does Sveltos add enough (label-based targeting, templating, drift detection) to justify a new controller? - How rendered tofu output coexists with the existing flux-clusters manifests (per-cluster dirs) — migration/ownership boundaries. - Sveltos RBAC multi-tenancy (`ClusterProfile` vs `Profile`) mapped to the catalog capability groups (ADR-0006/0007). ## Related - T13 #28: kubeconfig distribution & connectivity architecture — **the firewall prerequisite for the firewalled cluster** - T10 #25: tofu-controller sync for provider roots — the `k8s/` root is not wired - VSO deployment #1525 (flux-clusters) / T14 #29 (iac)

ginjiruu referenced this issue

2026-08-12 16:01:24 +00:00

Multi-cluster catalog: cluster registry + per-app deployment targets #32

ginjiruu referenced this issue

2026-08-12 16:01:47 +00:00

Multi-cluster catalog: cluster registry + per-app deployment targets #33

ginjiruu commented

2026-08-12 16:10:37 +00:00

Author

Owner

Flux integration design (from plans/sveltos-flux-integration.md)

The Sveltos×Flux integration (https://projectsveltos.io/main/use_cases/use_case_gitops/) lets a ClusterProfile on mini reference the existing flux-system GitRepository (the flux-clusters repo) via kustomizationRefs with a path, and fans the rendered kustomization out to every cluster matching its clusterSelector. The generic-templates/ dirs stay the single source of truth; the per-cluster pointer files (mini/portable/remote vault-secrets-operator.yaml, the 4 ESO HelmRelease sets, the 4 standard-cluster Kustomizations) are deleted and replaced by ONE ClusterProfile each.

Concrete de-dup delivered by the registry labels:

apps/clusters.yaml (iac)                          flux-clusters
  clusters.<name>.sveltos                          sveltos/vault-secrets-operator.yaml
    sveltos.io/vault-secrets-operator: enabled ──▶ ClusterProfile
                                                    └─ kustomizationRefs: flux-system
                                                       path: generic-templates/vault-secrets-operator

Ordering gate: the ClusterProfile CRD must be installed on mini before the profiles are applied (Flux root build fails on unknown CRDs), so the profiles are staged in flux-clusters sveltos/ and wired in after Sveltos install.

Net: new fleet member = add sveltos labels to apps/clusters.yaml + register the cluster with Sveltos. No new manifests, no per-cluster pointers.

apps/clusters.yaml now carries sveltos.io/fleet: lab + sveltos.io/vault-secrets-operator: enabled per cluster (mini/portable/remote).

## Flux integration design (from plans/sveltos-flux-integration.md) The Sveltos×Flux integration (https://projectsveltos.io/main/use_cases/use_case_gitops/) lets a `ClusterProfile` on mini reference the **existing** `flux-system` GitRepository (the flux-clusters repo) via `kustomizationRefs` with a `path`, and fans the rendered kustomization out to every cluster matching its `clusterSelector`. The `generic-templates/` dirs stay the single source of truth; the per-cluster pointer files (mini/portable/remote vault-secrets-operator.yaml, the 4 ESO HelmRelease sets, the 4 standard-cluster Kustomizations) are deleted and replaced by ONE ClusterProfile each. Concrete de-dup delivered by the registry labels: ``` apps/clusters.yaml (iac) flux-clusters clusters.<name>.sveltos sveltos/vault-secrets-operator.yaml sveltos.io/vault-secrets-operator: enabled ──▶ ClusterProfile └─ kustomizationRefs: flux-system path: generic-templates/vault-secrets-operator ``` Ordering gate: the `ClusterProfile` CRD must be installed on mini before the profiles are applied (Flux root build fails on unknown CRDs), so the profiles are staged in flux-clusters `sveltos/` and wired in after Sveltos install. Net: new fleet member = add `sveltos` labels to `apps/clusters.yaml` + register the cluster with Sveltos. No new manifests, no per-cluster pointers. `apps/clusters.yaml` now carries `sveltos.io/fleet: lab` + `sveltos.io/vault-secrets-operator: enabled` per cluster (mini/portable/remote).

ginjiruu referenced this issue from a commit

2026-08-12 16:12:14 +00:00

sveltos (Add): stage ClusterProfiles for fleet-wide shared infrastructure

ginjiruu referenced this issue

2026-08-12 16:49:43 +00:00

Sveltos: install on mini + bootstrap cluster registration (static SA) #34

ginjiruu referenced this issue

2026-08-13 02:04:58 +00:00

T10: tofu-controller sync for provider roots (flux-clusters) #25

ginjiruu commented

2026-08-13 21:22:39 +00:00

Author