🤖 feat: support all-namespaces LIST across multiple CoderControlPlane instances (#85)

## Summary

Enable all-namespaces LIST to aggregate results across every eligible
`CoderControlPlane` instance (one per namespace) for `CoderTemplate` and
`CoderWorkspace` resources. Previously, `kubectl get codertemplates -A`
failed with:

> multiple eligible CoderControlPlane instances across namespaces;
multi-instance support is planned

## Background

The aggregated API server's storage assumed exactly one eligible
`CoderControlPlane` when handling all-namespaces LIST (request namespace
is empty). When multiple eligible control planes exist across
namespaces, client list-watch loops fail, blocking `kubectl get
codertemplates -A`, `kubectl get coderworkspaces -A`, and
controllers/informers that use list+watch.

## Implementation

**New interface** — `coder.NamespaceLister` with
`EligibleNamespaces(ctx) ([]string, error)`:
- Opt-in capability that lets storage enumerate namespaces a provider
can serve.
- Keeps storage decoupled from concrete provider types.

**Provider implementations**:
- `ControlPlaneClientProvider.EligibleNamespaces` — discovers eligible
CPs via `findEligibleControlPlanes`, groups by namespace, rejects
duplicates within a namespace, returns sorted namespace list.
- `StaticClientProvider.EligibleNamespaces` — returns the pinned
namespace.

**Storage fan-out** in `TemplateStorage.List` and
`WorkspaceStorage.List`:
- When request namespace is empty and provider implements
`NamespaceLister`: fan out across all eligible namespaces, query each,
convert with correct per-namespace metadata, and return aggregated
results sorted by `(namespace, name)`.
- Namespaced requests and providers without `NamespaceLister` keep
existing behavior unchanged.
- Error semantics: fail-fast (if any namespace fails, return error
immediately).

## Validation

- `make verify-vendor` ✅
- `make test` ✅ (8 new tests + no regressions)
- `make build` ✅
- `make lint` ✅

## Risks

Low risk — the fan-out path only activates for all-namespaces LIST when
the provider supports `NamespaceLister`. Single-namespace requests and
the static provider path are unchanged. Multiple eligible CPs within the
same namespace are still explicitly rejected.


---

<details>
<summary>📋 Implementation Plan</summary>

# Plan: Support querying multiple CoderControlPlane instances
(multi-namespace aggregation)

## Context / Why
Today, the aggregated API server’s `CoderTemplate` / `CoderWorkspace`
storage assumes **exactly one** eligible `CoderControlPlane` when
handling **all-namespaces LIST** (i.e. request namespace is empty). When
multiple eligible control planes exist across namespaces, client
list-watch loops fail with:

- `multiple eligible CoderControlPlane instances across namespaces;
multi-instance support is planned`

This blocks workflows like:

- `kubectl get codertemplates -A`
- controllers/informers that use list+watch against
`aggregation.coder.com/v1alpha1`

## Goal
Enable **multi-instance querying** by making **all-namespaces LIST**
aggregate results across every eligible `CoderControlPlane` (one per
namespace) for:

- `aggregation.coder.com/v1alpha1/CoderTemplate`
- `aggregation.coder.com/v1alpha1/CoderWorkspace`

Namespaced requests (`-n <namespace>`) must keep current behavior.

## Non-goals (v1)
- Supporting **multiple eligible** `CoderControlPlane` objects **within
the same Kubernetes namespace**.
- Implementing a “true” upstream-backed watch (watch events that reflect
changes that occur directly in Coder without going through the
aggregated API server).

## Acceptance Criteria
- `kubectl get codertemplates -A` returns templates from **all**
eligible `CoderControlPlane` namespaces.
- `kubectl get coderworkspaces -A` returns workspaces from **all**
eligible `CoderControlPlane` namespaces.
- `kubectl get codertemplates -A --watch` no longer fails due to the
multi-instance discovery error (because the initial LIST succeeds).
- Standalone `aggregated-apiserver` mode (static provider pinned to
`--coder-namespace`) continues working unchanged.

## Evidence (code references)
- **Single-instance guard + error message**:
`internal/aggregated/coder/controlplane_provider.go`
  - `ClientForNamespace` errors on `len(eligible) > 1`
  - `DefaultNamespace` errors on multiple eligible across namespaces
- `multipleEligibleControlPlaneMessage("")` returns the exact string
shown in the screenshot
- **LIST path uses default namespace + empty-namespace client
resolution**:
  - `internal/aggregated/storage/template.go`: `(*TemplateStorage).List`
- `internal/aggregated/storage/workspace.go`: `(*WorkspaceStorage).List`
- both call `namespaceForListConversion(...)` and then
`clientForNamespace(ctx, requestNamespace)` where `requestNamespace==""`
triggers the provider’s “pick exactly one CP” logic
- **WATCH path is an in-memory broadcaster**:
`internal/aggregated/storage/watch.go`, `template.go`, `workspace.go`
- the screenshot’s “Watcher failed …” is consistent with list-watch
clients failing during the initial LIST phase

---

## Implementation Details

### 1) Add an optional provider capability: list eligible namespaces
**File:** `internal/aggregated/coder/provider.go`

Add a small, opt-in interface that lets storage enumerate the set of
namespaces it can serve.

```go
// NamespaceLister can enumerate namespaces served by a ClientProvider.
// Used to implement all-namespaces LIST by fanning out across instances.
//
// Implementations should only return namespaces that are eligible/ready.
// Returned namespaces must be non-empty and should be deterministic (sorted).
type NamespaceLister interface {
    EligibleNamespaces(ctx context.Context) ([]string, error)
}
```

Rationale: keeps storage decoupled from concrete provider types
(`StaticClientProvider` vs `ControlPlaneClientProvider`).

### 2) Implement `NamespaceLister`

#### 2a) Dynamic control-plane provider
**File:** `internal/aggregated/coder/controlplane_provider.go`

Implement:

- `func (p *ControlPlaneClientProvider) EligibleNamespaces(ctx
context.Context) ([]string, error)`

Algorithm:
1. `eligibleCPs, err := p.findEligibleControlPlanes(ctx, "")`
2. If `len(eligibleCPs) == 0`: return
`ServiceUnavailable(noEligibleControlPlaneMessage(""))`.
3. Group eligible CPs by `cp.Namespace`.
4. If any namespace has >1 eligible CP: return
`BadRequest(multipleEligibleControlPlaneMessage(namespace))` (still not
supported).
5. Return the set of namespaces, **sorted** for determinism.

Defensive programming:
- Assert/validate non-nil receiver and non-nil context.
- Ensure returned namespaces are not empty; crash/assert if an eligible
CP has an empty namespace (should be impossible).

#### 2b) Static provider
**File:** `internal/aggregated/coder/provider.go`

Implement:

- `func (p *StaticClientProvider) EligibleNamespaces(ctx
context.Context) ([]string, error)`

Behavior:
- If `p.Namespace == ""`: return `ServiceUnavailable("static provider
has no default namespace")` (consistent with existing behavior).
- Else return `[]string{p.Namespace}`.

### 3) Update storage LIST to fan out when request namespace is empty

#### 3a) Templates
**File:** `internal/aggregated/storage/template.go`

Update `func (s *TemplateStorage) List(ctx context.Context, _
*metainternalversion.ListOptions) ...`:

- If request namespace is non-empty: keep existing behavior.
- If request namespace is empty:
  - If `s.provider` implements `coder.NamespaceLister`:
    - `namespaces := lister.EligibleNamespaces(ctx)`
    - For each namespace:
      - `sdk := s.clientForNamespace(ctx, namespace)`
      - `templates := sdk.Templates(ctx, codersdk.TemplateFilter{})`
- Convert each template using `convert.TemplateToK8s(namespace,
template)` and append.
    - Sort `list.Items` by `(namespace, name)` for deterministic output.
- Else: fallback to existing `namespaceForListConversion` +
`clientForNamespace(ctx, "")` behavior.

#### 3b) Workspaces
**File:** `internal/aggregated/storage/workspace.go`

Update `func (s *WorkspaceStorage) List(ctx context.Context, _
*metainternalversion.ListOptions) ...` with the same fan-out strategy.

Concurrency (recommended, not required for correctness):
- Use an `errgroup.Group` + a small semaphore/limit (e.g. 4) so large
numbers of namespaces don’t produce N sequential slow requests.

Error semantics (v1):
- Fail-fast: if any namespace fails to resolve a client or list objects,
return an error (preserves today’s “LIST is authoritative” behavior).

### 4) Tests

#### 4a) Provider unit tests
**File:** `internal/aggregated/coder/controlplane_provider_test.go`

Add tests for `EligibleNamespaces`:
- Returns namespaces for multiple eligible CPs across namespaces.
- Returns `BadRequest` when a single namespace has multiple eligible
CPs.
- Returns `ServiceUnavailable` when no eligible CPs exist.

#### 4b) Storage aggregation tests
**File:** `internal/aggregated/storage/storage_test.go`

Add tests verifying all-namespaces aggregation:
- Create two mock Coder servers with distinct seeded
templates/workspaces.
- Create a test provider implementing:
  - `coder.ClientProvider` (map namespace → client)
  - `coder.NamespaceLister` (returns both namespaces)
- Assert:
- `TemplateStorage.List(context.Background(), nil)` returns items from
both namespaces and each item has the correct `metadata.namespace`.
  - same for `WorkspaceStorage.List`.

### 5) Docs / behavior notes
**File:** `internal/aggregated/storage/doc.go`

Update the “v1 semantics” comment to note:
- all-namespaces LIST aggregates across eligible `CoderControlPlane`
namespaces when the provider supports it.

### 6) Validation
Run locally after implementation:
- `make test`
- `make build`
- `make lint`
- `make verify-vendor` (expected no-op; only if deps are added)

---

<details>
<summary>Optional follow-ups (not required to fix the observed
error)</summary>

1) **Upstream-backed watch**: implement per-control-plane
polling/websocket to generate watch events even when changes happen
directly in Coder.

2) **Partial failure mode**: consider returning partial results for
all-namespaces LIST when one instance is down (would require a clear
API/UX decision; Kubernetes APIs generally favor fail-fast).

3) **Performance & caching**: optionally cache per-namespace SDK
clients/tokens (with invalidation on secret changes) to reduce
per-request secret reads.

</details>

</details>

---

_Generated with `mux` • Model: `anthropic:claude-opus-4-6` • Thinking:
`xhigh` • Cost: `$3.28`_

<!-- mux-attribution: model=anthropic:claude-opus-4-6 thinking=xhigh
costs=3.28 -->

Author: ThomasK33

internal/aggregated/coder/controlplane_provider.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"log"
 	"net/url"
+	"sort"
 	"strings"
 	"time"
 
@@ -26,6 +27,7 @@ type ControlPlaneClientProvider struct {
 var (
 	_ ClientProvider    = (*ControlPlaneClientProvider)(nil)
 	_ NamespaceResolver = (*ControlPlaneClientProvider)(nil)
+	_ NamespaceLister   = (*ControlPlaneClientProvider)(nil)
 )
 
 // NewControlPlaneClientProvider constructs a dynamic ClientProvider backed by CoderControlPlane resources.
@@ -206,6 +208,45 @@ func (p *ControlPlaneClientProvider) DefaultNamespace(ctx context.Context) (stri
 	}
 }
 
+// EligibleNamespaces returns namespaces served by eligible CoderControlPlane instances.
+func (p *ControlPlaneClientProvider) EligibleNamespaces(ctx context.Context) ([]string, error) {
+	if p == nil {
+		return nil, fmt.Errorf("assertion failed: control plane client provider must not be nil")
+	}
+	if ctx == nil {
+		return nil, fmt.Errorf("assertion failed: context must not be nil")
+	}
+
+	eligible, err := p.findEligibleControlPlanes(ctx, "")
+	if err != nil {
+		return nil, err
+	}
+	if len(eligible) == 0 {
+		return nil, apierrors.NewServiceUnavailable(noEligibleControlPlaneMessage(""))
+	}
+
+	byNamespace := make(map[string]int, len(eligible))
+	for i := range eligible {
+		namespace := strings.TrimSpace(eligible[i].Namespace)
+		if namespace == "" {
+			return nil, fmt.Errorf("assertion failed: eligible CoderControlPlane namespace must not be empty")
+		}
+		byNamespace[namespace]++
+	}
+
+	namespaces := make([]string, 0, len(byNamespace))
+	for namespace, count := range byNamespace {
+		if count > 1 {
+			return nil, apierrors.NewBadRequest(multipleEligibleControlPlaneMessage(namespace))
+		}
+		namespaces = append(namespaces, namespace)
+	}
+
+	sort.Strings(namespaces)
+
+	return namespaces, nil
+}
+
 func (p *ControlPlaneClientProvider) findEligibleControlPlanes(
 	ctx context.Context,
 	namespace string,

internal/aggregated/coder/controlplane_provider_test.go

@@ -266,6 +266,161 @@ func TestControlPlaneClientProviderDefaultNamespaceReturnsBadRequestForMultipleE
 	}
 }
 
+func TestControlPlaneClientProviderEligibleNamespacesMultipleNamespaces(t *testing.T) {
+	t.Parallel()
+
+	ignoredControlPlane := eligibleControlPlane("team-a", "coder-disabled")
+	ignoredControlPlane.Spec.OperatorAccess.Disabled = true
+
+	provider, secretReader := newControlPlaneProviderForTest(
+		t,
+		[]coderv1alpha1.CoderControlPlane{
+			eligibleControlPlane("team-a", "coder-a"),
+			eligibleControlPlane("team-b", "coder-b"),
+			ignoredControlPlane,
+		},
+		nil,
+	)
+
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err != nil {
+		t.Fatalf("resolve eligible namespaces: %v", err)
+	}
+	if got, want := secretReader.getCalls, 0; got != want {
+		t.Fatalf("expected %d secret reads, got %d", want, got)
+	}
+	if got, want := len(namespaces), 2; got != want {
+		t.Fatalf("expected %d namespaces, got %d: %v", want, got, namespaces)
+	}
+	if got, want := namespaces[0], "team-a"; got != want {
+		t.Fatalf("expected first namespace %q, got %q", want, got)
+	}
+	if got, want := namespaces[1], "team-b"; got != want {
+		t.Fatalf("expected second namespace %q, got %q", want, got)
+	}
+}
+
+func TestControlPlaneClientProviderEligibleNamespacesNoEligible(t *testing.T) {
+	t.Parallel()
+
+	provider, secretReader := newControlPlaneProviderForTest(t, nil, nil)
+
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if namespaces != nil {
+		t.Fatalf("expected nil namespaces on error, got %v", namespaces)
+	}
+	if got, want := secretReader.getCalls, 0; got != want {
+		t.Fatalf("expected %d secret reads, got %d", want, got)
+	}
+	if !apierrors.IsServiceUnavailable(err) {
+		t.Fatalf("expected ServiceUnavailable, got %v", err)
+	}
+	if !strings.Contains(err.Error(), "no eligible CoderControlPlane") {
+		t.Fatalf("expected no-eligible message, got %v", err)
+	}
+}
+
+func TestControlPlaneClientProviderEligibleNamespacesDuplicateInNamespace(t *testing.T) {
+	t.Parallel()
+
+	provider, secretReader := newControlPlaneProviderForTest(
+		t,
+		[]coderv1alpha1.CoderControlPlane{
+			eligibleControlPlane("team-a", "coder-a"),
+			eligibleControlPlane("team-a", "coder-b"),
+		},
+		nil,
+	)
+
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if namespaces != nil {
+		t.Fatalf("expected nil namespaces on error, got %v", namespaces)
+	}
+	if got, want := secretReader.getCalls, 0; got != want {
+		t.Fatalf("expected %d secret reads, got %d", want, got)
+	}
+	if !apierrors.IsBadRequest(err) {
+		t.Fatalf("expected BadRequest, got %v", err)
+	}
+	if !strings.Contains(err.Error(), "multi-instance support is planned") {
+		t.Fatalf("expected multi-instance message, got %v", err)
+	}
+}
+
+func TestControlPlaneClientProviderEligibleNamespacesSingleNamespace(t *testing.T) {
+	t.Parallel()
+
+	provider, secretReader := newControlPlaneProviderForTest(
+		t,
+		[]coderv1alpha1.CoderControlPlane{eligibleControlPlane("team-a", "coder-a")},
+		nil,
+	)
+
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err != nil {
+		t.Fatalf("resolve eligible namespaces: %v", err)
+	}
+	if got, want := secretReader.getCalls, 0; got != want {
+		t.Fatalf("expected %d secret reads, got %d", want, got)
+	}
+	if got, want := len(namespaces), 1; got != want {
+		t.Fatalf("expected %d namespace, got %d", want, got)
+	}
+	if got, want := namespaces[0], "team-a"; got != want {
+		t.Fatalf("expected namespace %q, got %q", want, got)
+	}
+}
+
+func TestControlPlaneClientProviderEligibleNamespacesAssertions(t *testing.T) {
+	t.Parallel()
+
+	provider, _ := newControlPlaneProviderForTest(t, nil, nil)
+
+	tests := []struct {
+		name            string
+		provider        *ControlPlaneClientProvider
+		ctx             context.Context
+		wantErrContains string
+	}{
+		{
+			name:            "rejects nil provider",
+			provider:        nil,
+			ctx:             context.Background(),
+			wantErrContains: "assertion failed: control plane client provider must not be nil",
+		},
+		{
+			name:            "rejects nil context",
+			provider:        provider,
+			ctx:             nil,
+			wantErrContains: "assertion failed: context must not be nil",
+		},
+	}
+
+	for _, testCase := range tests {
+		testCase := testCase
+		t.Run(testCase.name, func(t *testing.T) {
+			t.Parallel()
+
+			namespaces, err := testCase.provider.EligibleNamespaces(testCase.ctx)
+			if err == nil {
+				t.Fatalf("expected error containing %q, got nil", testCase.wantErrContains)
+			}
+			if namespaces != nil {
+				t.Fatalf("expected nil namespaces on error, got %v", namespaces)
+			}
+			if !strings.Contains(err.Error(), testCase.wantErrContains) {
+				t.Fatalf("expected error containing %q, got %q", testCase.wantErrContains, err.Error())
+			}
+		})
+	}
+}
+
 func TestControlPlaneClientProviderClientForNamespaceDefaultsSecretKeyToToken(t *testing.T) {
 	t.Parallel()
 

internal/aggregated/coder/provider.go

@@ -22,6 +22,12 @@ type NamespaceResolver interface {
 	DefaultNamespace(ctx context.Context) (string, error)
 }
 
+// NamespaceLister can enumerate namespaces served by a ClientProvider.
+// Used to implement all-namespaces LIST by fanning out across instances.
+type NamespaceLister interface {
+	EligibleNamespaces(ctx context.Context) ([]string, error)
+}
+
 // StaticClientProvider returns one static client, optionally restricted to one namespace.
 type StaticClientProvider struct {
 	Client    *codersdk.Client
@@ -31,6 +37,7 @@ type StaticClientProvider struct {
 var (
 	_ ClientProvider    = (*StaticClientProvider)(nil)
 	_ NamespaceResolver = (*StaticClientProvider)(nil)
+	_ NamespaceLister   = (*StaticClientProvider)(nil)
 )
 
 // ClientForNamespace returns the static client.
@@ -77,6 +84,18 @@ func (p *StaticClientProvider) DefaultNamespace(_ context.Context) (string, erro
 	return p.Namespace, nil
 }
 
+// EligibleNamespaces returns namespaces served by the static provider.
+func (p *StaticClientProvider) EligibleNamespaces(_ context.Context) ([]string, error) {
+	if p == nil {
+		return nil, fmt.Errorf("assertion failed: static client provider must not be nil")
+	}
+	if p.Namespace == "" {
+		return nil, apierrors.NewServiceUnavailable("static provider has no default namespace")
+	}
+
+	return []string{p.Namespace}, nil
+}
+
 // NewStaticClientProvider creates a StaticClientProvider from cfg and optional namespace restriction.
 func NewStaticClientProvider(cfg Config, namespace string) (*StaticClientProvider, error) {
 	client, err := NewSDKClient(cfg)

internal/aggregated/coder/provider_test.go

@@ -209,6 +209,58 @@ func TestStaticClientProviderDefaultNamespaceAssertions(t *testing.T) {
 	}
 }
 
+func TestStaticClientProviderEligibleNamespaces(t *testing.T) {
+	t.Parallel()
+
+	provider := &StaticClientProvider{Namespace: "control-plane"}
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err != nil {
+		t.Fatalf("resolve eligible namespaces: %v", err)
+	}
+	if got, want := len(namespaces), 1; got != want {
+		t.Fatalf("expected %d namespace, got %d", want, got)
+	}
+	if got, want := namespaces[0], "control-plane"; got != want {
+		t.Fatalf("expected namespace %q, got %q", want, got)
+	}
+}
+
+func TestStaticClientProviderEligibleNamespacesNoNamespace(t *testing.T) {
+	t.Parallel()
+
+	provider := &StaticClientProvider{}
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if namespaces != nil {
+		t.Fatalf("expected nil namespaces on error, got %v", namespaces)
+	}
+	if !apierrors.IsServiceUnavailable(err) {
+		t.Fatalf("expected ServiceUnavailable, got %v", err)
+	}
+	if !strings.Contains(err.Error(), "static provider has no default namespace") {
+		t.Fatalf("expected missing namespace message, got %v", err)
+	}
+}
+
+func TestStaticClientProviderEligibleNamespacesNilReceiver(t *testing.T) {
+	t.Parallel()
+
+	var provider *StaticClientProvider
+	namespaces, err := provider.EligibleNamespaces(context.Background())
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if namespaces != nil {
+		t.Fatalf("expected nil namespaces on error, got %v", namespaces)
+	}
+	wantErrContains := "assertion failed: static client provider must not be nil"
+	if !strings.Contains(err.Error(), wantErrContains) {
+		t.Fatalf("expected error containing %q, got %q", wantErrContains, err.Error())
+	}
+}
+
 func TestNewStaticClientProvider(t *testing.T) {
 	t.Parallel()
 

internal/aggregated/storage/doc.go

@@ -9,4 +9,6 @@
 //     while Kubernetes object names allow dots (DNS-1123 subdomains).
 //   - A single admin session token is used for all API calls (no per-request impersonation in v1).
 //   - Storage resolves the backing codersdk.Client via a ClientProvider interface.
+//   - All-namespaces LIST aggregates results across eligible CoderControlPlane namespaces
+//     when the provider implements NamespaceLister.
 package storage