Skip to content

Commit 9dd2773

Browse files
authored
chore(build): enhance proto caching and install binary plugins for improved performance, remove reliance on BSR (zitadel#11634)
1 parent 369b468 commit 9dd2773

23 files changed

Lines changed: 363 additions & 125 deletions

File tree

.devcontainer/devcontainer.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
"remoteEnv": {
4747
"PATH": "${containerEnv:PATH}:/workspaces/zitadel/.artifacts/bin/linux/amd64"
4848
},
49-
"postStartCommand": "echo 'export PATH=\"${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH):$PATH\"' >> ~/.bashrc",
49+
"postStartCommand": "npm install -g nx@21.6.9 && echo 'export PATH=\"${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH):$PATH\"' >> ~/.bashrc",
5050
"customizations": {
5151
"vscode": {
5252
"extensions": [

AGENTS.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -85,10 +85,13 @@ Run commands from the repository root.
8585
- Known exception: `@zitadel/console` has no configured `test` target.
8686

8787
## Verified Common Targets
88-
- `@zitadel/api`: `prod`, `build`, `generate`, `lint`, `test`, `test-unit`, `test-integration`
88+
- `@zitadel/api`: `prod`, `build`, `generate`, `generate-install`, `lint`, `test`, `test-unit`, `test-integration`
8989
- `@zitadel/login`: `dev`, `build`, `lint`, `test`, `test-unit`, `test-integration`
90-
- `@zitadel/docs`: `dev`, `build`, `generate`, `check-links`, `check-types`, `test`, `lint`
91-
- `@zitadel/console`: `dev`, `build`, `generate`, `lint`
90+
- `@zitadel/docs`: `dev`, `build`, `generate`, `install-proto-plugins`, `check-links`, `check-types`, `test`, `lint`
91+
- `@zitadel/console`: `dev`, `build`, `generate`, `install-proto-plugins`, `lint`
92+
93+
## Proto Plugin Binaries
94+
All proto plugins are installed to `.artifacts/bin/<GOOS>/<GOARCH>/` and Nx-cached. `generate` targets wire up the correct install dependency and prepend `.artifacts/bin/` to `$PATH` — no manual install step is needed.
9295

9396
## PR Title Convention
9497

CONTRIBUTING.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,9 @@ You can build and start any project with Nx commands.
4949
| **Develop** | `pnpm nx run PROJECT:dev` | Development server | |
5050
| **Generate** | `pnpm nx run PROJECT:generate` | Generate .gitignored files | |
5151
| **Generate Go Files** | `pnpm nx run @zitadel/api:generate-go` | Regenerate checked-in files | This is needed to generate files using [Stringer](https://pkg.go.dev/golang.org/x/tools/cmd/stringer), [Enumer](https://github.com/dmarkham/enumer) or [gomock](https://github.com/uber-go/mock) |
52+
| **Install Proto Plugins** | `pnpm nx run @zitadel/api:generate-install` | Install proto toolchain | Installs Go-based plugins (protoc-gen-go, connect-go, …) to `.artifacts/bin/`. Run automatically by `generate` targets; Nx caches the outputs. |
5253
| **Test - Unit** | `pnpm nx run PROJECT:test-unit` | Run unit tests | |
53-
| **Test - Integration** | `pnpm nx run PROJECT:test-integration` | Run integration tests | Learn mnore about how to [debug API integration tests](#run-api-integration-tests) |
54+
| **Test - Integration** | `pnpm nx run PROJECT:test-integration` | Run integration tests | Learn more about how to [debug API integration tests](#run-api-integration-tests) |
5455
| **Test - Integration Stop** | `pnpm nx run PROJECT:test-integration-stop` | Stop integration containers | |
5556
| **Test - Functional UI** | `pnpm nx run @zitadel/functional-ui:test` | Run functional UI tests | Learn more about how to [develop the Management Console and opening the interactive Test Suite](#pass-management-console-quality-checks) |
5657
| **Test - Functional UI Stop** | `pnpm nx run @zitadel/functional-ui:stop` | Run functional UI containers | |
@@ -578,6 +579,18 @@ pnpm nx run @zitadel/proto:generate # Regenerate after proto changes
578579
pnpm nx run @zitadel/client:build # Build after changes
579580
```
580581

582+
### Proto Plugin Convention
583+
584+
All binary proto plugins are installed to `.artifacts/bin/<GOOS>/<GOARCH>/` and declared as Nx target outputs, making them eligible for Nx remote cache.
585+
586+
| Scope | Target | Installs |
587+
|---|---|---|
588+
| `@zitadel/api` | `generate-install` | Go-based plugins: `buf`, `protoc-gen-go`, `protoc-gen-connect-go`, `protoc-gen-openapiv2`, `protoc-gen-validate`, `protoc-gen-authoption`, … |
589+
| `@zitadel/console` | `install-proto-plugins` | `protoc-gen-grpc-web`, `protoc-gen-js`, `protoc-gen-openapiv2` (pre-built binaries, no Go required) |
590+
| `@zitadel/docs` | `install-proto-plugins` | `protoc-gen-connect-openapi` (pre-built binary, no Go required) |
591+
592+
`generate` targets depend on the appropriate install targets and prepend `.artifacts/bin/` to `$PATH` automatically. Running `pnpm nx run PROJECT:generate` is sufficient — no manual plugin installation needed.
593+
581594
### Contribute to Docs
582595

583596
Project documentation is located under [./apps/docs](./apps/docs).

apps/api/AGENTS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ The **API App** (`apps/api`) is the Nx application target for building and runni
1212
- **Run API (prod profile)**: `pnpm nx run @zitadel/api:prod`
1313
- **Build**: `pnpm nx run @zitadel/api:build`
1414
- **Generate (all)**: `pnpm nx run @zitadel/api:generate`
15+
- **Install Proto Plugins**: `pnpm nx run @zitadel/api:generate-install` — installs all Go-based proto plugins (`buf`, `protoc-gen-go`, `protoc-gen-connect-go`, `protoc-gen-openapiv2`, `protoc-gen-validate`, `protoc-gen-authoption`, etc.) to `.artifacts/bin/$(GOOS)/$(GOARCH)/`. Output is Nx-cached; only reruns when plugin versions or local protoc sources change.
1516
- **Lint**: `pnpm nx run @zitadel/api:lint`
1617
- **Test (all)**: `pnpm nx run @zitadel/api:test`
1718
- **Test (unit)**: `pnpm nx run @zitadel/api:test-unit`
@@ -20,3 +21,4 @@ The **API App** (`apps/api`) is the Nx application target for building and runni
2021
## Generation Notes
2122
- `@zitadel/api:generate` can update generated, tracked files (stubs/assets/statik). Run it intentionally.
2223
- API changes in `proto/` often require regenerating API, package, and docs artifacts.
24+
- Proto plugins are installed to `.artifacts/bin/` — do not commit these binaries; they are declared as Nx outputs and restored from cache.

apps/api/project.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,6 @@
261261
"GOBIN=${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH) go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@v2.22.0",
262262
"GOBIN=${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH) go install github.com/envoyproxy/protoc-gen-validate@v1.1.0",
263263
"GOBIN=${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH) go install connectrpc.com/connect/cmd/protoc-gen-connect-go@v1.18.1",
264-
"GOBIN=${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH) go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.35.1",
265264
"GOBIN=${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH) go install ./internal/protoc/protoc-gen-authoption",
266265
"GOBIN=${PWD}/.artifacts/bin/$(go env GOOS)/$(go env GOARCH) go install ./internal/protoc/protoc-gen-zitadel"
267266
]

apps/docs/buf.gen.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
version: v2
22
plugins:
3-
- remote: buf.build/community/sudorandom-connect-openapi:v0.19.1
3+
- local: protoc-gen-connect-openapi
44
out: .
55
opt:
66
- format=json

apps/docs/content/guides/solution-scenarios/introduction/index.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import { FileText, Folder, Link as LinkIcon } from 'lucide-react';
1212

1313
<Cards>
1414
<Card title="B2b" href="/guides/solution-scenarios/b2b" icon={<FileText />} />
15-
<Card title="Common Settings" href="/guides/solution-scenarios/configurations" icon={<FileText />} />
15+
<Card title="Configurations" href="/guides/solution-scenarios/configurations" icon={<FileText />} />
1616
<Card title="Saas" href="/guides/solution-scenarios/saas" icon={<FileText />} />
1717
<Card title="B2c" href="/guides/solution-scenarios/b2c" icon={<FileText />} />
1818
<Card title="Frontend Calling Backend API" href="/guides/solution-scenarios/frontend-calling-backend-API" icon={<FileText />} />

apps/docs/package.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,6 @@
4949
"zod": "^4.3.6"
5050
},
5151
"devDependencies": {
52-
"@bufbuild/buf": "^1.65.0",
5352
"@eslint/eslintrc": "^3.3.3",
5453
"@eslint/js": "^10.0.1",
5554
"@tailwindcss/postcss": "^4.1.18",

apps/docs/project.json

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,24 +63,44 @@
6363
"generate-proto-docs": {
6464
"executor": "nx:run-commands",
6565
"options": {
66-
"command": "pnpm generate:proto-docs",
66+
"command": "bash -c 'GOOS=$(uname -s | tr \"[:upper:]\" \"[:lower:]\"); GOARCH=$(uname -m); GOARCH=${GOARCH/x86_64/amd64}; GOARCH=${GOARCH/aarch64/arm64}; PATH=\"${PWD}/../../.artifacts/bin/${GOOS}/${GOARCH}:$PATH\" pnpm generate:proto-docs'",
6767
"cwd": "apps/docs"
6868
},
6969
"dependsOn": [
70-
"fetch-remote-content"
70+
"fetch-remote-content",
71+
"install-proto-plugins"
7172
],
7273
"outputs": [
7374
"{projectRoot}/openapi"
7475
],
7576
"cache": true,
7677
"inputs": [
77-
"default",
78+
"{projectRoot}/scripts/generate-proto-docs.mjs",
79+
"{projectRoot}/buf.gen.yaml",
7880
"{workspaceRoot}/proto/**/*",
7981
{
8082
"dependentTasksOutputFiles": "apps/docs/content/versions.json"
83+
},
84+
{
85+
"dependentTasksOutputFiles": ".artifacts/bin/*/*/protoc-gen-connect-openapi"
8186
}
8287
]
8388
},
89+
"install-proto-plugins": {
90+
"executor": "nx:run-commands",
91+
"options": {
92+
"command": "bash ./apps/docs/scripts/install-proto-plugins.sh"
93+
},
94+
"cache": true,
95+
"inputs": [
96+
{ "runtime": "uname -s" },
97+
{ "runtime": "uname -m" },
98+
"{projectRoot}/scripts/install-proto-plugins.sh"
99+
],
100+
"outputs": [
101+
"{workspaceRoot}/.artifacts/bin/*/*/protoc-gen-connect-openapi"
102+
]
103+
},
84104
"generate-api-reference": {
85105
"executor": "nx:run-commands",
86106
"options": {
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/usr/bin/env bash
2+
# Downloads protoc-gen-connect-openapi for docs OpenAPI generation.
3+
# Must be run from the workspace root.
4+
# Supports Linux (including WSL2) and macOS only.
5+
#
6+
# WHY a binary download instead of "go install":
7+
# This script runs as an Nx target that docs/generate-proto-docs depends on.
8+
# Docs OpenAPI generation happens both in CI (has Go) and on Vercel
9+
# (Node.js-only, no Go). Using a pre-built binary means Vercel never needs a
10+
# Go toolchain. Nx remote cache means the download is skipped on repeated runs.
11+
#
12+
# WHY uname instead of "go env GOOS/GOARCH":
13+
# Same reason — this script must work without Go present.
14+
set -euo pipefail
15+
16+
# ── VERSIONS & CHECKSUMS ────────────────────────────────────────────────────
17+
# To upgrade: update VERSION and the SHA256_<os>_<arch> vars below.
18+
# Note: macOS ships a universal binary, so darwin_amd64 and darwin_arm64 share
19+
# the same tarball and checksum.
20+
21+
CONNECT_OPENAPI_VERSION="0.25.2"
22+
# checksums from upstream checksums.txt
23+
CONNECT_OPENAPI_SHA256_linux_amd64="90821ab96f16747dc5a4ab93900a6bc6b968d63f47553a08274dc594301bced3"
24+
CONNECT_OPENAPI_SHA256_linux_arm64="9726418d7af55f87e3bbb2a4fa6233016b200caf5a285ce1c6e6428ef225536d"
25+
CONNECT_OPENAPI_SHA256_darwin_amd64="a6bc3d87b4caaa7048d22fe0f45e3b5e778b4b209f84a836b3335fb2fd14b588"
26+
CONNECT_OPENAPI_SHA256_darwin_arm64="a6bc3d87b4caaa7048d22fe0f45e3b5e778b4b209f84a836b3335fb2fd14b588"
27+
28+
# ── HELPERS ──────────────────────────────────────────────────────────────────
29+
30+
verify_sha256() {
31+
local file="$1" expected="$2"
32+
local actual
33+
if command -v sha256sum &>/dev/null; then
34+
actual=$(sha256sum "$file" | cut -d' ' -f1)
35+
else
36+
actual=$(shasum -a 256 "$file" | cut -d' ' -f1)
37+
fi
38+
if [ "$actual" != "$expected" ]; then
39+
echo "ERROR: SHA256 mismatch for $(basename "$file")" >&2
40+
echo " expected: $expected" >&2
41+
echo " actual: $actual" >&2
42+
exit 1
43+
fi
44+
}
45+
46+
# ── PLATFORM DETECTION ───────────────────────────────────────────────────────
47+
48+
_uname_os=$(uname -s | tr '[:upper:]' '[:lower:]')
49+
_uname_arch=$(uname -m)
50+
case "$_uname_os" in
51+
linux) GOOS="linux" ;;
52+
darwin) GOOS="darwin" ;;
53+
*) echo "Unsupported OS: $_uname_os" >&2; exit 1 ;;
54+
esac
55+
case "$_uname_arch" in
56+
x86_64) GOARCH="amd64" ;;
57+
aarch64 | arm64) GOARCH="arm64" ;;
58+
*) echo "Unsupported arch: $_uname_arch" >&2; exit 1 ;;
59+
esac
60+
61+
BIN_DIR="${PWD}/.artifacts/bin/${GOOS}/${GOARCH}"
62+
mkdir -p "$BIN_DIR"
63+
64+
TMP=$(mktemp -d "${TMPDIR:-/tmp}/zitadel-docs-proto-plugins.XXXXXX")
65+
trap 'rm -rf "$TMP"' EXIT
66+
67+
# ── INSTALL ───────────────────────────────────────────────────────────────────
68+
69+
# ----- protoc-gen-connect-openapi (sudorandom/protoc-gen-connect-openapi) -----
70+
# macOS ships a single universal ("all") tarball for both amd64 and arm64.
71+
case "$GOOS" in
72+
linux) OAI_ARCH="$GOARCH" ;;
73+
darwin) OAI_ARCH="all" ;;
74+
esac
75+
sha256_var="CONNECT_OPENAPI_SHA256_${GOOS}_${GOARCH}"
76+
echo "Downloading protoc-gen-connect-openapi v${CONNECT_OPENAPI_VERSION} (${GOOS}/${OAI_ARCH})..."
77+
curl -fsSL \
78+
"https://github.com/sudorandom/protoc-gen-connect-openapi/releases/download/v${CONNECT_OPENAPI_VERSION}/protoc-gen-connect-openapi_${CONNECT_OPENAPI_VERSION}_${GOOS}_${OAI_ARCH}.tar.gz" \
79+
-o "${TMP}/oai.tar.gz"
80+
verify_sha256 "${TMP}/oai.tar.gz" "${!sha256_var}"
81+
tar -xzf "${TMP}/oai.tar.gz" -C "${TMP}" protoc-gen-connect-openapi
82+
install -m 755 "${TMP}/protoc-gen-connect-openapi" "${BIN_DIR}/protoc-gen-connect-openapi"
83+
84+
echo "protoc-gen-connect-openapi installed to ${BIN_DIR}"

0 commit comments

Comments
 (0)