Skip to content

Commit 8072c08

Browse files
AhmedTMMclaude
andauthored
feat: pre-built agent tarballs for fast install (#2232)
* feat: pre-built agent tarballs on GitHub Releases for fast install Adds a nightly GitHub Actions workflow that builds and uploads agent tarballs to rolling GitHub Releases. During provisioning, the CLI now attempts to download and extract a tarball before falling back to live install. Priority chain: snapshot > tarball > live install. - New workflow: .github/workflows/agent-tarballs.yml - New capture script: packer/scripts/capture-agent.sh - New module: packages/cli/src/shared/agent-tarball.ts - Orchestrate tries tarball first on non-local clouds - Skip tarball when using DO snapshot (skipTarball flag) - Tests for tarball install + orchestration integration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use global.fetch mock pattern and address security review - Use `global.fetch = mock(...)` instead of `spyOn(globalThis, "fetch")` to match codebase convention and fix CI mock interception - Add URL validation regex to reject shell metacharacters (CRITICAL) - Add agent name validation in workflow input (MEDIUM) - Add `jq has()` check before executing install commands (CRITICAL) - Use `tar -T` instead of unquoted word-splitting in capture-agent.sh (MEDIUM) - Resolve merge conflicts with upstream/main (keep Docker fields, adapt to simplified DO flow, bump version to 0.15.0) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use globalThis.fetch for testability in CI Bun's native fetch binding doesn't go through global.fetch property lookup, so global.fetch = mock(...) doesn't intercept it. Using globalThis.fetch explicitly ensures the mock interception works. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing packer dependencies and harden install command safety - Add packer/agents.json (agent tier + install command definitions) - Add packer/scripts/tier-{minimal,node,bun,full}.sh (dependency scripts) - Add basic command safety check rejecting suspicious patterns - Document packer/agents.json as a trust boundary requiring PR review Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tarballs): fix npm prefix mismatch, add apt-get update, cleanup - Add apt-get update -y before apt-get install in all tier scripts - Add --prefix ~/.npm-global to npm install commands in agents.json so installed packages land where capture-agent.sh expects them - Rename misleading MARKER_DIR → MARKER_FILE in capture-agent.sh - Remove stale comment referencing packer snapshots in workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tarballs): detect empty agent installs in capture script The "no files found" check was dead code — the marker file is always created before filtering, so FILTERED_FILE always had at least one entry. Now we count non-marker entries to catch cases where the agent install silently fails and no actual files are on disk. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tarballs): use bare fetch() for Bun mock compatibility in CI In Bun, global.fetch = mock(...) overrides bare fetch() calls but NOT globalThis.fetch() calls. Every other source file in the codebase uses bare fetch() and their mocks work fine in CI. Switch to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tarballs): use dependency injection for fetch in tests Bun's global.fetch mock doesn't reliably intercept bare fetch() calls across all Bun versions in CI. Instead of fighting the runtime, accept an optional fetchFn parameter (defaults to fetch) and pass mock fetch directly in tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tarballs): bypass mock.module bleed in agent-tarball tests orchestrate.test.ts uses mock.module("../shared/agent-tarball", ...) which is process-global in Bun and bleeds into agent-tarball.test.ts. Import via URL (import.meta.url resolution) to bypass the specifier- based mock matching and get the real module. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tarballs): eliminate mock.module bleed between test files Bun's mock.module is process-global — orchestrate.test.ts mocking agent-tarball poisoned agent-tarball.test.ts (the mock function ignored the fetchFn parameter and always returned false). Fix: make tryTarballInstall injectable via OrchestrationOptions. orchestrate.test.ts passes the mock directly via options instead of using mock.module. agent-tarball.test.ts imports the real module. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tests): mock Bun.which in credential priority tests Tests assumed no cloud CLIs were installed, but machines with hcloud/ doctl would get "CLI installed" hint overrides, failing the assertion. Spy on Bun.which to return null so tests are environment-independent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: fix import ordering after rebase Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * security: add curl domain allowlist and expand command blocklist Addresses security review findings: - Add domain allowlist for curl/wget targets (claude.ai, opencode.ai, raw.githubusercontent.com, registry.npmjs.org, crates.io, github.com) - Expand suspicious command blocklist (python -c, perl -e, ruby -e, dd, /dev/) - Document 4-layer security model in workflow comments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * security: add rm -rf to command blocklist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: Ahmed Abushagur <ahmed@abushagur.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 8bc45b4 commit 8072c08

14 files changed

Lines changed: 724 additions & 9 deletions
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
name: Agent Tarballs
2+
3+
on:
4+
schedule:
5+
# 5 AM UTC daily
6+
- cron: "0 5 * * *"
7+
workflow_dispatch:
8+
inputs:
9+
agent:
10+
description: "Single agent to build (leave empty for all)"
11+
required: false
12+
type: string
13+
14+
permissions:
15+
contents: write
16+
17+
jobs:
18+
matrix:
19+
runs-on: ubuntu-latest
20+
outputs:
21+
agents: ${{ steps.set-matrix.outputs.agents }}
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- id: set-matrix
26+
env:
27+
AGENT_INPUT: ${{ inputs.agent }}
28+
run: |
29+
if [ -n "${AGENT_INPUT:-}" ]; then
30+
# Validate: agent name must be alphanumeric/hyphens only
31+
if ! printf '%s' "${AGENT_INPUT}" | grep -qE '^[a-z][a-z0-9-]*$'; then
32+
echo "::error::Invalid agent name: ${AGENT_INPUT}"
33+
exit 1
34+
fi
35+
echo "agents=$(jq -cn --arg a "${AGENT_INPUT}" '[$a]')" >> "$GITHUB_OUTPUT"
36+
else
37+
echo "agents=$(jq -c 'keys' packer/agents.json)" >> "$GITHUB_OUTPUT"
38+
fi
39+
40+
build:
41+
needs: matrix
42+
runs-on: ubuntu-latest
43+
strategy:
44+
max-parallel: 3
45+
fail-fast: false
46+
matrix:
47+
agent: ${{ fromJson(needs.matrix.outputs.agents) }}
48+
steps:
49+
- uses: actions/checkout@v4
50+
51+
- name: Install Bun
52+
uses: oven-sh/setup-bun@v2
53+
with:
54+
bun-version: latest
55+
56+
- name: Install agent under /root
57+
env:
58+
AGENT_NAME: ${{ matrix.agent }}
59+
run: |
60+
set -eo pipefail
61+
62+
# Validate agent exists in agents.json (prevents path traversal / injection)
63+
if ! jq -e --arg a "${AGENT_NAME}" 'has($a)' packer/agents.json > /dev/null; then
64+
echo "::error::Unknown agent: ${AGENT_NAME}"
65+
exit 1
66+
fi
67+
68+
# Read the agent's tier from packer/agents.json
69+
TIER=$(jq -r --arg a "${AGENT_NAME}" '.[$a].tier' packer/agents.json)
70+
echo "==> Agent: ${AGENT_NAME}, Tier: ${TIER}"
71+
72+
# Run tier script (sets up node/bun/etc. under /root)
73+
if [ -f "packer/scripts/tier-${TIER}.sh" ]; then
74+
echo "==> Running tier script: tier-${TIER}.sh"
75+
sudo HOME=/root bash "packer/scripts/tier-${TIER}.sh"
76+
fi
77+
78+
# TRUST BOUNDARY: packer/agents.json is version-controlled and requires
79+
# PR review to modify. Install commands are executed via bash -c, so any
80+
# change to agents.json MUST be reviewed carefully for command safety.
81+
#
82+
# Security layers:
83+
# 1. agents.json changes require PR review (branch protection)
84+
# 2. curl/wget targets validated against domain allowlist
85+
# 3. Suspicious command patterns rejected via blocklist
86+
# 4. Runs in ephemeral GitHub Actions container (destroyed after each run)
87+
echo "==> Installing agent..."
88+
89+
# Allowed domains for curl/wget downloads (official agent vendor domains)
90+
ALLOWED_DOMAINS="claude.ai|opencode.ai|raw.githubusercontent.com|registry.npmjs.org|crates.io|github.com"
91+
92+
CMD_COUNT=$(jq -r --arg a "${AGENT_NAME}" '.[$a].install | length' packer/agents.json)
93+
i=0
94+
while [ "$i" -lt "$CMD_COUNT" ]; do
95+
cmd=$(jq -r --arg a "${AGENT_NAME}" --argjson i "$i" '.[$a].install[$i]' packer/agents.json)
96+
97+
# Safety layer 1: reject suspicious command patterns
98+
if printf '%s' "${cmd}" | grep -qE '(mktemp|eval |base64 -d|/dev/tcp|nc -[elp]|python[23]? -c|perl -e|ruby -e|\bdd\b|>[[:space:]]*/dev/|rm -rf)'; then
99+
echo "::error::Suspicious install command rejected: ${cmd}"
100+
exit 1
101+
fi
102+
103+
# Safety layer 2: validate curl/wget URLs against domain allowlist
104+
if printf '%s' "${cmd}" | grep -qE '(curl|wget)'; then
105+
urls=$(printf '%s' "${cmd}" | grep -oE 'https?://[^[:space:]"|'\'']+' || true)
106+
for url in ${urls}; do
107+
domain=$(printf '%s' "${url}" | sed -E 's|^https?://([^/]+).*|\1|')
108+
if ! printf '%s' "${domain}" | grep -qE "^(${ALLOWED_DOMAINS})$"; then
109+
echo "::error::curl/wget to unapproved domain '${domain}' in: ${cmd}"
110+
exit 1
111+
fi
112+
done
113+
fi
114+
115+
echo "==> Running: ${cmd}"
116+
sudo HOME=/root bash -c "${cmd}"
117+
i=$((i + 1))
118+
done
119+
120+
- name: Capture agent files into tarball
121+
env:
122+
AGENT_NAME: ${{ matrix.agent }}
123+
run: |
124+
set -eo pipefail
125+
sudo bash packer/scripts/capture-agent.sh "${AGENT_NAME}"
126+
127+
- name: Create or update GitHub Release
128+
env:
129+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
130+
AGENT_NAME: ${{ matrix.agent }}
131+
run: |
132+
set -eo pipefail
133+
TAG="agent-${AGENT_NAME}-latest"
134+
DATE=$(date -u +%Y%m%d)
135+
TARBALL="spawn-agent-${AGENT_NAME}-x86_64-${DATE}.tar.gz"
136+
137+
# Move tarball to expected name
138+
mv "/tmp/spawn-agent-${AGENT_NAME}.tar.gz" "${TARBALL}"
139+
140+
# Delete existing release if present (rolling release)
141+
gh release delete "${TAG}" --yes 2>/dev/null || true
142+
143+
# Create fresh release with the tarball
144+
gh release create "${TAG}" "${TARBALL}" \
145+
--title "Agent tarball: ${AGENT_NAME} (${DATE})" \
146+
--notes "Pre-built tarball for \`${AGENT_NAME}\` agent. Auto-generated nightly." \
147+
--prerelease

packages/cli/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@openrouter/spawn",
3-
"version": "0.14.4",
3+
"version": "0.15.0",
44
"type": "module",
55
"bin": {
66
"spawn": "cli.js"
Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
/**
2+
* agent-tarball.test.ts — Tests for pre-built tarball install logic.
3+
*
4+
* Verifies that tryTarballInstall:
5+
* - Queries the correct GitHub Release tag
6+
* - Runs curl | tar on the remote via runner.runServer
7+
* - Returns false when the release doesn't exist
8+
* - Returns false when runner.runServer throws
9+
* - Rejects URLs with shell injection characters
10+
*/
11+
12+
import { afterEach, beforeEach, describe, expect, it, mock, spyOn } from "bun:test";
13+
14+
// Suppress stderr (logStep/logWarn) with a spy in beforeEach.
15+
16+
const { tryTarballInstall } = await import("../shared/agent-tarball");
17+
18+
// ── Helpers ──────────────────────────────────────────────────────────────
19+
20+
function createMockRunner() {
21+
return {
22+
runServer: mock(() => Promise.resolve()),
23+
uploadFile: mock(() => Promise.resolve()),
24+
};
25+
}
26+
27+
const RELEASE_PAYLOAD = {
28+
assets: [
29+
{
30+
name: "spawn-agent-openclaw-x86_64-20260305.tar.gz",
31+
browser_download_url:
32+
"https://github.com/OpenRouterTeam/spawn/releases/download/agent-openclaw-latest/spawn-agent-openclaw-x86_64-20260305.tar.gz",
33+
},
34+
],
35+
};
36+
37+
// ── Tests ────────────────────────────────────────────────────────────────
38+
39+
describe("tryTarballInstall", () => {
40+
let stderrSpy: ReturnType<typeof spyOn>;
41+
42+
beforeEach(() => {
43+
stderrSpy = spyOn(process.stderr, "write").mockImplementation(() => true);
44+
});
45+
46+
afterEach(() => {
47+
stderrSpy.mockRestore();
48+
});
49+
50+
/** Create a mock fetch that returns the given response. */
51+
function mockFetch(response: Response): typeof fetch {
52+
return mock(async () => response);
53+
}
54+
55+
/** Create a mock fetch that captures the URL and returns the given response. */
56+
function mockFetchCapture(response: Response): {
57+
fetchFn: typeof fetch;
58+
getUrl: () => string;
59+
} {
60+
let url = "";
61+
const fetchFn: typeof fetch = mock(async (input: string | URL | Request) => {
62+
url = String(input);
63+
return response;
64+
});
65+
return {
66+
fetchFn,
67+
getUrl: () => url,
68+
};
69+
}
70+
71+
it("queries correct GitHub Release tag", async () => {
72+
const { fetchFn, getUrl } = mockFetchCapture(new Response(JSON.stringify(RELEASE_PAYLOAD)));
73+
const runner = createMockRunner();
74+
75+
await tryTarballInstall(runner, "openclaw", fetchFn);
76+
77+
expect(getUrl()).toContain("/releases/tags/agent-openclaw-latest");
78+
});
79+
80+
it("runs curl | tar xz -C / on the remote VM", async () => {
81+
const fetchFn = mockFetch(new Response(JSON.stringify(RELEASE_PAYLOAD)));
82+
const runner = createMockRunner();
83+
84+
const result = await tryTarballInstall(runner, "openclaw", fetchFn);
85+
86+
expect(result).toBe(true);
87+
expect(runner.runServer).toHaveBeenCalledTimes(1);
88+
const cmd = String(runner.runServer.mock.calls[0][0]);
89+
expect(cmd).toContain("curl -fsSL");
90+
expect(cmd).toContain("tar xz -C /");
91+
expect(cmd).toContain(".spawn-tarball");
92+
});
93+
94+
it("returns false when release does not exist (404)", async () => {
95+
const fetchFn = mockFetch(
96+
new Response("Not Found", {
97+
status: 404,
98+
}),
99+
);
100+
const runner = createMockRunner();
101+
102+
const result = await tryTarballInstall(runner, "nonexistent", fetchFn);
103+
104+
expect(result).toBe(false);
105+
expect(runner.runServer).not.toHaveBeenCalled();
106+
});
107+
108+
it("returns false when runner.runServer throws", async () => {
109+
const fetchFn = mockFetch(new Response(JSON.stringify(RELEASE_PAYLOAD)));
110+
const runner = createMockRunner();
111+
runner.runServer.mockImplementation(() => Promise.reject(new Error("SSH connection refused")));
112+
113+
const result = await tryTarballInstall(runner, "openclaw", fetchFn);
114+
115+
expect(result).toBe(false);
116+
});
117+
118+
it("returns false when release has no .tar.gz asset", async () => {
119+
const noTarball = {
120+
assets: [
121+
{
122+
name: "README.md",
123+
browser_download_url: "https://example.com",
124+
},
125+
],
126+
};
127+
const fetchFn = mockFetch(new Response(JSON.stringify(noTarball)));
128+
const runner = createMockRunner();
129+
130+
const result = await tryTarballInstall(runner, "openclaw", fetchFn);
131+
132+
expect(result).toBe(false);
133+
expect(runner.runServer).not.toHaveBeenCalled();
134+
});
135+
136+
it("returns false when release response has unexpected format", async () => {
137+
const fetchFn = mockFetch(
138+
new Response(
139+
JSON.stringify({
140+
unexpected: true,
141+
}),
142+
),
143+
);
144+
const runner = createMockRunner();
145+
146+
const result = await tryTarballInstall(runner, "openclaw", fetchFn);
147+
148+
expect(result).toBe(false);
149+
});
150+
151+
it("returns false when URL contains shell injection characters", async () => {
152+
const malicious = {
153+
assets: [
154+
{
155+
name: "evil.tar.gz",
156+
browser_download_url: "https://github.com/x/y/releases/download/v1/a.tar.gz'; rm -rf / ; echo '",
157+
},
158+
],
159+
};
160+
const fetchFn = mockFetch(new Response(JSON.stringify(malicious)));
161+
const runner = createMockRunner();
162+
163+
const result = await tryTarballInstall(runner, "openclaw", fetchFn);
164+
165+
expect(result).toBe(false);
166+
expect(runner.runServer).not.toHaveBeenCalled();
167+
});
168+
});

0 commit comments

Comments
 (0)