Skip to content

Commit b5ea9eb

Browse files
NathanFlurryclaude
andcommitted
feat: US-058 - Record context snapshot benchmark baseline (before)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8d8cac5 commit b5ea9eb

3 files changed

Lines changed: 274 additions & 0 deletions

File tree

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
{
2+
"description": "Benchmark baseline BEFORE context snapshot optimization (isolate-only snapshot)",
3+
"date": "2026-03-19",
4+
"commit": "8d8cac5d2cd37a0a9c5d63756a621a1196d8c744",
5+
"hardware": {
6+
"cpu": "12th Gen Intel(R) Core(TM) i7-12700KF",
7+
"cores": 20,
8+
"ram": "62 GB",
9+
"node": "v24.13.0",
10+
"os": "Linux 6.1.0-41-amd64",
11+
"arch": "x64"
12+
},
13+
"benchmark": "quick-bench.ts (5 sequential create+run cycles per iteration)",
14+
"iterations": 5,
15+
"runs": [
16+
{
17+
"iteration": 1,
18+
"cold_ms": [90.9, 48.8, 46.3, 48.1, 48.4],
19+
"warm_ms": [13.5, 14.2, 13.3, 14.0, 14.4]
20+
},
21+
{
22+
"iteration": 2,
23+
"cold_ms": [94.8, 53.1, 50.2, 47.0, 44.1],
24+
"warm_ms": [15.5, 12.7, 14.1, 12.5, 15.5]
25+
},
26+
{
27+
"iteration": 3,
28+
"cold_ms": [89.6, 46.5, 48.3, 49.1, 49.9],
29+
"warm_ms": [14.7, 13.2, 13.7, 12.8, 13.1]
30+
},
31+
{
32+
"iteration": 4,
33+
"cold_ms": [126.0, 47.0, 46.6, 46.6, 44.6],
34+
"warm_ms": [14.1, 13.6, 13.9, 12.9, 12.7]
35+
},
36+
{
37+
"iteration": 5,
38+
"cold_ms": [98.1, 47.4, 50.1, 51.3, 51.1],
39+
"warm_ms": [13.9, 14.9, 15.6, 13.5, 13.4]
40+
}
41+
],
42+
"summary": {
43+
"warm": {
44+
"mean_ms": 13.75,
45+
"p50_ms": 13.7,
46+
"min_ms": 12.5,
47+
"max_ms": 15.6,
48+
"all_samples": [13.5, 14.2, 13.3, 14.0, 14.4, 15.5, 12.7, 14.1, 12.5, 15.5, 14.7, 13.2, 13.7, 12.8, 13.1, 14.1, 13.6, 13.9, 12.9, 12.7, 13.9, 14.9, 15.6, 13.5, 13.4]
49+
},
50+
"cold_process_start": {
51+
"mean_ms": 99.88,
52+
"samples": [90.9, 94.8, 89.6, 126.0, 98.1]
53+
},
54+
"cold_steady_state": {
55+
"mean_ms": 48.23,
56+
"p50_ms": 48.2,
57+
"min_ms": 44.1,
58+
"max_ms": 53.1,
59+
"all_samples": [48.8, 46.3, 48.1, 48.4, 53.1, 50.2, 47.0, 44.1, 46.5, 48.3, 49.1, 49.9, 47.0, 46.6, 46.6, 44.6, 47.4, 50.1, 51.3, 51.1]
60+
}
61+
},
62+
"notes": "Warm start (~13.75ms mean) represents per-session cost with isolate-only snapshot. This is the baseline for context snapshot optimization which targets <6ms warm start. Process hangs on cleanup after benchmark completes (exit code 124 from timeout), but all 5 runs complete successfully before hang."
63+
}

scripts/ralph/prd.json

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -970,6 +970,201 @@
970970
"priority": 57,
971971
"passes": true,
972972
"notes": "Spec: docs-internal/specs/v8-startup-snapshot.md. Rust: 3 new test parts (WASM blocked after restore, session isolation via fresh contexts, external references survive restore with bridge fn registration). TypeScript: 7 integration tests (WASM blocked, session state isolation, sync/async bridge dispatch on restored isolate, WarmSnapshot cache hit, NO_SNAPSHOT_WARMUP=1 lazy fallback, different bridge code variants get separate entries)."
973+
},
974+
{
975+
"id": "US-058",
976+
"title": "Record context snapshot benchmark baseline (before)",
977+
"description": "As a developer, I need to record benchmark baseline numbers before any context snapshot code changes, so we can measure the improvement afterwards.",
978+
"acceptanceCriteria": [
979+
"Run packages/secure-exec/benchmarks/quick-bench.ts and record cold/warm start times",
980+
"Save results to packages/secure-exec/benchmarks/results/context_snapshot_before.json with hardware info and commit hash",
981+
"Record at least 5 runs to get stable averages",
982+
"Document current warm start baseline (~13ms expected from isolate-only snapshot)",
983+
"No code changes to runtime, bridge, or snapshot code in this story",
984+
"Typecheck passes"
985+
],
986+
"priority": 58,
987+
"passes": true,
988+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 8. Baseline recorded: warm mean=13.75ms, p50=13.7ms; cold steady-state mean=48.23ms. 5 iterations x 5 runs each. Results in packages/secure-exec/benchmarks/results/context_snapshot_before.json."
989+
},
990+
{
991+
"id": "US-059",
992+
"title": "Fix setupFsFacade to use getter-based delegation",
993+
"description": "As a developer, I need setupFsFacade to use getter-based property delegation instead of capturing bridge function references at setup time, so snapshot-restored contexts pick up replaced globals.",
994+
"acceptanceCriteria": [
995+
"setup-fs-facade.ts replaces direct property assignment (_fs.readFile = globalThis._fsReadFile) with Object.defineProperties using getters",
996+
"Each _fs property getter resolves globalThis._fsXxx at call time, not setup time",
997+
"All fs bridge function names are covered (readFile, writeFile, readFileBinary, writeFileBinary, readDir, mkdir, rmdir, exists, stat, unlink, rename, chmod, chown, link, symlink, readlink, lstat, truncate, utimes)",
998+
"All existing fs tests pass unchanged — getter delegation is transparent to callers",
999+
"Typecheck passes",
1000+
"Tests pass"
1001+
],
1002+
"priority": 59,
1003+
"passes": false,
1004+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 1. Prerequisite for context snapshot."
1005+
},
1006+
{
1007+
"id": "US-060",
1008+
"title": "Defer config-dependent bridge setup to post-restore init",
1009+
"description": "As a developer, I need config-dependent values (payload limits, timing mitigation) to be read lazily from globals at call time instead of captured at bridge IIFE setup time, enabling context snapshot reuse across sessions with different configs.",
1010+
"acceptanceCriteria": [
1011+
"__jsonPayloadLimitBytes reads from globalThis.__runtimeJsonPayloadLimitBytes at call time instead of capturing at setup",
1012+
"__payloadLimitErrorCode reads from globalThis.__runtimePayloadLimitErrorCode at call time",
1013+
"Timing mitigation freeze is applied via globalThis.__runtimeApplyConfig(config) called post-restore, not baked into bridge IIFE",
1014+
"globalThis.__runtimeApplyConfig function exists after bridge IIFE runs, accepts config object with timingMitigation, frozenTimeMs, payloadLimitBytes fields",
1015+
"__runtimeApplyConfig deletes itself from globalThis after execution",
1016+
"All existing timing mitigation tests pass",
1017+
"All existing payload limit tests pass",
1018+
"Typecheck passes",
1019+
"Tests pass"
1020+
],
1021+
"priority": 60,
1022+
"passes": false,
1023+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 2. Prerequisite for context snapshot."
1024+
},
1025+
{
1026+
"id": "US-061",
1027+
"title": "Split composeBridgeCode into static and post-restore parts",
1028+
"description": "As a developer, I need composeBridgeCode() split into a static bridge IIFE (identical across all sessions) and a short post-restore config script, so one snapshot serves all session variants.",
1029+
"acceptanceCriteria": [
1030+
"New composeStaticBridgeCode() returns the bridge IIFE without any per-session config literals (timingMitigation, frozenTimeMs, maxTimers, maxHandles not embedded)",
1031+
"New composePostRestoreScript(config) returns a short script that calls __runtimeApplyConfig({...}) and loads polyfills",
1032+
"composeStaticBridgeCode() output is byte-for-byte identical regardless of session options",
1033+
"Existing composeBridgeCode() refactored to use the new functions (or replaced entirely)",
1034+
"Post-restore script handles timing mitigation, payload limits, CJS globals, and polyfill loading",
1035+
"Typecheck passes",
1036+
"Tests pass"
1037+
],
1038+
"priority": 61,
1039+
"passes": false,
1040+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 3."
1041+
},
1042+
{
1043+
"id": "US-062",
1044+
"title": "Add stub bridge context and registration for snapshot creation",
1045+
"description": "As a developer, I need BridgeCallContext::stub() and register_stub_bridge_fns() so snapshot creation can register all 38 bridge functions as no-op stubs that exist for the IIFE to reference during setup.",
1046+
"acceptanceCriteria": [
1047+
"BridgeCallContext::stub() creates a no-op context that panics if sync_call or async_send is called",
1048+
"register_stub_bridge_fns(scope, sync_fns, async_fns) registers all bridge functions with stub External data on the global",
1049+
"Stub functions use the same sync_bridge_callback/async_bridge_callback (required for ExternalReferences in snapshot)",
1050+
"Bridge IIFE can execute against stub functions without errors (verifies setup-time code doesn't call bridge functions)",
1051+
"cargo test passes",
1052+
"Typecheck passes"
1053+
],
1054+
"priority": 62,
1055+
"passes": false,
1056+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 4."
1057+
},
1058+
{
1059+
"id": "US-063",
1060+
"title": "Implement context snapshot creation with fully-initialized bridge",
1061+
"description": "As a developer, I need create_snapshot() to register stub bridge functions, inject default config, run the static bridge IIFE, and snapshot the context with all bridge infrastructure set up.",
1062+
"acceptanceCriteria": [
1063+
"create_snapshot() registers stub bridge functions on the context global before running bridge IIFE",
1064+
"Default config globals are injected (initialCwd, processConfig, osConfig, maxTimers, maxHandles, customGlobalPolicy)",
1065+
"Static bridge IIFE is compiled and executed in the context",
1066+
"set_default_context(context) is called before create_blob — the context is part of the snapshot",
1067+
"Restored context has all bridge infrastructure: require(), console, fs facade, module system, custom globals",
1068+
"Snapshot blob is cached in SnapshotCache keyed by bridge code hash",
1069+
"cargo test passes",
1070+
"Typecheck passes"
1071+
],
1072+
"priority": 63,
1073+
"passes": false,
1074+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 5."
1075+
},
1076+
{
1077+
"id": "US-064",
1078+
"title": "Implement context restore with bridge function replacement",
1079+
"description": "As a developer, I need the session thread to restore the snapshot context, replace stub bridge functions with real session-local ones, and run the post-restore config script instead of recompiling the bridge IIFE.",
1080+
"acceptanceCriteria": [
1081+
"Session thread gets default context from snapshot via scope.get_current_context() instead of creating fresh context",
1082+
"replace_bridge_fns(scope, ctx, buffers, sync_fns, async_fns) overwrites all 38 stub globals with real session-local bridge functions",
1083+
"Per-session config (_processConfig, _osConfig) is injected via inject_globals_from_payload",
1084+
"Post-restore config script is executed to apply timing mitigation and load polyfills",
1085+
"No bridge IIFE compilation or execution on restore path",
1086+
"disable_wasm() is called on restored isolate",
1087+
"All existing tests pass — behavioral parity with fresh-context path",
1088+
"cargo test passes",
1089+
"Typecheck passes",
1090+
"Tests pass"
1091+
],
1092+
"priority": 64,
1093+
"passes": false,
1094+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 6."
1095+
},
1096+
{
1097+
"id": "US-065",
1098+
"title": "Wire post-restore init script through IPC",
1099+
"description": "As a developer, I need the post-restore script sent from the host and executed on the Rust side after context restore, handling config application, polyfill loading, and mutable state reset.",
1100+
"acceptanceCriteria": [
1101+
"Execute message includes a post_restore_script field (or it is composed on the Rust side from config)",
1102+
"Post-restore script calls __runtimeApplyConfig({timingMitigation, frozenTimeMs, payloadLimitBytes})",
1103+
"Post-restore script loads polyfills via _loadPolyfill() calls",
1104+
"Post-restore script resets mutable state: require.cache cleared, process.exitCode reset, _processStartTime set to Date.now()",
1105+
"Post-restore script runs BEFORE user code but AFTER bridge function replacement",
1106+
"Typecheck passes",
1107+
"Tests pass"
1108+
],
1109+
"priority": 65,
1110+
"passes": false,
1111+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 7."
1112+
},
1113+
{
1114+
"id": "US-066",
1115+
"title": "Add context snapshot tests",
1116+
"description": "As a developer, I need tests covering all context snapshot behaviors: getter facade, config deferral, bridge replacement, timing mitigation, polyfill loading, and full round-trip.",
1117+
"acceptanceCriteria": [
1118+
"Test: _fs.readFile resolves to the current global, not a stale reference from snapshot",
1119+
"Test: __runtimeApplyConfig correctly applies timing freeze, payload limits",
1120+
"Test: restored context has working require(), console, fs, module system",
1121+
"Test: replacing stub bridge functions on restored context correctly dispatches to Rust callbacks (sync and async)",
1122+
"Test: timing mitigation freeze applied via post-restore script correctly freezes Date.now and removes SharedArrayBuffer",
1123+
"Test: polyfills loaded via post-restore script work correctly",
1124+
"Test: exec() and run() produce correct results on snapshot-restored context",
1125+
"Test: all existing test suites pass with no regressions",
1126+
"cargo test passes",
1127+
"Typecheck passes",
1128+
"Tests pass"
1129+
],
1130+
"priority": 66,
1131+
"passes": false,
1132+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 9."
1133+
},
1134+
{
1135+
"id": "US-067",
1136+
"title": "Verify context snapshot benchmark improvement (after)",
1137+
"description": "As a developer, I need to run benchmarks after context snapshot implementation and compare against the baseline to verify the expected performance improvement.",
1138+
"acceptanceCriteria": [
1139+
"Run packages/secure-exec/benchmarks/quick-bench.ts and record cold/warm start times",
1140+
"Save results to packages/secure-exec/benchmarks/results/context_snapshot_after.json",
1141+
"Warm start (per-session cost) must be < 6ms (down from ~13ms baseline)",
1142+
"Cold start must not regress (should stay ~47-67ms steady state or improve)",
1143+
"Create packages/secure-exec/benchmarks/results/context_snapshot_comparison.md with before/after table",
1144+
"If warm start improvement < 40%, add per-phase timing instrumentation and investigate",
1145+
"All existing test suites still pass",
1146+
"Typecheck passes"
1147+
],
1148+
"priority": 67,
1149+
"passes": false,
1150+
"notes": "Spec: docs-internal/specs/v8-context-snapshot.md Phase 10."
1151+
},
1152+
{
1153+
"id": "US-068",
1154+
"title": "Research additional V8 runtime performance optimizations",
1155+
"description": "As a developer, I need to research and document further optimization opportunities beyond context snapshots, including per-session sockets, mmap/shared-memory IPC, and other latency reduction techniques.",
1156+
"acceptanceCriteria": [
1157+
"Profile the warm start path with per-phase timing (context restore, bridge fn replacement, config injection, user code compile+run, IPC round-trips) and document where time is spent",
1158+
"Evaluate per-session UDS sockets: measure head-of-line blocking impact with large payloads under concurrent sessions, estimate FD cost",
1159+
"Evaluate mmap/shared-memory ring buffer IPC: estimate latency savings vs implementation complexity, document lock-free sync requirements and crash cleanup concerns",
1160+
"Evaluate V8 code caching for user code (not just bridge): measure compilation time for typical user scripts, estimate cache hit rate",
1161+
"Document findings in docs-internal/specs/v8-perf-research.md with recommendation priority (quick wins vs high-effort)",
1162+
"Update docs-internal/todo.md with any new actionable items discovered",
1163+
"Typecheck passes"
1164+
],
1165+
"priority": 68,
1166+
"passes": false,
1167+
"notes": "Research story — no code changes required beyond documentation and profiling scripts."
9731168
}
9741169
]
9751170
}

scripts/ralph/progress.txt

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,3 +381,19 @@ Started: Wed Mar 18 06:52:02 PM PDT 2026
381381
- Use `import("@secure-exec/v8").V8Runtime` inline type import to avoid adding a top-level import dependency in interfaces
382382
- Pre-existing OOM test timeout in `crash-isolation.test.ts` — unrelated to this story
383383
---
384+
385+
## 2026-03-19 - US-058
386+
- Recorded context snapshot benchmark baseline (before any context snapshot code changes)
387+
- Ran `packages/secure-exec/benchmarks/quick-bench.ts` 5 iterations (5 runs each = 25 warm samples, 25 cold samples)
388+
- Saved results to `packages/secure-exec/benchmarks/results/context_snapshot_before.json`
389+
- Baseline results:
390+
- Warm start (per-session cost): mean=13.75ms, p50=13.7ms, min=12.5ms, max=15.6ms
391+
- Cold start (process cold): mean=99.88ms (first run of each iteration)
392+
- Cold start (steady state): mean=48.23ms, p50=48.2ms
393+
- Hardware: i7-12700KF, 20 cores, 62GB RAM, Node v24.13.0, Linux x64
394+
- Files created: `packages/secure-exec/benchmarks/results/context_snapshot_before.json`
395+
- **Learnings for future iterations:**
396+
- `quick-bench.ts` process hangs on cleanup after completing all runs — use `timeout 60` wrapper to avoid blocking
397+
- Run 1 of each iteration is the true process cold-start (~90-126ms); runs 2-5 are steady state (~44-53ms)
398+
- Warm start baseline is consistent at ~13-14ms — this is the per-session cost dominated by bridge IIFE compilation+execution
399+
---

0 commit comments

Comments
 (0)