Skip to content

Commit dcc49aa

Browse files
NathanFlurryclaude
andcommitted
fix: reclassify 52 misclassified tests in roadmap, add cross-validation mitigations
- Move 49 domain tests from FIX-12 to UNSUPPORTED-MODULE - Fix Sign/Verify classifier matching non-crypto tests - Move unsupported-module/api category check before reason-based classifiers - Add 3 additional cross-validation mitigations (crypto test vectors, error snapshots, host-side assert) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent f8025e1 commit dcc49aa

2 files changed

Lines changed: 869 additions & 914 deletions

File tree

CLAUDE.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,13 @@
3333

3434
- conformance tests live in `packages/secure-exec/tests/node-conformance/` — they are vendored upstream Node.js v22.14.0 test/parallel/ tests run through the sandbox
3535
- `docs-internal/nodejs-compat-roadmap.md` tracks every non-passing test with its fix category and resolution
36-
- when implementing bridge/polyfill features where both sides go through our code (e.g., loopback HTTP server + client), prevent overfitting with these three mitigations:
36+
- when implementing bridge/polyfill features where both sides go through our code (e.g., loopback HTTP server + client), prevent overfitting:
3737
- **wire-level snapshot tests**: capture raw protocol bytes and compare against known-good captures from real Node.js
3838
- **project-matrix cross-validation**: add a project-matrix fixture (`tests/projects/`) using a real npm package that exercises the feature — the matrix compares sandbox output to host Node.js
3939
- **real-server control tests**: for network features, maintain tests that hit real external endpoints (not loopback) to validate the client independently of the server
40+
- **known-test-vector validation**: for crypto, validate against NIST/RFC test vectors — not just round-trip verification
41+
- **error object snapshot testing**: for ERR_* codes, snapshot-test full error objects (code, message, constructor) against Node.js — not just check `.code` exists
42+
- **host-side assertion verification**: periodically run assert-heavy conformance tests through host Node.js to verify the assert polyfill isn't masking failures
4043
- never inflate conformance numbers — if a test self-skips (exits 0 without testing anything), mark it `vacuous-skip` in expectations.json, not as a real pass
4144
- every entry in `expectations.json` must have a specific, verifiable reason — no vague "fails in sandbox" reasons
4245

0 commit comments

Comments
 (0)