docs: warn against --test_keep_going=false with quarantining

[mintlify]

Adds a warning to the Bazel flaky tests guide explaining that --test_keep_going=false (or --notest_keep_going) shouldn't be combined with quarantining, since Bazel will stop at the first failure and skipped tests won't appear in the BEP report for Trunk to quarantine.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
trunk	🟢 Ready	View Preview	Jun 5, 2026, 4:39 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[samgutentag]

Verification status (2026-06-05): unknown

Could not determine rollout state from available signals. Chaining to verify-docs-against-code for content-accuracy check.

• Flag state: not applicable. This PR documents existing Bazel/BEP behavior (--test_keep_going=false short-circuiting the report), not a flagged feature, so there is no rollout to verify.
• Eng PR: none referenced in the PR body.
• Flag: none.
• Signals: no trunk-io/<repo>#NNN or PR-link refs in the body; no TRUNK-XXX Linear ticket; single-file docs change to flaky-tests/get-started/frameworks/bazel.mdx.

Next: the content claim (Bazel stops at the first failure under --test_keep_going=false, so skipped tests never reach the BEP report and cannot be quarantined) is being verified against Trunk source via verify-docs-against-code. No Linear ticket found in the PR body, so no Linear comment was posted.

[samgutentag]

Code verification (2026-06-05): 2 confirmed / 1 contradicted

Claim	Verdict	Source
--test_keep_going defaults to true; --notest_keep_going is the same as =false; Bazel stops at the first test failure	confirmed	Bazel CLI reference
Skipped tests never reach the BEP report	confirmed	follows from Bazel stopping at the first failure
Consequence: a quarantined test skipped after an earlier failure "will still break your CI"	contradicted	context_quarantine.rs:412-436

The recommendation is right; the stated consequence is inverted. Leaving --test_keep_going at its default when quarantining is correct, and it matches both the customer's and eng's conclusion in the origin thread. But the warning describes the wrong hazard. The real risk is not that CI breaks, it is that CI silently passes: if the first failing test is quarantined, Bazel exits before the remaining tests run, the only failure Trunk sees is already quarantined, so the CLI overrides the exit code to success and an unsafe PR can merge. That is exactly the scenario the customer raised ("CI will pass even though there are other tests that might have failed") and that Matt Matheson confirmed ("in the case that a quarantined test fails, we would mark the test step as passing, which could let a PR merge without actually running all tests").

Suggested replacement callout (Warning, since the failure mode is a silent unsafe merge):

<Warning>
  Leave `--test_keep_going` at its default (`true`) when you quarantine tests. With `--test_keep_going=false` (or `--notest_keep_going`), Bazel stops at the first test failure, so the remaining tests never run and never reach the BEP report. If that first failure is a quarantined test, the only failure Trunk sees is already quarantined, so it marks the run as passing and your PR can merge even though the skipped tests never ran and may have failed.
</Warning>

Heads up before publishing: the eng thread is still open. The last word is Matt's "let me keep thinking about this," and there is an unadopted alternative (splitting flaky vs non-flaky targets into two bazel test runs) that the customer rejected as doubling their test invocations. Tyler Jang did ask to get this documented, so the page should ship, but worth a quick confirm with Tyler or Matt that the false-pass framing is the one they want before merging.

---

Source #1 — quarantine pass/fail is computed only over failures present in the report (contradicted)

File: trunk-io/analytics-cli/cli/src/context_quarantine.rs#L412-L436

quarantine_results.group_is_quarantined =
    quarantine_results.quarantine_results.len() == total_failures;

// use the exit code from the command if the group is not quarantined
// override exit code to be exit_success if the group is quarantined
let exit_code = if total_failures == 0 {
    exit_code
} else if !quarantine_results.group_is_quarantined {
    exit_code
} else if exit_code != EXIT_SUCCESS {
    // All test failures were quarantined, overriding exit code to be exit_success
    EXIT_SUCCESS
} else {
    exit_code
};

Reasoning: total_failures counts only the failures the analytics CLI finds in the uploaded report. When group_is_quarantined is true (every observed failure is quarantined) and the original exit code was non-zero, the CLI overrides it to EXIT_SUCCESS. With --test_keep_going=false, Bazel stops at the first failing test, so if that test is quarantined it is the sole failure in the BEP, group_is_quarantined is trivially true, and the run is forced green. The tests Bazel skipped are never in total_failures at all, so they cannot "break CI." The outcome is a false pass, not a broken build. That inverts the PR's stated consequence.

Source #2 — Bazel flag default and short-circuit behavior (confirmed)

File: Bazel command-line reference, --test_keep_going

Reasoning: --test_keep_going is a Bazel boolean flag that defaults to true. --notest_keep_going is the standard Bazel negation form and is equivalent to --test_keep_going=false. With it off, Bazel stops running tests after the first failure, so later targets never execute and never emit test events into the BEP stream. This is third-party (Bazel) behavior and is stated correctly in the PR. Only the Trunk-side consequence (Source #1) is wrong.

[samgutentag]

Docs PR Verdict — 2026-06-08

Verdict	⚪ unknown
Eng refs	—
LD flag	—
Notes	Mintlify-generated warning PR (Bazel --test_keep_going + quarantine); no eng ref found. Manual review needed to confirm accuracy.

Next sweep: 2026-06-09

---

Generated by Claude Code

[samgutentag]

⚪ Verdict: unknown — June 9, 2026

No feature flag reference or engineering PR link was found in this PR's description. Unable to verify whether the documented feature is live in production.

Action needed: Please add a LaunchDarkly flag name or link to the engineering PR in the PR description so this sweep can auto-verify next run.

Verified by Daily Docs Sweep · June 9, 2026

---

Generated by Claude Code

[samgutentag]

❓ Verdict: unknown — June 10, 2026

Signal	Detail
Eng PR	none referenced
LD flag	none
Linear	none

Could not determine rollout state from available signals. No eng PR, no LD flag, no Linear ticket to verify against.

Verified by Daily Docs Sweep · June 10, 2026

---

Generated by Claude Code

[samgutentag]

❓ Verdict: unknown — June 11, 2026

No LaunchDarkly flag identified. Eng PR dependency on trunk2 not verifiable (private repo). Manual review needed to confirm deployment state.

Verified by Daily Docs Sweep · June 11, 2026

---

Generated by Claude Code