fix(alerting): stop warehouse-timeout fan-out storm in error/anomaly crons

[Makisuo]

What & why

On 2026-07-01 (03:30–05:10 UTC) the alerting worker fired a burst of ~3,300 WarehouseQueryError: "The operation was aborted due to timeout" from its error_tick / anomaly_tick crons. It was not a Tinybird outage (maple-api warehouse queries were fine, alerting cron volume was flat) — it was a self-inflicted fan-out storm.

Root cause: active-org discovery failed open. A brief warehouse slowdown made the unbounded cross-org discovery query (SELECT OrgId FROM error_events_by_time … GROUP BY OrgId, no OrgId predicate ⇒ no primary-key pruning, no cost profile ⇒ no server-side max_execution_time) time out. On that failure the tick fell back to scanning every known org (hundreds) instead of the ~26 active ones — dumping maximum load exactly when the warehouse was already struggling, which sustained the storm for ~90 min until the blip cleared.

The telemetry fingerprint (per-10-min executeSql spans, distinct orgs scanned):

Time (UTC)	queries	timeouts	distinct orgs	p90
02:30–03:20	~630	0	26	0.3s
03:40–04:20	~38	~28	1	30.2s
04:40	722	483	101	30.2s
05:00	2521	58	252	0.9s
05:10+	~625	0	26	0.4s

The fix (low-risk hardening, reuses existing machinery)

• P0a — bound the trigger: discovery queries now use the existing discovery cost profile (5s server-side) instead of riding the ~30s client abort.
• P0b — fail closed: on discovery failure, reuse the last-known active-org set (edge-cached, 6h TTL) instead of fanning out to all orgs. Cold cache → just BYO + the withState / open-incident floor (so resolution/aging keeps running). The "all" sentinel is removed entirely. Wires EdgeCacheService into ErrorsService (Anomaly already had it).
• P1a — bound per-org scans: errorIssuesScan now carries the list profile (15s) so a scan can't hang the full ~30s.
• P1b — client timeout: each managed query attempt in the executor is bounded at server budget + 5s (or a 30s hard cap), mapped to a non-transient WarehouseQueryError so a queued query fails fast instead of feeding the retry loop. This directly addresses the observed case where a list query with max_execution_time=15 still rode to 30s because Tinybird queued it.

Deferred: P2 (per-tick circuit breaker + overall tick timeout) — follow-up.

Reviewer notes

• Happy-path behavior is unchanged: discovery success still returns the active set (now also cached); only the failure path changed (was "all", now cached/BYO).
• ErrorsService gains an EdgeCacheService dependency — wired in app.ts and the test layer.
• Backend cron/warehouse logic, nothing browser-observable.

Tests

New: fail-closed gating (stateful orgs still scanned, idle orgs skipped), profile assertions (discovery→discovery, scan→list), and a hung-query timeout test (bounded + non-transient).

• query-engine: 648 pass, typecheck clean
• apps/api ErrorsService + anomaly: 85 pass, typecheck clean

🤖 Generated with Claude Code

---

[pullfrog]

Your LLM provider API key was rejected. Rotate the key in your provider dashboard, then update the matching GitHub Actions secret.

Update repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot ｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}