feat(billing): stop ingestion once an org is 3 days overdue and never paid

[Makisuo]

What & why

Hard-stops OTLP + Cloudflare Logpush ingestion for orgs that signed up, never paid an invoice, and are now 3+ days past due. Today the gateway's only billing enforcement is the per-request Autumn /v1/check (gated by AUTUMN_ENFORCE_LIMITS), which blocks hard-capped/zero-balance or no-subscription orgs but ignores past_due entirely.

Verified against Autumn's docs, two facts shaped the design:

• check() does not block on past_due by default — Autumn: "If you'd like to block feature access when a subscription is past_due, please contact us." The native toggle is also all-or-nothing, so it can't express "3 days and never paid". Custom logic is required.
• Autumn recommends replicating billing state via webhooks (billing.updated, Svix-signed), not polling.

So this is the Autumn-idiomatic shape: webhook replicates the overdue clock → a small time-based reconcile cron promotes/clears → the gateway reads a DB flag and 402s.

How it works

1. Webhook (billing-webhook.http.ts) — Svix-verified POST /api/billing/autumn/webhook (Web Crypto HMAC, mirrors the GitHub webhook verifier — no new dependency). On billing.updated it re-derives the org's pastDue from Autumn and upserts/clears its overdue row.
2. Reconcile cron (BillingSuspensionService + pure BillingSuspensionPolicy) — daily, scoped to the overdue set only (not a full-org scan; the "+3 days" transition is a timer no webhook fires for). Promotes overdue-≥3d + never-paid (no status:"paid" invoice) → suspended_at, clears rows once the org pays. Wired via event.cron dispatch in worker.ts + alchemy/wrangler crons.
3. Gateway (main.rs) — ingest_suspended threaded through OrgRouting (refreshes on the 1s routing TTL → fast un-suspension; fail-open on DB error), a LEFT JOIN org_billing_suspensions in the key/connector/routing queries, and a 402 "billing_suspended" before the entitlement gate on both OTLP and Logpush paths.

Schema

New org_billing_suspensions table (incremental migration 0003). overdue_since is set by the webhook; suspended_at IS NOT NULL is the gateway's enforcement flag.

Decisions baked in

• Never paid = no invoice ever has status === "paid" (targets free/trial signups that never converted; lapsed paying customers are left to normal dunning).
• Overdue clock = past-due anchor + 3 days. Autumn exposes no literal due_date, so the anchor is when the webhook first observes pastDue.
• Enforcement = HTTP 402, always on (no gateway env flag — the table is only ever written where Autumn is configured).

Reviewer notes

• Also extracted the shared Autumn-call helper into AutumnClient.ts and added invoices to the domain BillingCustomer (decoded only when expanded).
• Deploy prerequisites: set AUTUMN_WEBHOOK_SECRET and register the billing.updated webhook in the Autumn dashboard (without it the receiver 401s and only the daily cron drives state). The webhook payload's customer_id location is parsed defensively — worth confirming against a real billing.updated delivery in a preview stage.

Testing

• New: policy boundaries, PGlite reconcile (insert/promote/clear), Svix signature accept/tamper/wrong-secret, customer_id extraction, gateway suspension propagation.
• Full @maple/api suite 654/654, ingest cargo tests 39/39, bun typecheck 24/24 packages; migration applies cleanly on a fresh DB via the real drizzle migrator.

🤖 Generated with Claude Code

[pullfrog]

Your LLM provider API key was rejected. Rotate the key in your provider dashboard, then update the matching GitHub Actions secret.

Update repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot ｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[github-actions]

Ingest Rust Test + Benchmark Results

Commit: 993116c89f00954374a70cc5498274404329fe5b

Load Benchmark — tinybird mode, median of 3 run(s) vs main

Metric	main (median)	PR (median)	Delta
Requests/sec	609.77	1109.92	+82.0% better
Rows/sec	6097.73	11099.18	+82.0% better
p50 latency	77.80 ms	38.97 ms	-49.9% better
p95 latency	305.15 ms	161.01 ms	-47.2% better
p99 latency	504.06 ms	280.42 ms	-44.4% better
Export catch-up	0.026 s	0.027 s	+1.7% worse
Max RSS	106.54 MiB	104.78 MiB	-1.7% better
Failures	0	0	same

Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.

PR load benchmark JSON (per-iteration)

[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 25,
    "duration_seconds": 1.5710919049999998,
    "export_catchup_seconds": 0.026657121,
    "request_rps": 1273.0000031411275,
    "row_rps": 12730.000031411277,
    "p50_ms": 38.968,
    "p95_ms": 110.595,
    "p99_ms": 129.699,
    "max_rss_mb": 103.73828125,
    "max_cpu_percent": 37.5,
    "avg_cpu_percent": 30.299999999999997
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 25,
    "duration_seconds": 3.785971262,
    "export_catchup_seconds": 0.025215308,
    "request_rps": 528.2660278154008,
    "row_rps": 5282.660278154008,
    "p50_ms": 54.978,
    "p95_ms": 328.75,
    "p99_ms": 1157.642,
    "max_rss_mb": 105.78125,
    "max_cpu_percent": 26.7,
    "avg_cpu_percent": 18.9
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 28,
    "duration_seconds": 1.801934901,
    "export_catchup_seconds": 0.026730813,
    "request_rps": 1109.918010295534,
    "row_rps": 11099.180102955339,
    "p50_ms": 38.141,
    "p95_ms": 161.01,
    "p99_ms": 280.419,
    "max_rss_mb": 104.77734375,
    "max_cpu_percent": 42.8,
    "avg_cpu_percent": 32.025
  }
]

main load benchmark JSON (per-iteration)

[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 27,
    "duration_seconds": 4.370867142,
    "export_catchup_seconds": 0.026205154,
    "request_rps": 457.57510695803256,
    "row_rps": 4575.751069580326,
    "p50_ms": 97.947,
    "p95_ms": 436.35,
    "p99_ms": 491.501,
    "max_rss_mb": 104.60546875,
    "max_cpu_percent": 16.6,
    "avg_cpu_percent": 10.633333333333333
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 26,
    "duration_seconds": 2.399513216,
    "export_catchup_seconds": 0.025813347,
    "request_rps": 833.5023898447242,
    "row_rps": 8335.023898447243,
    "p50_ms": 47.667,
    "p95_ms": 305.148,
    "p99_ms": 504.055,
    "max_rss_mb": 108.15625,
    "max_cpu_percent": 30.3,
    "avg_cpu_percent": 24.76
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 27,
    "duration_seconds": 3.279909851,
    "export_catchup_seconds": 0.026371769,
    "request_rps": 609.7728568333142,
    "row_rps": 6097.728568333142,
    "p50_ms": 77.798,
    "p95_ms": 268.808,
    "p99_ms": 516.428,
    "max_rss_mb": 106.5390625,
    "max_cpu_percent": 23.6,
    "avg_cpu_percent": 19.74285714285714
  }
]

WAL-acked microbench (cargo bench --bench ingest_bench)

   Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
    Finished `bench` profile [optimized] target(s) in 32.43s
     Running benches/ingest_bench.rs (target/release/deps/ingest_bench-581d2100de893627)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench:     2820887 ns/iter (+/- 1804311)
test ingest_accept/traces_10_spans_wal_ack ... bench:     1196406 ns/iter (+/- 1046054)

cargo test

test telemetry::tests::metric_encoder_matches_all_tinybird_datasource_shapes ... ok
test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::metrics_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::migrate_legacy_shard_relocates_frames_into_lanes ... ok
test telemetry::tests::pipeline_can_start_for_clickhouse_only_without_tinybird_credentials ... ok
test telemetry::tests::clickhouse_export_drops_passworded_non_https_endpoint_without_sending ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::scraper_contract::scraper_otlp_json_decodes_with_gateway_serde_and_encodes_to_rows ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::wal_partial_drain_advances_cursor_without_truncating ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::wal_truncates_after_full_drain_allowing_further_appends ... ok
test telemetry::tests::pipeline_exports_ready_org_to_clickhouse_without_tinybird_calls ... ok
test telemetry::tests::slow_clickhouse_lane_does_not_block_cosharded_tinybird_org ... ok
test telemetry::tests::clickhouse_breaker_sheds_after_threshold_failures ... ok

test result: ok. 33 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s

     Running unittests src/bin/load_test.rs (target/debug/deps/load_test-661a0aa1eb3f6d6d)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/maple_ingest-c33bf80c577edb95)

running 39 tests
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::hash_is_deterministic ... ok
test tests::rejection_span_status_is_error_only_for_5xx ... ok
test tests::resolve_ingest_key_is_not_suspended_by_default ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::resolve_ingest_key_propagates_ingest_suspended_flag ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok

test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.21s

   Doc-tests maple_ingest

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s