Commit 481bafc
Deprioritize CancelOutstandingWorkerPolls in matching rate limiter (#10010)
## What
Move `CancelOutstandingWorkerPolls` from priority 1 to priority 2 in the
matching service rate limiter, and add priority level 2 to
`APIPrioritiesOrdered`.
## Why
`CancelOutstandingWorkerPolls` fans out to `numPartitions ×
numTaskTypes` RPCs per worker shutdown. Under concurrent worker
shutdowns, this burst competes with Poll and AddTask RPCs (both priority
1) for the matching rate limiter budget. When the limiter is exhausted,
`ForwardPoll` and `ForwardTask` between partitions fail, causing tasks
to be stuck for up to 60s (the long-poll timeout). Deprioritizing cancel
RPCs ensures task-critical RPCs are served first under pressure.
## How did you test it?
Reproduced the issue by running SDK Core integration tests against a
local dev server with `enableCancelWorkerPollsOnShutdown` enabled.
Before the fix: 3/5 iterations had 60s+ stalls with 340
`ResourceExhausted` errors. After the fix: 15/15 iterations passed
cleanly (6-7s each) with 0 `ResourceExhausted` errors at default 1200
RPS.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>1 parent c357e0d commit 481bafc
1 file changed
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
0 commit comments