Skip to content

Commit 481bafc

Browse files
rkannan82claude
andauthored
Deprioritize CancelOutstandingWorkerPolls in matching rate limiter (#10010)
## What Move `CancelOutstandingWorkerPolls` from priority 1 to priority 2 in the matching service rate limiter, and add priority level 2 to `APIPrioritiesOrdered`. ## Why `CancelOutstandingWorkerPolls` fans out to `numPartitions × numTaskTypes` RPCs per worker shutdown. Under concurrent worker shutdowns, this burst competes with Poll and AddTask RPCs (both priority 1) for the matching rate limiter budget. When the limiter is exhausted, `ForwardPoll` and `ForwardTask` between partitions fail, causing tasks to be stuck for up to 60s (the long-poll timeout). Deprioritizing cancel RPCs ensures task-critical RPCs are served first under pressure. ## How did you test it? Reproduced the issue by running SDK Core integration tests against a local dev server with `enableCancelWorkerPollsOnShutdown` enabled. Before the fix: 3/5 iterations had 60s+ stalls with 340 `ResourceExhausted` errors. After the fix: 15/15 iterations passed cleanly (6-7s each) with 0 `ResourceExhausted` errors at default 1200 RPS. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent c357e0d commit 481bafc

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

service/matching/configs/quotas.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ var (
1616
"/temporal.server.api.matchingservice.v1.MatchingService/AddActivityTask": 1,
1717
"/temporal.server.api.matchingservice.v1.MatchingService/AddWorkflowTask": 1,
1818
"/temporal.server.api.matchingservice.v1.MatchingService/CancelOutstandingPoll": 1,
19-
"/temporal.server.api.matchingservice.v1.MatchingService/CancelOutstandingWorkerPolls": 1,
19+
"/temporal.server.api.matchingservice.v1.MatchingService/CancelOutstandingWorkerPolls": 2,
2020
"/temporal.server.api.matchingservice.v1.MatchingService/DescribeTaskQueue": 1,
2121
"/temporal.server.api.matchingservice.v1.MatchingService/ListTaskQueuePartitions": 1,
2222
"/temporal.server.api.matchingservice.v1.MatchingService/PollActivityTaskQueue": 1,
@@ -55,7 +55,7 @@ var (
5555
"/temporal.server.api.matchingservice.v1.MatchingService/CheckTaskQueueVersionMembership": 1,
5656
}
5757

58-
APIPrioritiesOrdered = []int{0, 1}
58+
APIPrioritiesOrdered = []int{0, 1, 2}
5959
)
6060

6161
func NewPriorityRateLimiter(

0 commit comments

Comments
 (0)