fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic by aoiasd · Pull Request #48894 · milvus-io/milvus

aoiasd · 2026-04-09T11:51:47Z

Summary

Increment fileResourceRefCnt during validateSchema instead of in the async ack callback's AddCollection, closing the TOCTOU race where RemoveFileResource could delete a resource between validation and AddCollection
On failure before Broadcast, refCnt is decremented immediately; on restart, refCnt for pending broadcast tasks is recovered from etcd before rootcoord becomes Healthy
Remove refCnt++ from addCollectionMeta since it's now done at validation time (reload path unchanged)

Test plan

Existing file resource E2E tests pass (the race that caused [Bug]: Milvus standalone crashes with panic when creating collection that references a non-existent file resource #48612)
CreateCollection with file resource → verify refCnt incremented
RemoveFileResource blocked during in-flight CreateCollection
Restart during CreateCollection → verify refCnt recovered from pending broadcast tasks

issue: #48612
pr: #48893

🤖 Generated with Claude Code

sre-ci-robot · 2026-04-09T11:52:52Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-09T11:52:54Z

[ci-v2-notice]
Notice: ci-v2 system is enabled for this PR (3.0 branch).

To rerun ci-v2 checks, comment with:

/ci-rerun-build-ut-cov // for ci-v2/build-ut-cov (build + unit tests)
/ci-rerun-e2e-amd // for ci-v2/e2e-amd (e2e tests)
/ci-rerun-gosdk // for ci-v2/go-sdk (Go SDK E2E tests)

If you have any questions or requests, please contact @zhikunyao.

sre-ci-robot · 2026-04-09T12:55:44Z

❌ CI Loop Results `5487d79`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	10.5min	-
✅ Code-Check	SUCCESS	6.5min	-
❌ UT-GO	FAILURE	18.7min	926 passed
✅ UT-Integration	SUCCESS	24.4min	46 passed
✅ UT-CPP-Cov	SUCCESS	43.1min	7069 passed

Total: 62min | Pipeline | Artifacts

Failed Test Logs:

UT-GO: view log

sre-ci-robot · 2026-04-09T13:15:36Z

✅ CI Loop Results `5487d79`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	8.2min	-

Total: 14min | Pipeline | Artifacts

sre-ci-robot · 2026-04-09T13:48:10Z

✅ CI Loop Results `5487d79`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	8.1min	-

Total: 12min | Pipeline | Artifacts

sre-ci-robot · 2026-04-09T14:08:08Z

❌ CI Loop Results `5487d79`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	10.6min	-
✅ Code-Check	SUCCESS	7.0min	-
❌ UT-GO	FAILURE	19.6min	926 passed
✅ UT-Integration	SUCCESS	24.3min	46 passed
✅ UT-CPP-Cov	SUCCESS	43.5min	7069 passed

Total: 66min | Pipeline | Artifacts

Failed Test Logs:

UT-GO: view log

sre-ci-robot · 2026-04-13T07:29:24Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-13T08:27:39Z

❌ CI Loop Results `44956b1`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	8.2min	-
✅ Code-Check	SUCCESS	5.4min	-
❌ UT-GO	FAILURE	17.3min	926 passed
✅ UT-Integration	SUCCESS	23.9min	46 passed
✅ UT-CPP-Cov	SUCCESS	36.7min	7069 passed

Total: 57min | Pipeline | Artifacts

Failed Test Logs:

UT-GO: view log

sre-ci-robot · 2026-04-14T08:28:48Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-14T09:26:53Z

❌ CI Loop Results `44956b1`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	8.4min	-
✅ Code-Check	SUCCESS	5.4min	-
❌ UT-GO	FAILURE	17.3min	926 passed
✅ UT-Integration	SUCCESS	23.8min	46 passed
✅ UT-CPP-Cov	SUCCESS	32.1min	7069 passed

Total: 57min | Pipeline | Artifacts

Failed Test Logs:

UT-GO: view log

sre-ci-robot · 2026-04-15T09:24:58Z

❌ CI Loop Results `44956b1`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	10.4min	-
✅ Code-Check	SUCCESS	6.2min	-
❌ UT-GO	FAILURE	17.6min	926 passed
✅ UT-Integration	SUCCESS	24.4min	46 passed
✅ UT-CPP-Cov	SUCCESS	44.2min	7069 passed

Total: 68min | Pipeline | Artifacts

Failed Test Logs:

UT-GO: view log

sre-ci-robot · 2026-04-16T02:40:48Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-16T03:50:09Z

✅ CI Loop Results `d92306e`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	11.5min	-
✅ Code-Check	SUCCESS	8.1min	-
✅ UT-GO	SUCCESS	22.3min	926 passed
✅ UT-Integration	SUCCESS	24.3min	46 passed
✅ UT-CPP-Cov	SUCCESS	44.9min	7071 passed

Total: 69min | Pipeline | Artifacts

sre-ci-robot · 2026-04-16T07:56:41Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-16T08:06:16Z

❌ CI Loop Results `ceb10aa`

Stage	Result	Duration	Tests

Total: 9min | Pipeline | Artifacts

sre-ci-robot · 2026-04-16T08:21:12Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-16T08:30:52Z

❌ CI Loop Results `2241ded`

Stage	Result	Duration	Tests

Total: 8min | Pipeline | Artifacts

sre-ci-robot · 2026-04-16T09:33:09Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-16T09:42:17Z

❌ CI Loop Results `aee90f3`

Stage	Result	Duration	Tests

Total: 8min | Pipeline | Artifacts

sre-ci-robot · 2026-04-22T08:07:47Z

[INFO] PR Label Summary by Default
[FAILED] PR #48893 not merged

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Status: OPEN
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-22T08:23:22Z

❌ CI Loop Results `79f4d90`

Stage	Result	Duration	Tests

Total: 13min | Pipeline | Artifacts

sre-ci-robot · 2026-04-23T02:43:57Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #48893 merged to master

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

Increment fileResourceRefCnt during validateSchema (when file resource IDs are resolved), rather than in the async ack callback's AddCollection. This closes the TOCTOU race window where RemoveFileResource could delete a resource between validation and AddCollection, causing streaming node to panic when creating the tokenizer. On failure before Broadcast, refCnt is decremented immediately. On restart, refCnt for pending broadcast tasks is recovered from etcd before rootcoord becomes Healthy. issue: milvus-io#48612 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

…nt panic (milvus-io#48893) Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

sre-ci-robot · 2026-04-23T03:25:49Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #48893 merged to master

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

[WARNING] Milestone not set

PR: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic #48894
Title: fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

sre-ci-robot · 2026-04-23T03:51:34Z

✅ CI Loop Results `b51d657`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	8.2min	-
✅ Code-Check	SUCCESS	6.4min	-
✅ UT-GO	SUCCESS	20.8min	927 passed
✅ UT-Integration	SUCCESS	24.5min	46 passed
✅ UT-CPP-Cov	SUCCESS	44.3min	7113 passed

Total: 67min | Pipeline | Artifacts

sre-ci-robot · 2026-04-23T04:37:23Z

✅ CI Loop Results `4360bdf`

Stage	Result	Duration	Tests
✅ Build	SUCCESS	10.8min	-
✅ Code-Check	SUCCESS	6.4min	-
✅ UT-GO	SUCCESS	20.0min	927 passed
✅ UT-Integration	SUCCESS	24.5min	46 passed
✅ UT-CPP-Cov	SUCCESS	43.6min	7134 passed

Total: 69min | Pipeline | Artifacts

sre-ci-robot · 2026-04-23T07:10:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aoiasd, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [zhengbuqian]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot · 2026-04-23T07:11:14Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #48893 merged to master

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

Use /refresh-label to update related check and label manually

zhengbuqian · 2026-04-23T07:11:58Z

/lgtm

sre-ci-robot · 2026-04-23T07:12:35Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #48893 merged to master

Title: fix: bind file resource refCnt to collection lifecycle to prevent panic
Link: fix: bind file resource refCnt to collection lifecycle to prevent panic #48893

Use /refresh-label to update related check and label manually

sre-ci-robot requested review from chyezh and sunby April 9, 2026 11:52

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Apr 9, 2026

mergify Bot added the dco-passed DCO check passed. label Apr 9, 2026

sre-ci-robot added the do-not-merge/need-merge-master-first any pr merge to release branch need to merge master first label Apr 9, 2026

mergify Bot added the kind/bug Issues or changes related a bug label Apr 9, 2026

sre-ci-robot added the do-not-merge/need-milestone generate by v2-label-manager label Apr 9, 2026

aoiasd force-pushed the file_resource_panic_3.0 branch from 5487d79 to 44956b1 Compare April 13, 2026 07:28

aoiasd changed the title ~~fix: bind file resource refCnt to collection lifecycle to prevent panic~~ fix: [3.0] bind file resource refCnt to collection lifecycle to prevent panic Apr 14, 2026

aoiasd force-pushed the file_resource_panic_3.0 branch from 44956b1 to d92306e Compare April 16, 2026 02:39

mergify Bot added the ci-passed label Apr 16, 2026

sre-ci-robot removed the ci-passed label Apr 16, 2026

aoiasd force-pushed the file_resource_panic_3.0 branch from ceb10aa to 2241ded Compare April 16, 2026 08:20

aoiasd force-pushed the file_resource_panic_3.0 branch from aee90f3 to 79f4d90 Compare April 22, 2026 08:06

sre-ci-robot removed the do-not-merge/need-merge-master-first any pr merge to release branch need to merge master first label Apr 23, 2026

aoiasd and others added 4 commits April 23, 2026 11:24

fix: [3.0] bind file resource refCnt to collection lifecycle to preve…

0e65fc8

…nt panic (milvus-io#48893) Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

fix: add warn log when file resource refCnt underflow on decrement

8ca9e1e

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

fix: remove unused missing var in RecoverFileResourceRefCnt

4360bdf

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

aoiasd force-pushed the file_resource_panic_3.0 branch from b51d657 to 4360bdf Compare April 23, 2026 03:24

zhengbuqian added this to the 3.0 milestone Apr 23, 2026

mergify Bot added the ci-passed label Apr 23, 2026

zhengbuqian approved these changes Apr 23, 2026

View reviewed changes

sre-ci-robot added the approved label Apr 23, 2026

zhengbuqian removed the do-not-merge/need-milestone generate by v2-label-manager label Apr 23, 2026

sre-ci-robot assigned zhengbuqian Apr 23, 2026

sre-ci-robot added the lgtm label Apr 23, 2026

sre-ci-robot merged commit 305d9e4 into milvus-io:3.0 Apr 23, 2026
10 of 13 checks passed

Conversation

aoiasd commented Apr 9, 2026

Summary

Test plan

Uh oh!

sre-ci-robot commented Apr 9, 2026

Uh oh!

sre-ci-robot commented Apr 9, 2026

Uh oh!

sre-ci-robot commented Apr 9, 2026

❌ CI Loop Results 5487d79

Uh oh!

sre-ci-robot commented Apr 9, 2026

✅ CI Loop Results 5487d79

Uh oh!

sre-ci-robot commented Apr 9, 2026

✅ CI Loop Results 5487d79

Uh oh!

sre-ci-robot commented Apr 9, 2026

❌ CI Loop Results 5487d79

Uh oh!

sre-ci-robot commented Apr 13, 2026

Uh oh!

sre-ci-robot commented Apr 13, 2026

❌ CI Loop Results 44956b1

Uh oh!

sre-ci-robot commented Apr 14, 2026

Uh oh!

sre-ci-robot commented Apr 14, 2026

❌ CI Loop Results 44956b1

Uh oh!

sre-ci-robot commented Apr 15, 2026

❌ CI Loop Results 44956b1

Uh oh!

sre-ci-robot commented Apr 16, 2026

Uh oh!

sre-ci-robot commented Apr 16, 2026

✅ CI Loop Results d92306e

Uh oh!

sre-ci-robot commented Apr 16, 2026

Uh oh!

sre-ci-robot commented Apr 16, 2026

❌ CI Loop Results ceb10aa

Uh oh!

sre-ci-robot commented Apr 16, 2026

Uh oh!

sre-ci-robot commented Apr 16, 2026

❌ CI Loop Results 2241ded

Uh oh!

sre-ci-robot commented Apr 16, 2026

Uh oh!

sre-ci-robot commented Apr 16, 2026

❌ CI Loop Results aee90f3

Uh oh!

sre-ci-robot commented Apr 22, 2026

Uh oh!

sre-ci-robot commented Apr 22, 2026

❌ CI Loop Results 79f4d90

Uh oh!

sre-ci-robot commented Apr 23, 2026

Uh oh!

sre-ci-robot commented Apr 23, 2026

Uh oh!

sre-ci-robot commented Apr 23, 2026

✅ CI Loop Results b51d657

Uh oh!

sre-ci-robot commented Apr 23, 2026

✅ CI Loop Results 4360bdf

Uh oh!

sre-ci-robot commented Apr 23, 2026

Uh oh!

sre-ci-robot commented Apr 23, 2026

Uh oh!

zhengbuqian commented Apr 23, 2026

Uh oh!

sre-ci-robot commented Apr 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

❌ CI Loop Results `5487d79`

✅ CI Loop Results `5487d79`

✅ CI Loop Results `5487d79`

❌ CI Loop Results `5487d79`

❌ CI Loop Results `44956b1`

❌ CI Loop Results `44956b1`

❌ CI Loop Results `44956b1`

✅ CI Loop Results `d92306e`

❌ CI Loop Results `ceb10aa`

❌ CI Loop Results `2241ded`

❌ CI Loop Results `aee90f3`

❌ CI Loop Results `79f4d90`

✅ CI Loop Results `b51d657`

✅ CI Loop Results `4360bdf`