Skip to content

fix: [2.5] InvertedIndexTantivy/JsonInvertedIndex should also follow scalar index mmap in 2.5. #42626#48875

Open
lmzzzzz1 wants to merge 1 commit intomilvus-io:2.5from
lmzzzzz1:pr-42626-2.5
Open

fix: [2.5] InvertedIndexTantivy/JsonInvertedIndex should also follow scalar index mmap in 2.5. #42626#48875
lmzzzzz1 wants to merge 1 commit intomilvus-io:2.5from
lmzzzzz1:pr-42626-2.5

Conversation

@lmzzzzz1
Copy link
Copy Markdown
Contributor

@lmzzzzz1 lmzzzzz1 commented Apr 9, 2026

This PR backports PR #42676 and #42788 from master to the 2.5 branch.

The relevant issue has already been addressed in master and 2.6. This PR brings the same changes into 2.5 to keep behavior consistent across branches.
Any conflicts encountered during cherry-picking have been resolved.

see also: #42626

@lmzzzzz1
fix: [2.5] support RAM mode when mmap is disabled for text indexes (m…
e3f930e 
…ilvus-io#42626)

Cherry-pick PRs milvus-io#42676 and milvus-io#42788 from master to 2.5 branch with
conflict resolution.

PR milvus-io#42676: Refactor text/json index temp build paths to avoid
cross-field conflicts. Replace hardcoded TMP_*_PREFIX constants with
DiskFileManagerImpl methods (GetLocalTempTextIndexPrefix,
GetLocalTempJsonKeyIndexPrefix).

PR milvus-io#42788: Add mmap configuration support for InvertedIndex, TextMatch,
and JsonKeyStats indexes. When enable_mmap=true, indexes load via mmap
(MmapDirectory); when false, files are read into RAM (RamDirectory)
and local files are cleaned up early.

Key adaptations for 2.5 branch:
- Keep 2.5's path helpers (GetIndexPathPrefixWithBuildID,
  GetTextIndexIdentifier, GetJsonKeyIndexIdentifier)
- Add load_fs_index_into_ram_directory() in util.rs as standalone
  function (consistent with upstream tantivy fork's open_in_dir_in_ram
  API style) since 2.5's tantivy version (0.21.2-fix4) lacks this API
- Use isDataMmapEnable(f) for text/json indexes (already in 2.5)
- Pass enable_mmap through Proto -> CGo -> C++ -> Rust full chain

Signed-off-by: 1mmortal <lmzzzzz1@163.com>
@sre-ci-robot
Copy link
Copy Markdown
Contributor

sre-ci-robot commented Apr 9, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lmzzzzz1
To complete the pull request process, please assign yanliang567 after the PR has been reviewed.
You can assign the PR to them by writing /assign @yanliang567 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Apr 9, 2026
@mergify mergify Bot added the dco-passed DCO check passed. label Apr 9, 2026
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Apr 9, 2026

@lmzzzzz1 Please associate the related pr of master to the body of your Pull Request. (eg. "pr: #")

@mergify mergify Bot added do-not-merge/missing-related-pr kind/bug Issues or changes related a bug labels Apr 9, 2026
@sre-ci-robot sre-ci-robot added do-not-merge/need-merge-master-first any pr merge to release branch need to merge master first do-not-merge/need-milestone generate by v2-label-manager labels Apr 9, 2026
@sre-ci-robot
Copy link
Copy Markdown
Contributor

sre-ci-robot commented Apr 9, 2026

[INFO] PR Label Summary by Default
[WARNING] No dependent PR reference found

  • Target branch '2.5' requires a PR merged to master first
  • Please add reference in format 'pr: #number'

[WARNING] Milestone not set

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

@sre-ci-robot
Copy link
Copy Markdown
Contributor

sre-ci-robot commented Apr 9, 2026

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

  • /ci-rerun-code-check // for ci-v2/code-check
  • /ci-rerun-build // for ci-v2/build
  • /ci-rerun-build-all // for ci-v2/build-all (multi-arch builds)
  • /ci-rerun-buildenv // for ci-v2/build-env (build milvus-env builder images)
  • /ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
  • /ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
  • /ci-rerun-ut-cpp // for ci-v2/ut-cpp
  • /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
  • /ci-rerun-e2e-default // for ci-v2/e2e-default
  • /ci-rerun-e2e-amd // for ci-v2/e2e-amd (e2e pool dispatcher)
  • /ci-rerun-build-ut-cov // for ci-v2/build-ut-cov (build + unit tests in one pipeline)
  • /ci-rerun-gosdk // for ci-v2/go-sdk (Go SDK E2E tests, ARM)

If you have any questions or requests, please contact @zhikunyao.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 9, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.20%. Comparing base (3a7a08f) to head (e3f930e).
⚠️ Report is 108 commits behind head on 2.5.

Files with missing lines Patch % Lines
internal/querynodev2/segments/segment.go 0.00% 2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (48.20%) is below the target coverage (77.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##              2.5   #48875       +/-   ##
===========================================
- Coverage   82.10%   48.20%   -33.90%     
===========================================
  Files        1128     1046       -82     
  Lines      179181   172381     -6800     
===========================================
- Hits       147110    83099    -64011     
- Misses      26099    82270    +56171     
- Partials     5972     7012     +1040
Components Coverage Δ
Client ∅ <ø> (∅)
Core ∅ <ø> (∅)
Go 47.98% <49.93%> (-34.40%) ⬇️
Files with missing lines Coverage Δ
internal/querynodev2/segments/segment.go 54.53% <0.00%> (-9.85%) ⬇️

... and 900 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@yanliang567 yanliang567 added this to the 2.5.28 milestone Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@godchen0212 godchen0212 Awaiting requested review from godchen0212

Assignees

No one assigned

Labels

dco-passed DCO check passed. do-not-merge/missing-related-pr do-not-merge/need-merge-master-first any pr merge to release branch need to merge master first do-not-merge/need-milestone generate by v2-label-manager kind/bug Issues or changes related a bug size/L Denotes a PR that changes 100-499 lines.

Projects

None yet

Milestone

2.5.28

Development

Successfully merging this pull request may close these issues.

3 participants

@lmzzzzz1 @sre-ci-robot @yanliang567