perf: async .eml checksum, PROPFIND size reuse, E2EE parallel transfers#9995
Open
qoole wants to merge 1 commit intonextcloud:masterfrom
Open
perf: async .eml checksum, PROPFIND size reuse, E2EE parallel transfers#9995qoole wants to merge 1 commit intonextcloud:masterfrom
qoole wants to merge 1 commit intonextcloud:masterfrom
Conversation
- Make .eml checksum computation async during discovery to avoid blocking the event loop on large mailbox syncs. The previous synchronous computeLocalChecksum could pause discovery for several seconds on large .eml files; replaced with a ComputeChecksum job that defers the result via finalize(). - Reuse folder size from the PROPFIND response (RemoteInfo.sizeOfFolder) instead of issuing a redundant PropfindJob for selective sync size checks. The size is already available from the parent PROPFIND that populated serverEntry; an extra round trip per directory is wasteful. - Allow E2EE directories to report FullParallelism so encrypted folders on different paths can transfer concurrently. Within-folder serialization is preserved by PropagateItemJob, so this only relaxes the cross-folder constraint that previously serialized all E2EE transfers globally. Signed-off-by: Qoole <2862661+qoole@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three independent throughput improvements to discovery and propagation.
Async
.emlchecksum during discovery (discovery.cpp)processFileAnalyzeLocalInfocomputes a local checksum to detect false positives on.emlfile change detection (issues #4754, #4755). The computation was synchronous and could block the discovery thread for seconds on large mailbox files.Replaced with an async
ComputeChecksumjob. The result is captured into the existingfinalize()lambda, so flow is preserved: if the computed checksum matches the database, the item is downgraded toUPDATE_METADATA; otherwise discovery continues normally. While the checksum runs,_pendingAsyncJobskeeps the discovery state machine alive and other work proceeds.PROPFIND folder-size reuse (
discoveryphase.cpp/.h,discovery.cpp)checkSelectiveSyncNewFolderandcheckSelectiveSyncExistingFolderwere issuing a separatePropfindJobfor each newly-encountered remote directory just to learn its size. The parent PROPFIND that producedserverEntryalready has the folder size inRemoteInfo.sizeOfFolder.Both functions now accept the size as a parameter and skip the redundant PROPFIND when it's already known. On a large remote tree this eliminates one HTTP round trip per directory.
E2EE FullParallelism (
owncloudpropagator.cpp)OwncloudPropagator::pushDirectoryToPropagateJobwas forcing E2EE directories toWaitForFinished, which serialized all encrypted folder transfers globally — even folders on completely different paths. Within-folder ordering is already enforced byPropagateItemJob; the cross-folder serialization was a stronger guarantee than needed.Allow E2EE directories to report
FullParallelism. Encrypted folders on different paths can now transfer concurrently. Within-folder serialization is preserved.Checklist
AI (if applicable)