Skip to content

perf: batch E2EE queries, batch dir rename, hash lookup, larger Windows watcher buffer#9993

Open
qoole wants to merge 1 commit intonextcloud:masterfrom
qoole:perf/e2ee-batch-and-rename
Open

perf: batch E2EE queries, batch dir rename, hash lookup, larger Windows watcher buffer#9993
qoole wants to merge 1 commit intonextcloud:masterfrom
qoole:perf/e2ee-batch-and-rename

Conversation

@qoole
Copy link
Copy Markdown

@qoole qoole commented May 7, 2026

Summary

A set of related performance fixes touching sync startup, propagation, and the tray.

Batch E2EE path queries (syncjournaldb.cpp)

getRootE2eFolderRecord and findEncryptedAncestorForRecord previously walked up the path one level at a time, issuing a separate getFileRecord per level. For a path 8 levels deep that's 8 round trips to SQLite on every E2EE-related lookup.

Replaced with a single WHERE path IN (...) query that fetches all candidate ancestors at once. The result is then walked locally to find the deepest e2e-encrypted match.

Batch directory rename (propagatorjobs.cpp)

PropagateLocalRename::start previously processed child journal entries inline while iterating a streaming SELECT. Mixing reads and writes on the same SQLite statement is fragile and prevented batching. Refactored to:

  1. Collect all child records into a vector while the SELECT is open.
  2. Close the read.
  3. Apply the rename mutations in a single pass.

This makes the operation safer and lets the journal batch its writes.

Remove Instantiator destroy/recreate hack (MainWindow.qml, CurrentAccountHeaderButton.qml)

The tray was destroying and recreating its account Instantiator on every menu open/close as a workaround for an old binding glitch. This causes a full delegate teardown and rebuild every time the user opens the tray. The hack is no longer needed; removed it.

O(n²) -> O(1) folder adjustment (owncloudpropagator.cpp)

adjustDeletedFoldersWithNewChildren looked up each item with std::find_if over the full vector, making the function O(n²). Replaced with a QHash<QString, int> index built once and reused for each lookup.

Windows file watcher buffer (folderwatcher_win.cpp)

The ReadDirectoryChangesW buffer was 40KB initial / 64KB max, which overflows easily under heavy file churn (e.g. large extracts, IDE save cascades). On overflow Windows reports a single "buffer overflow" event and the watcher falls back to a full-tree rescan, which is extremely expensive on large folders.

Bumped to 256KB initial / 2MB max. The buffer only grows as needed.

Checklist

  • Sign-off message is added to all commits.
  • The commit history is clean with no merge commits.
  • Uploaded screenshots from before and after for UI changes (tray Instantiator change is not visually different to the user; account list still populates as before).
  • Documentation has been updated or is not required.
  • Backports requested where applicable (critical bugfixes).
  • Labels added where applicable (bug/enhancement).
  • Milestone added for target version.

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

…sh lookup

- Batch E2EE path queries: replace N serial getFileRecord calls with
  single IN query for getRootE2eFolderRecord and
  findEncryptedAncestorForRecord. With deeply nested encrypted folders
  this reduces sync startup latency by collapsing per-level DB round
  trips into a single statement.
- Batch directory rename: collect child records first, then process
  mutations after the streaming query completes. Avoids interleaved
  read/write on the same statement and lets the journal commit cleanly.
  Commit the journal once before the batch loop so each child rename
  starts from a clean transaction boundary instead of accumulating in
  the previous batch's transaction.
- Remove Instantiator destroy/recreate hack on tray open / menu close
  (eliminates full delegate teardown + rebuild on every interaction).
- O(n^2) to O(1) folder adjustment: QHash lookup replaces std::find_if
  in adjustDeletedFoldersWithNewChildren. Builds a name index once and
  reuses it instead of rescanning per item.
- Windows file watcher buffer: 40KB->256KB start, 64KB->2MB max.
  Larger initial buffer prevents buffer overflow under heavy file
  churn, which was triggering expensive full-tree rescans.

Signed-off-by: Qoole <2862661+qoole@users.noreply.github.com>
@qoole qoole force-pushed the perf/e2ee-batch-and-rename branch from 7fe0d3d to c260fac Compare May 7, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant