GODRIVER-3858 message size exceeds limit of 48000000#2382
Open
qingyang-hu wants to merge 3 commits intomongodb:release/2.5from
Open
GODRIVER-3858 message size exceeds limit of 48000000#2382qingyang-hu wants to merge 3 commits intomongodb:release/2.5from
qingyang-hu wants to merge 3 commits intomongodb:release/2.5from
Conversation
Contributor
🧪 Performance ResultsCommit SHA: 18348e1The following benchmark tests for version 69fd6bafdab28c0007badb59 had statistically significant changes (i.e., |z-score| > 1.96):
For a comprehensive view of all microbenchmark results for this PR's commit, please check out the Evergreen perf task for this patch. |
Contributor
API Change ReportNo changes found! |
qingyang-hu
force-pushed
the
godriver3858
branch
3 times, most recently
from
889b4ff to
eac6ff1
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes batch size accounting during OP_MSG batching so the already-written bytes in the destination buffer (e.g., OP_MSG header/body) are included when deciding how many documents can fit, preventing “message size exceeds limit” errors during large batched writes.
Changes:
- Initialize batch
sizeusing the currentdstlength inAppendBatchSequenceandAppendBatchArray. - Update unit tests to reflect the corrected size calculation behavior.
- Add an integration prose test that exercises InsertMany near
maxMessageSizeBytes.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
x/mongo/driver/batches.go |
Fixes size initialization so batching decisions include bytes already appended to dst. |
x/mongo/driver/batches_test.go |
Adjusts size limits in unit tests to match the corrected batching logic. |
internal/integration/crud_prose_test.go |
Adds an integration test intended to cover near-max message size InsertMany batching. |
Comments suppressed due to low confidence (1)
x/mongo/driver/batches.go:83
- AppendBatchArray's size accounting only adds len(doc) per element. The actual bytes appended by bsoncore.AppendDocumentElement also include the element header (type byte + cstring key) and AppendArrayEnd appends a trailing 0x00. As a result, this function can construct a batch whose encoded size exceeds totalSize (notably when totalSize is cryptMaxBsonObjectSize during auto-encryption). Consider including the per-element header bytes (based on strconv.Itoa(n)) and the final terminator in the size check, or perform the size check using the prospective encoded length after append.
aidx, dst := bsoncore.AppendArrayElementStart(dst, b.Identifier)
size := len(dst)
var n int
for i := b.offset; i < len(b.Documents); i++ {
if n == maxCount {
break
}
doc := b.Documents[i]
size += len(doc)
if size > totalSize {
break
}
dst = bsoncore.AppendDocumentElement(dst, strconv.Itoa(n), doc)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1032
to
+1035
| var docs []any | ||
| limit := hello.MaxBsonObjectSize - 30 | ||
| for need := hello.MaxMessageSizeBytes - 350; need > 0; need -= limit { | ||
| if need >= limit { |
| }) | ||
| } | ||
|
|
||
| func TestBatchSize(t *testing.T) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GODRIVER-3858
Summary
Include OP_MSG head when calculating the size.
Commits a954620 and 806fe4e are from the external PR #2360 credit to @koenno.
Added a prose test in 18348e1 for better coverage.
Background & Motivation
sizeshould not be 0 at very beginning because there are already some bytes written to dst therefore the size should be initialized with dst length.