Skip to content

feat(storage): add zstd replay compression#80

Merged
DriftN2Forty merged 1 commit into
JustinDevB:devfrom
DriftN2Forty:feat/zstd-replay-compression
Jun 24, 2026
Merged

feat(storage): add zstd replay compression#80
DriftN2Forty merged 1 commit into
JustinDevB:devfrom
DriftN2Forty:feat/zstd-replay-compression

Conversation

@DriftN2Forty

Copy link
Copy Markdown
Collaborator

Summary

  • Switch new finalized binary replay archives to Zstd level 1 for both replay.bin timeline payloads and independently compressed chunk payloads.
  • Add explicit timeline payload compression metadata via payloadCompression, while retaining magic-byte fallback for legacy manifests.
  • Keep existing LZ4 timeline archives and chunk payload codec ID 1 readable; new chunk payloads write Zstd codec ID 2.
  • Bump the replay compatibility floor to 1.5.0-alpha.12 so older alpha builds reject new Zstd archives before attempting LZ4 decode.
  • Update binary format docs, manifest schema docs, README, architecture notes, and changelog.

Benchmark Context

Benchmarks were run with /replay debug compression against existing LZ4 archives using 3 runs and 1 warmup.

Sample BRAR-2026-06-16-22-16-45:

  • Archive size: 17.82 MiB
  • Timeline raw/current: 67.65 MiB / 17.01 MiB
  • Timeline LZ4 default: 17.01 MiB, 25.14%, compress avg 116.47 ms, decompress avg 106.72 ms
  • Timeline Zstd level 1: 7.12 MiB, 10.52%, compress avg 80.26 ms, decompress avg 58.42 ms
  • Chunks raw/current: 2.24 MiB / 832.61 KiB across 105 payloads
  • Chunk LZ4 default: 830.95 KiB, 36.20%, compress avg 303.45 ms, decompress avg 675.85 ms
  • Chunk Zstd level 1: 557.80 KiB, 24.30%, compress avg 4.30 ms, decompress avg 1.85 ms

Sample BRAR-2026-06-17-21-52-51:

  • Archive size: 73.81 MiB
  • Timeline raw/current: 279.12 MiB / 73.74 MiB
  • Timeline LZ4 default: 73.74 MiB, 26.42%, compress avg 343.31 ms, decompress avg 231.77 ms
  • Timeline Zstd level 1: 6.25 MiB, 2.24%, compress avg 398.90 ms, decompress avg 505.22 ms
  • Chunks raw/current: 198.20 KiB / 73.42 KiB across 9 payloads
  • Chunk LZ4 default: 73.26 KiB, 36.96%, compress avg 298.63 ms, decompress avg 298.46 ms
  • Chunk Zstd level 1: 49.46 KiB, 24.95%, compress avg 0.49 ms, decompress avg 0.35 ms

These results favor Zstd level 1 as the default because it gives substantially smaller timelines and chunk payloads while keeping compression cost acceptable for new saves.

Compatibility

  • New archives include payloadCompression: "zstd".
  • Archives missing payloadCompression still decode by deterministic frame magic detection.
  • LZ4 frame magic: 04 22 4D 18.
  • Zstd frame magic: 28 B5 2F FD.
  • Existing chunk codec ID 1 remains LZ4_FRAME.
  • New chunk codec ID 2 is ZSTD.
  • The .br ZIP container layout is unchanged and chunk payloads remain selectively decompressible.

Tests

  • mvn test
  • Result: 550 tests, 0 failures, 0 errors, 0 skipped

Added/updated regression coverage for:

  • Zstd timeline archive decoding.
  • Existing LZ4 timeline archive decoding.
  • Missing manifest compression field fallback via magic bytes.
  • Unsupported timeline compression errors.
  • New Zstd chunk codec ID usage and decoding.
  • Existing LZ4 chunk payload decoding.
  • Unsupported chunk codec ID failures.

@DriftN2Forty DriftN2Forty merged commit 2152d40 into JustinDevB:dev Jun 24, 2026
1 check passed
@DriftN2Forty DriftN2Forty deleted the feat/zstd-replay-compression branch June 24, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant