Support Snappy compression and a configurable GZIP level for make_examples output

[tfenne]

Summary

make_examples has always written its tf.Example output as GZIP-compressed TFRecords at zlib's default level, with the codec hardcoded on both the write and read sides. On many-core machines the GZIP step is a meaningful slice of make_examples CPU time, and when output throughput is provisioned generously the cheaper Snappy codec is often the better trade. This PR makes the examples codec selectable and exposes the GZIP level. Output compression type is keyed off filenames, compression level by a new option.

Going from gzip level 6 to level 1 on chr20 saved ~7% of make_examples runtime, at the cost of about doubling the output size. Going gzip level 6 to snappy saved 10% at the cost of about 4x more storage. The tradeoff is user selectable based on available storage, and is much more modest with the small-model handling a large fraction of calls now.

Design: the file name is the single source of truth

The compression codec is inferred from the examples file-name suffix rather than from a separate flag:

• a *.snappy examples path selects Snappy;
• any other suffix keeps the historical GZIP behaviour.

The C++ writer (nucleus::ExampleWriter, via the new CompressionTypeForPath) and every examples reader (call_variants, data_providers, show_examples, and the shape probe in dv_utils, via the new dv_utils.compression_type_for_examples_path) derive the codec the same way. Detection is case-insensitive and handles the usual sharded (@N, -ddddd-of-ddddd) and comma-separated path forms.

A new --examples_compression_level flag (proto MakeExamplesOptions.examples_compression_level) exposes the GZIP level (-1 or 0..9; -1 = library default). It only affects GZIP output. The proto field is declared optional so an unset value is distinguishable from a deliberate level 0 (otherwise a non-flag caller would silently get level 0 / no deflate), and a level supplied alongside a .snappy path is ignored with a warning (Snappy has no levels).

Backward compatibility

Default behaviour is unchanged: with no .snappy suffix and no level flag, output is GZIP at the library default level, exactly as before. The auxiliary make_examples outputs (call_variant_outputs, small_model_examples) are written through the Python TFRecord writer, which cannot emit Snappy, so they remain GZIP — and their file names are forced to .gz even when the main examples are Snappy, so each file's name matches its actual bytes.

What changed

• third_party/nucleus/io/example_writer.{h,cc}: codec inferred from the path; compression_level plumbed into RecordWriterOptions.zlib_options; new public CompressionTypeForPath.
• deepvariant/protos/deepvariant.proto: optional int32 examples_compression_level.
• deepvariant/make_examples_options.py: --examples_compression_level flag + range validator.
• deepvariant/make_examples_native.cc: passes the level (library default when unset).
• deepvariant/dv_utils.py, call_variants.py, data_providers.py, show_examples.py: readers autodetect the codec from the suffix.
• deepvariant/make_examples_core.py: side outputs forced to .gz.

Testing

• third_party/nucleus/io:example_writer_test — codec detection, Snappy write/read round-trip, and GZIP level actually applied (level 0 file larger than level 9).
• deepvariant:dv_utils_test — suffix detection across casing and sharding forms.
• deepvariant:make_examples_core_test — side-output .snappy → .gz renaming (incl. uppercase).
• End-to-end make_examples → call_variants round-trip on a chr20 region for both Snappy and GZIP examples (and an explicit GZIP level), confirming call_variants reads Snappy output and that an out-of-range level is rejected.

[tfenne]

Pushed an extra commit this morning that exposes the compression options in run_deepvariant.py too.

[pichuan]

Hi @tfenne ,

Thanks for the PR!

Since I believe you're already familiar with our process, I'll go ahead and start the review.

As a reminder, because of the way our project is set up, we aren't able to merge GitHub PRs directly. If the changes look good, I will commit them, crediting your GitHub username and referencing this PR in the commit description.

Please let me know if you have any concerns with this approach.

-pichuan

[tfenne]

I re-benchmarked this PR's changes vs. r1.10 through the standard docker build process, running the baseline and modified versions in the resulting docker containers on the chr20 short-read WGS set on a c8a.4xlarge at AWS with 16 cores / 16 shards. In that setup:

baseline runtime: 108.6
pr runtime: 102.7
% change: ~5.2% improvement

The baseline is different from my other PR, I think largely run-to-run variance; the above is the average of multiple interleaved replicates. Also, the improvement shown above is only seen if a) the option is supplied to switch to snappy intermediate compression, and b) the storage has enough throughput to absorb the extra writes (the snappy compressed outputs are ~3x the size of the gzip level 6, but with the small model on, the absolute size is not so bad).