Search before asking
Description
This issue tracks the Arrow-to-Parquet conversion part of splitting parent task #437.
We need a reusable utility or writer path that can convert Arrow RecordBatch directly into Parquet without first converting through row-oriented representations. This would improve the efficiency and clarity of Arrow-native data flows, especially for tiering and other lakehouse-related write paths.
Possible scope:
- provide a reusable Arrow
RecordBatch to Parquet conversion utility;
- define the supported Arrow/Parquet type mapping and failure behavior;
- make the conversion path suitable for direct reuse by tiering writers and other lake components.
This is intended to be one sub-task of #437, while enabling tiering source to read data as Arrow RecordBatch is tracked separately.
Willingness to contribute
Search before asking
Description
This issue tracks the Arrow-to-Parquet conversion part of splitting parent task #437.
We need a reusable utility or writer path that can convert Arrow
RecordBatchdirectly into Parquet without first converting through row-oriented representations. This would improve the efficiency and clarity of Arrow-native data flows, especially for tiering and other lakehouse-related write paths.Possible scope:
RecordBatchto Parquet conversion utility;This is intended to be one sub-task of #437, while enabling tiering source to read data as Arrow
RecordBatchis tracked separately.Willingness to contribute