Skip to content

Commit 7f4a736

Browse files
committed
store: Add parquet module with Arrow schema mapping
Create the foundation for Parquet-based dump/restore: a schema mapping module that converts relational Table definitions to Arrow Schema objects. - arrow_schema() maps Table -> Arrow Schema with system columns (vid, block tracking, causality_region) followed by data columns - data_sources_arrow_schema() provides fixed schema for data_sources$ table - All ColumnType variants mapped (TSVector skipped, Enum -> Utf8) - List columns wrapped in Arrow List type based on Column.is_list()
1 parent 8dc6a48 commit 7f4a736

6 files changed

Lines changed: 494 additions & 8 deletions

File tree

Cargo.lock

Lines changed: 110 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ url = "2.5.8"
116116
ahash = "0.8.11"
117117
arrow = { version = "=58.0.0" }
118118
arrow-flight = { version = "=58.0.0", features = ["flight-sql-experimental"] }
119+
parquet = { version = "=58.0.0" }
119120
futures = "0.3.31"
120121
half = "2.7.1"
121122
indoc = "2.0.7"

store/postgres/Cargo.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ pretty_assertions = "1.4.1"
3838
sqlparser = { workspace = true }
3939
thiserror = { workspace = true }
4040
serde_yaml.workspace = true
41+
arrow = { workspace = true }
42+
parquet = { workspace = true }
4143

4244
[dev-dependencies]
4345
clap.workspace = true

store/postgres/src/lib.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ mod fork;
2424
mod functions;
2525
mod jobs;
2626
mod notification_listener;
27+
mod parquet;
2728
mod pool;
2829
mod primary;
2930
pub mod query_store;

store/postgres/src/parquet/mod.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#[allow(dead_code)]
2+
pub(crate) mod schema;

0 commit comments

Comments
 (0)