Commit e65d222
authored
feat(btree): Add BTree global index reader with async on-demand block loading (#229)
Implement BTree index file format support (block, footer, meta, SST file,
var_len encoding) and global index scanner for evaluating predicates against
BTree global indexes to produce row ID ranges.
Key features:
- Async on-demand data block reading via FileRead trait instead of loading
entire file into memory
- Scanner-level BTreeIndexReader cache for reader reuse across evaluations
- AND predicate grouping by field_id to minimize file opens
- Between pattern detection (GtEq/Gt + LtEq/Lt merged into single range_query)
- Row ID predicate extraction from data predicates into row ranges
- Support for point lookup, range, IN, NOT IN, IS NULL, IS NOT NULL queries
- Zstd block compression/decompression support
- File-level pruning using BTreeIndexMeta (first_key, last_key, has_nulls)1 parent dd71c58 commit e65d222
28 files changed
Lines changed: 4631 additions & 13 deletions
File tree
- crates/paimon
- src
- btree
- io
- spec
- table
- testdata/btree
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| 59 | + | |
58 | 60 | | |
59 | 61 | | |
60 | 62 | | |
| |||
0 commit comments