feat(compact): Support global LookupFileCache for compact lookup mode

[lxy-9602]

Purpose

Linked issue: #93
Introduce a global LookupFileCache based on a new generic LRU cache (GenericLruCache) to manage lookup files across LookupLevels instances in compact lookup mode. This enables:

• Weight-based eviction: Limits local disk usage via lookup.cache-max-disk-size (default: unlimited).
• Time-based expiration: Automatically expires cached lookup files via lookup.cache-file-retention (default: 1 hour). Expired files are re-read from DFS on next access.
• Shared cache across compactions: Multiple LookupLevels instances share a single LookupFileCache owned by MergeTreeCompactManagerFactory, with each instance tracking its own cached files for proper cleanup on close.
• Refactored LruCache: The existing block-level LruCache is refactored to delegate to GenericLruCache, eliminating duplicated LRU logic.
• Refactored CoreOptions

Tests

• GenericLruCacheTest.*
• LookupFileTest.TestLookupFileCacheLifecycle
• LookupLevelsTest.TestLookupFileCacheIntegration
• LookupLevelsTest.TestCacheEvictionBySmallMaxDiskSize
• LookupLevelsTest.TestCacheEvictionByExpiration
• PkCompactionInteTest.PkDvAndAggWithIOException

API and Format

• include/paimon/defs.h: Added two new option constants:
  • Options::LOOKUP_CACHE_FILE_RETENTION ("lookup.cache-file-retention")
  • Options::LOOKUP_CACHE_MAX_DISK_SIZE ("lookup.cache-max-disk-size")

Documentation

Generative AI tooling

Generated-by: Aone Copilot (Claude claude4.6)

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces a global, shared LookupFileCache for compact lookup mode using a new generic LRU cache, enabling disk-size-based eviction and time-based expiration of locally cached lookup files.

Changes:

• Added GenericLruCache with weight-based eviction, expiration, and removal callbacks; refactored block LruCache to delegate to it.
• Implemented a global LookupFileCache integration in LookupLevels and MergeTreeCompactManagerFactory, including per-instance cleanup tracking.
• Added new CoreOptions/Options (lookup.cache-file-retention, lookup.cache-max-disk-size) and extensive UT/IT coverage for caching behavior.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
test/inte/pk_compaction_inte_test.cpp	Adds an integration test covering DV + aggregation with injected I/O errors.
src/paimon/core/mergetree/lookup_levels_test.cpp	Updates tests for shared cache API and adds cache integration/eviction/expiration coverage.
src/paimon/core/mergetree/lookup_levels.h	Switches per-instance map cache to shared global cache + ownership tracking; adds Close/dtor.
src/paimon/core/mergetree/lookup_levels.cpp	Implements shared cache usage, weight capture, ownership tracking, and per-instance cleanup.
src/paimon/core/mergetree/lookup_file_test.cpp	Extends LookupFile tests to validate cache lifecycle and file deletion on eviction.
src/paimon/core/mergetree/lookup_file.h	Adds LookupFileCache type alias, cache factory, weight/removal callbacks, and per-file callback.
src/paimon/core/mergetree/lookup/remote_lookup_file_manager_test.cpp	Updates helper creation to pass the new lookup file cache into LookupLevels.
src/paimon/core/mergetree/lookup/remote_lookup_file_manager.cpp	Propagates AddLocalFile status handling.
src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory.h	Introduces factory-owned global lookup_file_cache_ and invalidates it on Close.
src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory.cpp	Lazily creates and wires the global cache into lookup-based rewriters.
src/paimon/core/mergetree/compact/lookup_merge_tree_compact_rewriter_test.cpp	Updates tests to construct and pass LookupFileCache.
src/paimon/core/core_options_test.cpp	Adds coverage for new lookup cache retention/max-disk options parsing/defaults.
src/paimon/core/core_options.h	Adds getters for lookup cache retention and max disk size.
src/paimon/core/core_options.cpp	Implements parsing/storage of retention/max-disk options.
src/paimon/common/utils/generic_lru_cache_test.cpp	Adds comprehensive tests for eviction, expiration, callbacks, and concurrency.
src/paimon/common/utils/generic_lru_cache.h	Adds the new generic LRU cache implementation.
src/paimon/common/memory/memory_segment.h	Adds operator== used by generic cache equality helpers for pointer values.
src/paimon/common/io/cache/lru_cache_test.cpp	Updates tests to assert Put() status after signature change.
src/paimon/common/io/cache/lru_cache.h	Refactors block cache to wrap GenericLruCache and changes Put() to return Status.
src/paimon/common/io/cache/lru_cache.cpp	Implements LruCache as a thin wrapper over GenericLruCache.
src/paimon/common/io/cache/cache.h	Updates Cache::Put() to return Status and adds CacheValue::operator==.
src/paimon/common/defs.cpp	Defines new option string constants.
src/paimon/CMakeLists.txt	Registers the new generic LRU cache test target.
include/paimon/defs.h	Exposes new public option constants for retention and max disk size.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.