feat(rocksdb): add memory limit configuration options#302
Open
frperezr wants to merge 3 commits intocozodb:mainfrom
Open
feat(rocksdb): add memory limit configuration options#302frperezr wants to merge 3 commits intocozodb:mainfrom
frperezr wants to merge 3 commits intocozodb:mainfrom
Conversation
Add support for configuring RocksDB's block cache size through the
options JSON parameter when creating a database with the rocksdb engine.
Changes:
- Add block_cache_size setter to DbBuilder (cozorocks)
- Fix bug: use actual block_cache_size value instead of hardcoded 1GB
- Fix bug: apply cache to default options, not just when options_path exists
- Fix bug: preserve cache when bloom_filter is enabled
- Add RocksDbOpts struct and new_cozo_rocksdb_with_opts function
- Parse block_cache_size from JSON options in DbInstance::new
Usage:
DbInstance::new("rocksdb", "/path/to/db", "{\"block_cache_size\": 2147483648}")
This allows users to control memory usage by limiting the block cache,
preventing unbounded memory growth during large ingestion workloads.
Add three new RocksDB options to limit memory usage: - write_buffer_size: size of each memtable (default 64MB) - max_write_buffer_number: max concurrent memtables (default 2) - db_write_buffer_size: total memory budget for all memtables These options help prevent unbounded memory growth during heavy write workloads like bulk ingestion.
frperezr
changed the title
…ilter enabled When use_bloom_filter is true, BlockBasedTableOptions was created without cache_index_and_filter_blocks=true, causing index and filter blocks to be stored outside the block cache and grow unbounded. Copy the essential settings from default_db_options() to ensure consistent behavior regardless of bloom filter configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add RocksDB memory configuration options to prevent unbounded memory growth:
block_cache_size: Limit LRU block cache (read cache)write_buffer_size: Size per memtablemax_write_buffer_number: Max concurrent memtablesdb_write_buffer_size: Total memtable memory budgetMotivation
Without these limits, RocksDB can consume all available memory during heavy write workloads, leading to OOM kills on containers.
Usage
{ "block_cache_size": 2147483648, "write_buffer_size": 67108864, "max_write_buffer_number": 4, "db_write_buffer_size": 536870912 }