Use HDD for data storage, SSH for metadata

[georgeboot]

Our storage servers have 12 hdd's for actual data, and 2 NVMe ssd's for metadata. How would I configure RustFS to storage metadata on the ssd's so metadata operations don't have to hit the (relatively slow) hdd's?

In Ceph its common practice to add ssd's to servers with hdd storage to store WAL and metadata on, and I'm wondering how to get similar results with RustFS.

[loverustfs]

This feature is not supported. Our metadata and data are stored on the hard drive.

[georgeboot]

Right, that was my conclusion as well. Thank you.

Are there plans to change this in the future? Or does it matter less because of architectural differences between Ceph and RustFS?

[inkeliz]

Garage has a similar design: SSD for metadata, HDD for storage.
I'm testing Garage and RustFS and well... something else.

Mixing metadata and storage doesn't seem like a good idea at all. Even on ZFS you can split it.

[andy-freefly]

RustFS doesn't separate metadata from data at the application level, but you can achieve the same effect transparently at the block layer using bcache in writeback mode.

Our setup (4-node cluster, each node):

• 10× 15TB HDD (data) bcache backing devices
• 2× 1.92TB NVMe SSD (SK Hynix SE5031) bcache caching set, RAID-1 via LVM mirror
• 10× bcache devices (bcache0–bcache9), each HDD fronted by the SSD cset
• XFS on each bcache device with external journals on a dedicated SSD LV

Tuning for metadata-heavy workloads (object size 700KB–1MB):

# Prioritize random small I/O (metadata) over sequential data
echo 2097152 > /sys/block/bcacheN/bcache/sequential_cutoff  # cache sequential < 2MB
echo writeback > /sys/block/bcacheN/bcache/cache_mode
echo 20 > /sys/block/bcacheN/bcache/writeback_percent        # flush when SSD >20% dirty
echo 60 > /sys/block/bcacheN/bcache/writeback_delay

# XFS: external journal on SSD eliminates metadata write latency entirely
mkfs.xfs -l logdev=/dev/vg_xfslog/xfslog_diskNN,size=2g ...
mount -o logdev=/dev/vg_xfslog/xfslog_diskNN,logbsize=256k,allocsize=256k ...

Why this works for RustFS metadata:
RustFS stores bucket metadata, IAM state, and object index in .rustfs.sys/ — small, frequently-accessed files. With sequential_cutoff=2MB, bcache caches all of this on SSD automatically. Large object payloads (> 2MB) bypass the cache and go directly to HDD, keeping SSD space free for metadata.

Result: metadata latency approaches NVMe, data throughput is unaffected. Effectively the same outcome as Ceph's BlueStore with db/wal on SSD just at the block layer instead of application layer.

The limitation vs. Ceph BlueStore: cache eviction is LRU across all data, not explicitly reserved for metadata. But with 2× 1.92TB SSD and objects of 700KB–1MB, the working set fits comfortably.

We've been running this configuration in production for several weeks on a 4-node cluster migrated from MinIO, no issues so far.