[Bug]: The mooncake.store module must be imported before mooncake.transferengine; otherwise, a segmentation fault occurs.

[chenhao-stick-to]

Bug Report

mooncake version：v0.3.8
system：almalinux9

As shown in the figure below, in a multithreaded environment, if we comment out the from mooncake.store import MooncakeDistributedStore inside init_store, and instead import mooncake.engine first followed by mooncake.store, a segmentation fault (as shown in the figure) occurs. However, if we import mooncake.store first, the issue disappears.
This problem arises when using SGLang: when its new rfork feature is enabled, it uses TransferEngine for fast model loading, and later initializes MooncakeStore (via HiCache). At that point, the aforementioned segmentation fault appears. The code snippet above is a minimal reproducible example of this issue.
The root cause seems to be related to some kind of race condition or initialization conflict between the two modules and coro_rpc.
I’ve already verified that this crash is 100% reproducible. There must be some underlying low-level issue here. If there’s already a known fix or workaround, please let me know immediately!!!

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

[chenhao-stick-to]

erros are follwed:

[00fish0]

Thanks for reporting this with a detailed reproducible example. I'll look into the root cause and see if I can provide a fix for this initialization conflict.

[00fish0]

Hi @chenhao-stick-to,

I tried to reproduce the segfault with your repro script (importing mooncake.engine before mooncake.store in a multithreaded init), but I still can’t reproduce it on my side. I tested both mooncake v0.3.8 and the current commit (8383b93), and the behavior/logs look the same: 8/8 threads initialize successfully, no crash.

Environment:

• OS: Ubuntu 22.04.5 LTS
• Kernel: 5.15.0-161-generic
• Python: 3.10.12
• mooncake: v0.3.8 and commit 8383b93

Log excerpt (same on both versions):

tmp
I0128 15:53:10.774407 2594358 client_metric.cpp:80] Client metrics enabled (default enabled)
I0128 15:53:10.775859 2594358 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I0128 15:53:10.775872 2594358 client_service.cpp:47] client_id=7804482528259093943-4390027363224591520
I0128 15:53:10.775883 2594358 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0)
I0128 15:53:10.778149 2594358 client_service.cpp:434] Storage root directory is not set. persisting data is disabled.
I0128 15:53:10.778280 2594358 transfer_engine_impl.cpp:586] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0128 15:53:10.778294 2594358 client_service.cpp:247] Set auto discovery ON by default for RDMA protocol, since no device names provided
I0128 15:53:10.778306 2594358 client_service.cpp:258] Transfer engine auto discovery is enabled for protocol: rdma
I0128 15:53:10.778324 2594358 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: miaomiaomiao port: 13130
I0128 15:53:10.778358 2594358 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on miaomiaomiao:15789
I0128 15:53:10.778537 2594358 transfer_engine_impl.cpp:217] Auto-discovering topology...
I0128 15:53:10.814659 2594358 topology.cpp:120] Device ibp12s0 port 1 is available
I0128 15:53:10.850422 2594358 topology.cpp:120] Device ibp18s0 port 1 is available
I0128 15:53:10.886314 2594358 topology.cpp:120] Device ibp75s0 port 1 is available
I0128 15:53:10.922328 2594358 topology.cpp:120] Device ibp84s0 port 1 is available
I0128 15:53:10.939482 2594358 topology.cpp:120] Device ibp141s0 port 1 is available
I0128 15:53:10.974284 2594358 topology.cpp:120] Device ibp148s0 port 1 is available
I0128 15:53:10.994010 2594358 topology.cpp:120] Device ibp186s0 port 1 is available
I0128 15:53:11.015743 2594358 topology.cpp:120] Device ibp204s0 port 1 is available
I0128 15:53:11.049692 2594358 topology.cpp:86] rocep225s0f0:1 is not active (state: 1)
W0128 15:53:11.050479 2594358 topology.cpp:128] Device rocep225s0f0 has no active ports, skipping
W0128 15:53:11.050493 2594358 topology.cpp:165] Skipping unavailable device: rocep225s0f0
I0128 15:53:11.085651 2594358 topology.cpp:86] rocep225s0f1:1 is not active (state: 1)
W0128 15:53:11.086448 2594358 topology.cpp:128] Device rocep225s0f1 has no active ports, skipping
W0128 15:53:11.086462 2594358 topology.cpp:165] Skipping unavailable device: rocep225s0f1
I0128 15:53:11.093745 2594358 transfer_engine_impl.cpp:232] Topology discovery complete. Found 8 HCAs.
I0128 15:53:11.093766 2594358 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering.
I0128 15:53:11.093782 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.137823 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp186s0/ (network state: without network device)
I0128 15:53:11.151326 2594358 rdma_context.cpp:140] RDMA device: ibp186s0, LID: 4, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d0:08:12
I0128 15:53:11.151360 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.188025 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp12s0/ (network state: without network device)
I0128 15:53:11.194366 2594358 rdma_context.cpp:140] RDMA device: ibp12s0, LID: 7, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:f2
I0128 15:53:11.194396 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.232322 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp18s0/ (network state: without network device)
I0128 15:53:11.239403 2594358 rdma_context.cpp:140] RDMA device: ibp18s0, LID: 2, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ff:50:c8
I0128 15:53:11.239432 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.276087 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp75s0/ (network state: without network device)
I0128 15:53:11.281599 2594358 rdma_context.cpp:140] RDMA device: ibp75s0, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:1b:2e
I0128 15:53:11.281629 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.320220 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp84s0/ (network state: without network device)
I0128 15:53:11.322026 2594358 rdma_context.cpp:140] RDMA device: ibp84s0, LID: 1, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:1b:1e
I0128 15:53:11.322057 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.359941 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp141s0/ (network state: without network device)
I0128 15:53:11.381256 2594358 rdma_context.cpp:140] RDMA device: ibp141s0, LID: 3, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:56
I0128 15:53:11.381290 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.419966 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp148s0/ (network state: without network device)
I0128 15:53:11.430357 2594358 rdma_context.cpp:140] RDMA device: ibp148s0, LID: 5, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:7e
I0128 15:53:11.430390 2594358 rdma_context.cpp:77] Using SIEVE endpoint store
I0128 15:53:11.449609 2594358 rdma_context.cpp:575] Find best gid index: 0 on ibp204s0/ (network state: without network device)
I0128 15:53:11.457861 2594358 rdma_context.cpp:140] RDMA device: ibp204s0, LID: 6, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:22:36
I0128 15:53:11.458019 2594358 transfer_engine_impl.cpp:297] installTransport, type=rdma
I0128 15:53:11.459439 2594358 real_client.cpp:225] Successfully created client on port 13130 after 1 attempt(s)
I0128 15:53:11.464496 2594358 real_client.cpp:253] Registering local memory: 16777216 bytes
I0128 15:53:11.536280 2594358 real_client.cpp:277] Mounting segment: 44975581388 bytes, 44975581388 of 44975581388
Multi-thread init: 8/8 succeeded
I0128 15:53:48.117645 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:16261 finish
I0128 15:53:55.355418 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:16600 finish
I0128 15:54:00.781497 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:16669 finish
I0128 15:54:07.511737 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:15431 finish
I0128 15:54:13.798511 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:15789 finish
I0128 15:54:20.532135 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:15722 finish
I0128 15:54:27.611367 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:15092 finish
I0128 15:54:34.989173 2594354 transfer_metadata.cpp:301] removeSegmentDesc miaomiaomiao:16134 finish

If there’s any specific config or build flag to match your environment more closely, please let me know and I can try again.

[chenhao-stick-to]

`import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import List, Optional, Tuple
import os
import time
import sys
local_hostname = "10.148.0.152"
metadata_server = "P2PHANDSHAKE"
master_server_address = "10.45.180.240:50051"
protocol = os.getenv("MOONCAKE_PROTOCOL", "tcp")
device_name = ""
global_segment_size = 449755813
DEFAULT_LOCAL_BUFFER_SIZE = 1610241024
def init_transfer(num_threads: int = 8):
from mooncake.engine import TransferEngine
remote_instance_transfer_engine = TransferEngine()
remote_instance_transfer_engine.initialize(
local_hostname, "P2PHANDSHAKE", protocol, device_name
)
print("begin mooncake-transferengine")
time.sleep(10)
def init_mooncakestore():
print("begine mooncake-store")
from mooncake.store import MooncakeDistributedStore
store = MooncakeDistributedStore()
print("tmp")
ret_code = store.setup( local_hostname, metadata_server, global_segment_size,DEFAULT_LOCAL_BUFFER_SIZE, protocol, device_name, master_server_address)
time.sleep(30)
def init_mooncake_multi_thread(num_threads: int = 8) -> Tuple[bool, List, List]:
stores = [None] * num_threads
errors = [None] * num_threads
def init_store(thread_id: int):
try：
init_transfer()
init_mooncakestore()
if ret_code:
return (thread_id, None, f"Setup failed: {ret_code}")
return (thread_id, store, None)
except Exception as e:
return (thread_id, None, str(e))

with ThreadPoolExecutor(max_workers=num_threads) as executor:
    futures = {executor.submit(init_store, i): i for i in range(num_threads)}
    for future in as_completed(futures):
        thread_id, store, error = future.result()
        stores[thread_id] = store
        errors[thread_id] = error
success = all(err is None for err in errors)
print(f"Multi-thread init: {sum(1 for err in errors if err is None)}/{num_threads} succeeded")
return (success, stores, errors)

init_mooncake_multi_thread(8)`
I used this script to reproduce the issue。could you try again？

[00fish0]

Thanks for providing the updated reproduction script.

I have tested this new script using mooncake v0.3.8 in my environment, but unfortunately, I am still unable to reproduce the segmentation fault. The issue might be tied to specific environmental factors.

[chenhao-stick-to]

ok，thank you for your work；may is the environment problem.lastly,i used the v0.5.7 version of yalantinglibs to compile,worked it out