我的配置：
include: [
'../deim_dfine/dfine_hgnetv2_n_coco.yml',
'../base/deim.yml'
]
output_dir: ./outputs/deim_hgnetv2_n_virdrone
print_freq: 100
checkpoint_freq: 12
DEIM:
backbone: HGNetv2
HGNetv2:
pretrained: True
HybridEncoder:
in_channels: [512, 1024]
feat_strides: [16, 32]
intra
hidden_dim: 128
use_encoder_idx: [1]
dim_feedforward: 512
cross
expansion: 0.34
depth_mult: 0.5
optimizer:
type: AdamW
params:
- params: '^(?=.backbone)(?!.norm|bn).$'
lr: 0.0004
- params: '^(?=.backbone)(?=.norm|bn).$'
lr: 0.0004
weight_decay: 0.
- params: '^(?=.(?:encoder|decoder))(?=.(?:norm|bn|bias)).*$'
weight_decay: 0.
lr: 0.0008
betas: [0.9, 0.999]
weight_decay: 0.0001
Increase to search for the optimal ema
epoches: 160 # 148 + 12
Our LR-Scheduler
flat_epoch: 7800 # 4 + epoch // 2, e.g., 40 = 4 + 72 / 2
no_aug_epoch: 12
lr_gamma: 1.0
Our DataAug
train_dataloader:
dataset:
transforms:
policy:
epoch: [4, 78, 148] # list
collate_fn:
mixup_epochs: [4, 78]
stop_epoch: 148
base_size_repeat: ~
total_batch_size: 32
val_dataloader:
total_batch_size: 32
运行命令：
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --master_port=7777 --nproc_per_node=8 train.py -c /workspace/DEIM-main/configs/deim_add/deim_hgnetv2_n_custom.yml --seed=0
报错如下：
[rank1]: Traceback (most recent call last):
[rank1]: File "/workspace/DEIM-main/train.py", line 86, in
[rank1]: main(args)
[rank1]: File "/workspace/DEIM-main/train.py", line 56, in main
[rank1]: solver.fit()
[rank1]: File "/workspace/DEIM-main/engine/solver/det_solver.py", line 72, in fit
[rank1]: self.load_resume_state(str(self.output_dir / 'best_stg1.pth'))
[rank1]: File "/workspace/DEIM-main/engine/solver/_solver.py", line 159, in load_resume_state
[rank1]: state = torch.load(path, map_location='cpu')
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1114, in load
[rank1]: return _legacy_load(
[rank1]: ^^^^^^^^^^^^^
[rank1]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1338, in _legacy_load
[rank1]: magic_number = pickle_module.load(f, **pickle_load_args)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: EOFError: Ran out of input
[rank2]: Traceback (most recent call last):
[rank2]: File "/workspace/DEIM-main/train.py", line 86, in
[rank2]: main(args)
[rank2]: File "/workspace/DEIM-main/train.py", line 56, in main
[rank2]: solver.fit()
[rank2]: File "/workspace/DEIM-main/engine/solver/det_solver.py", line 72, in fit
[rank2]: self.load_resume_state(str(self.output_dir / 'best_stg1.pth'))
[rank2]: File "/workspace/DEIM-main/engine/solver/_solver.py", line 159, in load_resume_state
[rank2]: state = torch.load(path, map_location='cpu')
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1097, in load
[rank2]: return _load(
[rank2]: ^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1525, in _load
[rank2]: result = unpickler.load()
[rank2]: ^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1492, in persistent_load
[rank2]: typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1457, in load_tensor
[rank2]: storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage)._typed_storage()._untyped_storage
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: PytorchStreamReader failed reading file data/1778: file read failed
[rank4]: Traceback (most recent call last):
[rank4]: File "/workspace/DEIM-main/train.py", line 86, in
[rank4]: main(args)
[rank4]: File "/workspace/DEIM-main/train.py", line 56, in main
[rank4]: solver.fit()
[rank4]: File "/workspace/DEIM-main/engine/solver/det_solver.py", line 72, in fit
[rank4]: self.load_resume_state(str(self.output_dir / 'best_stg1.pth'))
[rank4]: File "/workspace/DEIM-main/engine/solver/_solver.py", line 159, in load_resume_state
[rank4]: state = torch.load(path, map_location='cpu')
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1097, in load
[rank4]: return _load(
[rank4]: ^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1525, in _load
[rank4]: result = unpickler.load()
[rank4]: ^^^^^^^^^^^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1492, in persistent_load
[rank4]: typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1457, in load_tensor
[rank4]: storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage)._typed_storage()._untyped_storage
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: RuntimeError: PytorchStreamReader failed reading file data/352: file read failed
/workspace/DEIM-main/engine/solver/_solver.py:159: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(path, map_location='cpu')
/workspace/DEIM-main/engine/solver/_solver.py:159: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(path, map_location='cpu')
/workspace/DEIM-main/engine/solver/_solver.py:159: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(path, map_location='cpu')
/workspace/DEIM-main/engine/solver/solver.py:159: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(path, map_location='cpu')
best_stat: {'epoch': 147, 'coco_eval_bbox': 0.16942706319677847}
/workspace/DEIM-main/engine/solver/solver.py:159: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(path, map_location='cpu')
Load last_epoch
Load model.state_dict
Load criterion.state_dict
Load postprocessor.state_dict
Load ema.state_dict
Load optimizer.state_dict
Load lr_warmup_scheduler.state_dict
Refresh EMA at epoch 148 with decay 0.9999
[rank0]:[E308 20:08:19.620193721 ProcessGroupNCCL.cpp:1375] [PG 0 (default_pg) Rank 0] First PG on this rank that detected no heartbeat of its watchdog.
[rank0]:[E308 20:08:19.620372258 ProcessGroupNCCL.cpp:1413] [PG 0 (default_pg) Rank 0] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList.size()=3
[rank0]:[F308 20:18:19.621154347 ProcessGroupNCCL.cpp:1224] [PG 0 (default_pg) Rank 0] [PG 0 (default_pg) Rank 0] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList.size() = 3
W0308 20:18:19.400000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858761 closing signal SIGTERM
W0308 20:18:19.402000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858762 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858763 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858764 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858765 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858766 closing signal SIGTERM
W0308 20:18:19.404000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858767 closing signal SIGTERM
E0308 20:18:30.726000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: -6) local_rank: 0 (pid: 858760) of binary: /root/miniconda3/envs/deim/bin/python3.11
Traceback (most recent call last):
File "/root/miniconda3/envs/deim/bin/torchrun", line 7, in
sys.exit(main())
^^^^^^
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2026-03-08_20:18:19
host : f08ab1506738
rank : 0 (local_rank: 0)
exitcode : -6 (pid: 858760)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 858760
请问是否有遇到过同样问题的伙伴，感谢你们能够提供一下解决意见
我的配置：
include: [
'../deim_dfine/dfine_hgnetv2_n_coco.yml',
'../base/deim.yml'
]
output_dir: ./outputs/deim_hgnetv2_n_virdrone
print_freq: 100
checkpoint_freq: 12
DEIM:
backbone: HGNetv2
HGNetv2:
pretrained: True
HybridEncoder:
in_channels: [512, 1024]
feat_strides: [16, 32]
intra
hidden_dim: 128
use_encoder_idx: [1]
dim_feedforward: 512
cross
expansion: 0.34
depth_mult: 0.5
optimizer:
type: AdamW
params:
- params: '^(?=.backbone)(?!.norm|bn).$'
lr: 0.0004
- params: '^(?=.backbone)(?=.norm|bn).$'
lr: 0.0004
weight_decay: 0.
- params: '^(?=.(?:encoder|decoder))(?=.(?:norm|bn|bias)).*$'
weight_decay: 0.
lr: 0.0008
betas: [0.9, 0.999]
weight_decay: 0.0001
Increase to search for the optimal ema
epoches: 160 # 148 + 12
Our LR-Scheduler
flat_epoch: 7800 # 4 + epoch // 2, e.g., 40 = 4 + 72 / 2
no_aug_epoch: 12
lr_gamma: 1.0
Our DataAug
train_dataloader:
dataset:
transforms:
policy:
epoch: [4, 78, 148] # list
collate_fn:
mixup_epochs: [4, 78]
stop_epoch: 148
base_size_repeat: ~
total_batch_size: 32
val_dataloader:
total_batch_size: 32
运行命令：
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --master_port=7777 --nproc_per_node=8 train.py -c /workspace/DEIM-main/configs/deim_add/deim_hgnetv2_n_custom.yml --seed=0
报错如下：
[rank1]: Traceback (most recent call last):
[rank1]: File "/workspace/DEIM-main/train.py", line 86, in
[rank1]: main(args)
[rank1]: File "/workspace/DEIM-main/train.py", line 56, in main
[rank1]: solver.fit()
[rank1]: File "/workspace/DEIM-main/engine/solver/det_solver.py", line 72, in fit
[rank1]: self.load_resume_state(str(self.output_dir / 'best_stg1.pth'))
[rank1]: File "/workspace/DEIM-main/engine/solver/_solver.py", line 159, in load_resume_state
[rank1]: state = torch.load(path, map_location='cpu')
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1114, in load
[rank1]: return _legacy_load(
[rank1]: ^^^^^^^^^^^^^
[rank1]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1338, in _legacy_load
[rank1]: magic_number = pickle_module.load(f, **pickle_load_args)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: EOFError: Ran out of input
[rank2]: Traceback (most recent call last):
[rank2]: File "/workspace/DEIM-main/train.py", line 86, in
[rank2]: main(args)
[rank2]: File "/workspace/DEIM-main/train.py", line 56, in main
[rank2]: solver.fit()
[rank2]: File "/workspace/DEIM-main/engine/solver/det_solver.py", line 72, in fit
[rank2]: self.load_resume_state(str(self.output_dir / 'best_stg1.pth'))
[rank2]: File "/workspace/DEIM-main/engine/solver/_solver.py", line 159, in load_resume_state
[rank2]: state = torch.load(path, map_location='cpu')
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1097, in load
[rank2]: return _load(
[rank2]: ^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1525, in _load
[rank2]: result = unpickler.load()
[rank2]: ^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1492, in persistent_load
[rank2]: typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1457, in load_tensor
[rank2]: storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage)._typed_storage()._untyped_storage
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: PytorchStreamReader failed reading file data/1778: file read failed
[rank4]: Traceback (most recent call last):
[rank4]: File "/workspace/DEIM-main/train.py", line 86, in
[rank4]: main(args)
[rank4]: File "/workspace/DEIM-main/train.py", line 56, in main
[rank4]: solver.fit()
[rank4]: File "/workspace/DEIM-main/engine/solver/det_solver.py", line 72, in fit
[rank4]: self.load_resume_state(str(self.output_dir / 'best_stg1.pth'))
[rank4]: File "/workspace/DEIM-main/engine/solver/_solver.py", line 159, in load_resume_state
[rank4]: state = torch.load(path, map_location='cpu')
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1097, in load
[rank4]: return _load(
[rank4]: ^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1525, in _load
[rank4]: result = unpickler.load()
[rank4]: ^^^^^^^^^^^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1492, in persistent_load
[rank4]: typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/serialization.py", line 1457, in load_tensor
[rank4]: storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage)._typed_storage()._untyped_storage
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: RuntimeError: PytorchStreamReader failed reading file data/352: file read failed
/workspace/DEIM-main/engine/solver/_solver.py:159: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.state = torch.load(path, map_location='cpu')
/workspace/DEIM-main/engine/solver/_solver.py:159: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.state = torch.load(path, map_location='cpu')
/workspace/DEIM-main/engine/solver/_solver.py:159: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.state = torch.load(path, map_location='cpu')
/workspace/DEIM-main/engine/solver/solver.py:159: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.state = torch.load(path, map_location='cpu')
best_stat: {'epoch': 147, 'coco_eval_bbox': 0.16942706319677847}
/workspace/DEIM-main/engine/solver/solver.py:159: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.state = torch.load(path, map_location='cpu')
Load last_epoch
Load model.state_dict
Load criterion.state_dict
Load postprocessor.state_dict
Load ema.state_dict
Load optimizer.state_dict
Load lr_warmup_scheduler.state_dict
Refresh EMA at epoch 148 with decay 0.9999
[rank0]:[E308 20:08:19.620193721 ProcessGroupNCCL.cpp:1375] [PG 0 (default_pg) Rank 0] First PG on this rank that detected no heartbeat of its watchdog.
[rank0]:[E308 20:08:19.620372258 ProcessGroupNCCL.cpp:1413] [PG 0 (default_pg) Rank 0] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList.size()=3
[rank0]:[F308 20:18:19.621154347 ProcessGroupNCCL.cpp:1224] [PG 0 (default_pg) Rank 0] [PG 0 (default_pg) Rank 0] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList.size() = 3
W0308 20:18:19.400000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858761 closing signal SIGTERM
W0308 20:18:19.402000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858762 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858763 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858764 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858765 closing signal SIGTERM
W0308 20:18:19.403000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858766 closing signal SIGTERM
W0308 20:18:19.404000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 858767 closing signal SIGTERM
E0308 20:18:30.726000 140491984159808 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: -6) local_rank: 0 (pid: 858760) of binary: /root/miniconda3/envs/deim/bin/python3.11
Traceback (most recent call last):
File "/root/miniconda3/envs/deim/bin/torchrun", line 7, in
sys.exit(main())
^^^^^^
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/deim/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2026-03-08_20:18:19
host : f08ab1506738
rank : 0 (local_rank: 0)
exitcode : -6 (pid: 858760)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 858760
请问是否有遇到过同样问题的伙伴，感谢你们能够提供一下解决意见