Skip to content

loss出现nan #1

@YCG1

Description

@YCG1

自己制作的9个类别的数据集
train: Fast image access (ping: 0.00.0 ms, read: 2391.01168.9 MB/s, size: 594.6 KB)
train: Scanning C:\Users\12818\Desktop\software\QAT.Ultralytics\coco\labels\train2017.cache... 204 images, 1 backgrounds, 0 corrupt: 100%|██████████| 204/204 [00:00<?, ?it/s]
val: Fast image access (ping: 0.00.0 ms, read: 1370.9526.7 MB/s, size: 664.7 KB)
val: Scanning C:\Users\12818\Desktop\software\QAT.Ultralytics\coco\labels\val2017.cache... 52 images, 0 backgrounds, 0 corrupt: 100%|██████████| 52/52 [00:00<?, ?it/s]
Plotting labels to runs\detect\train5\labels.jpg...
optimizer: SGD(lr=1e-05, momentum=0.937) with parameter groups 167 weight(decay=0.0), 174 weight(decay=0.0005), 173 bias(decay=0.0)
Image sizes 640 train, 640 val
Using 8 dataloader workers
Logging results to runs\detect\train5
Starting training for 6 epochs...

  Epoch    GPU_mem         lr   box_loss   cls_loss   dfl_loss  Instances       Size
    1/6      2.19G      1e-05        nan        nan        nan          9        640:   0%|          | 1/204 [00:00<01:40,  2.01it/s]

Traceback (most recent call last):
File "C:\Users\12818\Desktop\software\QAT.Ultralytics\train.py", line 10, in
results = model.train(data="mycoco.yaml", epochs=6, imgsz=640) #, device=[0, 1]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\Desktop\software\QAT.Ultralytics\ultralytics\engine\model.py", line 872, in train
self.trainer.train()
File "C:\Users\12818\Desktop\software\QAT.Ultralytics\ultralytics\engine\trainer.py", line 239, in train
self._do_train(world_size)
File "C:\Users\12818\Desktop\software\QAT.Ultralytics\ultralytics\engine\trainer.py", line 452, in _do_train
preds = self.qat_model(batch['img'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\fx\graph_module.py", line 822, in call_wrapped
return self._wrapped_call(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\fx\graph_module.py", line 400, in call
raise e
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\fx\graph_module.py", line 387, in call
return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<eval_with_key>.1167", line 2190, in forward
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\nn\modules\module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\nn\modules\module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\12818\AppData\Roaming\Python\Python312\site-packages\torch\ao\quantization\fake_quantize.py", line 408, in forward
return torch.fused_moving_avg_obs_fake_quant(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: zero_point must be between quant_min and quant_max.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions