I read somewhere it might be data loading issue or it can be pytorch version issue. Can you help to rectify?
Begin to resume from the last checkpoint.
Loading checkpoint from out/total3d/20110611514267/model_best.pth.
Warning: Could not find epoch in checkpoint!
Warning: Could not find min_loss in checkpoint!
Warning: Could not find step in checkpoint!
set() subnet missed.
Checkpoint out/total3d/20110611514267/model_best.pth resumed.
----------------------------------------------------------------------------------------------------
Loading data.
Traceback (most recent call last):
File "main.py", line 42, in <module>
demo.run(cfg)
File "/workspace/hardik/Implicit3DUnderstanding/demo.py", line 147, in run
est_data = net(data)
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/hardik/Implicit3DUnderstanding/models/total3d/modules/network.py", line 97, in forward
lo_centroid_result, lo_coeffs_result, a_features = self.layout_estimation(data['image'])
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/hardik/Implicit3DUnderstanding/models/total3d/modules/layout_estimation.py", line 63, in forward
cam = self.fc_1(x)
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
return F.linear(input, self.weight, self.bias)
File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/functional.py", line 1406, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-9xcrj8au/aten/src/THC/THCBlas.cu:259
I read somewhere it might be data loading issue or it can be pytorch version issue. Can you help to rectify?