环境：

在容器中跑pytorch模型的训练 

问题表现：

ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/root/anaconda3/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/root/anaconda3/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 358, in reduce_storage
    fd, size = storage._share_fd_cpu_()
RuntimeError: unable to write to file </torch_27822_2114653298_869>: No space left on device (28)

问题原因：

torch.utils.data.dataloader使用了共享内存。容器根目录没有空间，导致共享内存写到文件的时候报错。

解决方案，dataloader不使用共享内存：

from torch.utils.data import dataloader
from torch.multiprocessing import reductions
from multiprocessing.reduction import ForkingPicklerdefault_collate_func = dataloader.default_collatedef default_collate_override(batch):dataloader._use_shared_memory = Falsereturn default_collate_func(batch)setattr(dataloader, 'default_collate', default_collate_override)for t in torch._storage_classes:if sys.version_info[0] == 2:if t in ForkingPickler.dispatch:del ForkingPickler.dispatch[t]else:if t in ForkingPickler._extra_reducers:del ForkingPickler._extra_reducers[t]

 参考：

https://blog.csdn.net/u012796629/article/details/105936386