又是好久没更新博客，最近琐事缠身，写文档写到吐。没时间学习新的知识，刚空闲下来立刻就学习之前忘得差不多得Pytorch。Pytorch和tensorflow差不多，具体得就不多啰嗦了，觉得还有疑问的童鞋可以自行去官网学习，官网网址包括：

1.https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

2.https://pytorch-cn.readthedocs.io/zh/latest/package_references/torch-nn/#class-torchnndataparallelmodule-device_idsnone-output_devicenone-dim0source

 

个人觉得值得一说的是关于数据并行这部分：

根据官网的教程代码总结如下所示：

# -*- coding: utf-8 -*-
"""
Created on Tue Apr 16 14:32:58 2019@author: kofzh
"""import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader# Parameters and DataLoaders
input_size = 5
output_size = 2batch_size = 30
data_size = 100device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")class RandomDataset(Dataset):def __init__(self, size, length):self.len = lengthself.data = torch.randn(length, size)def __getitem__(self, index):return self.data[index]def __len__(self):return self.lenrand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),batch_size=batch_size, shuffle=True)class Model(nn.Module):# Our modeldef __init__(self, input_size, output_size):super(Model, self).__init__()self.fc = nn.Linear(input_size, output_size)def forward(self, input):output = self.fc(input)print("\tIn Model: input size", input.size(),"output size", output.size())return outputmodel = Model(input_size, output_size)
if torch.cuda.device_count() > 1:print("Let's use", torch.cuda.device_count(), "GPUs!")# dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUsmodel = nn.DataParallel(model)model.to(device)
for data in rand_loader:input = data.to(device)output = model(input)print("Outside: input size", input.size(),"output_size", output.size())

对于上述代码分别是 batchsize= 30在CPU和4路GPU环境下运行得出的结果：

CPU with batchsize= 30结果：

In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

4路 GPU with batchsize= 30 结果：

Let's use 4 GPUs!In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([6, 5]) output size torch.Size([6, 2])In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size    In Model: input sizetorch.Size([8, 5])  In Model: input size torch.Size([8, 5])output size      In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])torch.Size([6, 5]) output size torch.Size([6, 2])output size torch.Size([8, 2])torch.Size([8, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size    In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])
torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([6, 5]) output size torch.Size([6, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size    In Model: input sizetorch.Size([3, 5])  In Model: input size torch.Size([3, 5]) output size torch.Size([3, 2])torch.Size([3, 5]) output size torch.Size([3, 2])output size torch.Size([3, 2])In Model: input size torch.Size([1, 5]) output size torch.Size([1, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

对于上述代码分别是 batchsize= 25在CPU和4路GPU环境下运行得出的结果：

CPU with batchsize= 25结果：

 In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])

4路 GPU with batchsize= 25 结果：

Let's use 4 GPUs!In Model: input size    In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size    In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size    In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size    In Model: input size  torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])
torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size    In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])

个人总结：batchsize在实际训练时对于每个GPU的分配基本上采用的是平均分配数据的原则，每个GPU分配的数据相加 = batchsize。

备注：

1.官网上的CPU结果图是错误的，可能是图截错了，在此对于某些博客直接抄袭官网的结果图表示鄙视。

2.本文针对单服务器多GPU的情况，请不要混淆。

如有错误，敬请指正！