第28周：Transformer 实现文本分类

前言

一、前期准备

1.1 环境安装

1.2 加载数据

二、数据预处理

2.1 构建词典

2.2 进行one-hot编码

2.3 自定义数据集类

2.4 定义填充函数

2.5 构建数据集

三、模型构建

3.1 定义位置编码器

3.2 定义Transformer模型

3.3 定义训练函数

3.4 定义测试函数

四、训练模型

4.1 模型训练

4.2 模型评估

五、模型调优

总结

前言

🍨 本文为[🔗365天深度学习训练营]中的学习记录博客
🍖 原作者：[K同学啊]

说在前面

1）本周任务：在上周的代码基础上，将嵌入方式改为Embedding嵌入

2）运行环境：Python3.8、Pycharm2020、torch1.12.1+cu113

一、前期准备

1.1 环境安装

本文是基于Pytorch框架实现的文本分类

代码如下：

#一、准备工作
#1.1 环境安装
import torch,torchvision
print(torch.__version__)
print(torchvision.__version__)
import torch.nn as nn
from torchvision import transforms, datasets
import os, PIL,pathlib,warnings
import pandas as pdwarnings.filterwarnings("ignore")device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

打印输出：

2.0.0+cu118
0.15.1+cu118
cuda

1.2 加载数据

代码如下：


#1.2 加载数据
#加载自定义中文数据
train_data = pd.read_csv('train.csv', sep='\t', header=None)
print(train_data.head())
label_name = list(set(train_data[1].values[:]))
print('label name:', label_name)

打印输出：

0 1
0 还有双鸭山到淮阴的汽车票吗13号的 Travel-Query
1 从这里怎么回家 Travel-Query
2 随便播放一首专辑阁楼里的佛里的歌 Music-Play
3 给看一下墓王之王嘛 FilmTele-Play
4 我想看挑战两把s686打突变团竞的游戏视频 Video-Play
['Radio-Listen', 'TVProgram-Play', 'Video-Play', 'Travel-Query', 'Weather-Query', 'Music-Play', 'HomeAppliance-Control', 'Other', 'Calendar-Query', 'FilmTele-Play', 'Alarm-Update', 'Audio-Play']

二、数据预处理

2.1 构建词典

需要安装jieba分词库，安装语句pip install jieba

代码如下（示例）：

#二、数据预处理
#2.1 构建词典
from torchtext.vocab import build_vocab_from_iterator
from torchtext.data.utils import get_tokenizer
import jieba#中文分词方法
tokenizer = jieba.lcut
def yield_tokens(data_iter):for text,_ in data_iter:yield tokenizer(text)vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=["<unk>"])
vocab.set_default_index(vocab["<unk>"])text_pipeline = lambda x: vocab(tokenizer(x))
label_pipeline = lambda x: label_name.index(x)print(text_pipeline('我想看和平精英上战神必备技巧的游戏视频'))
print(label_pipeline('Video-Play'))

打印输出：

Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\XiaoMa\AppData\Local\Temp\jieba.cache
Loading model cost 0.320 seconds.
Prefix dict has been built successfully.
[2, 10, 13, 973, 1079, 146, 7724, 7574, 7793, 1, 186, 28]
11

2.2 进行one-hot编码

代码如下：

#from functools import partialX = [text_pipeline(i) for i in train_data[0].values[:]]
y = [label_pipeline(i) for i in train_data[1].values[:]]# 对便签 y 进行 one-hot 编码
numbers_array = np.array(y)             # 转换为 NumPy 数组
num_classes   = np.max(numbers_array)+1 # 计算类别数量
y = np.eye(num_classes)[numbers_array]  # 进行 one-hot 编码

2.3 自定义数据集类

代码如下：

#from torch.utils.data import DataLoader, Datasetclass TextDataset(Dataset):def __init__(self, texts, labels):self.texts  = textsself.labels = labelsdef __len__(self):return len(self.labels)def __getitem__(self, idx):return self.texts[idx], self.labels[idx]

2.4 定义填充函数

代码如下：

import torch.nn.functional as Fmax_len = max(len(i) for i in X)def collate_batch(batch, max_len):texts, labels = zip(*batch)padded_texts = [F.pad(text, (0, max_len - len(text)), value=0) for text in texts]padded_texts = torch.stack(padded_texts)labels = torch.tensor(labels, dtype=torch.float)#.unsqueeze(1)return padded_texts.to(device), labels.to(device)# 使用 partial 函数创建 collate_fn，传入参数
collate_fn = partial(collate_batch, max_len=max_len)

2.5 构建数据集

代码如下：

from torch.utils.data.dataset import random_split# 示例词典大小和嵌入维度
vocab_size = 10
embed_dim  = 6# 创建数据集和数据加载器
dataset = TextDataset(X, y)train_dataset, valid_dataset = random_split(dataset,[int(len(dataset)*0.8),int(len(dataset)*0.2)])train_dataloader = DataLoader(train_dataset, batch_size=128,shuffle=True, collate_fn=collate_fn)
valid_dataloader = DataLoader(valid_dataset, batch_size=128,shuffle=True, collate_fn=collate_fn)

三、模型构建

3.1 定义位置编码器

代码如下：

#三、模型构建
#3.1 定义位置编码函数
import math
#位置编码
class PositionalEncoding(nn.Module):"实现位置编码"def __init__(self, embed_dim, max_len=500):super(PositionalEncoding, self).__init__()# 初始化Shape为(max_len,embed_dim)的PE (positional encoding)pe = torch.zeros(max_len, embed_dim)# 初始化一个tensor [max_len, 1]position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)# 这里就是sin和cos括号中的内容，通过e和ln进行了变换div_term = torch.exp(torch.arange(0, embed_dim, 2) * (-math.log(100.0) / embed_dim))pe[:, 0::2] = torch.sin(position * div_term)  # 计算PE(pos, 2i)pe[:, 1::2] = torch.cos(position * div_term)  # 计算PE(pos, 2i+1)pe = pe.unsqueeze(0).transpose(0, 1)  # 为了方便计算，在最外面在unsqueeze出一个batch# 如果一个参数不参与梯度下降，但又希望保存model的时候将其保存下来# 这个时候就可以用register_buffer# 这里将位置编码张量注册为模型的缓冲区，参数不参与梯度下降self.register_buffer('pe', pe)def forward(self, x):# 将x和positional encoding相加。#print(x.shape)#x = x.unsqueeze(1)#print(x.shape)#print(self.pe[:x.size(0)].shape)#x = x.unsqueeze(1).expand(-1, 1, -1)  # 将 x 的形状调整为 [4, 1, 64]x = x + self.pe[:x.size(0)]return x

3.2 定义Transformer模型

代码如下：

#3.2 定义Transformer模型
from tempfile import TemporaryDirectory
from typing import Tuple
from torch import nn, tensor
from torch.nn import TransformerEncoder, TransformerEncoderLayer
from torch.utils.data import datasetclass TransformerModel(nn.Module):def __init__(self, vocab_size, embed_dim, num_class, nhead=8,d_hid=256,nlayers=12,dropout=0.1):super().__init__()self.embedding = nn.EmbeddingBag(vocab_size,embed_dim,sparse=False)self.pos_encoder = PositionalEncoding(embed_dim)#定义编码器层encoder_layers = TransformerEncoderLayer(embed_dim, nhead, d_hid, dropout)self.transformer_encoder = TransformerEncoder(encoder_layers,nlayers)self.embed_dim = embed_dimself.linear = nn.Linear(embed_dim*4, num_class)def forward(self, src, offsets, src_mask=None):src = self.embedding(src, offsets)src = self.pos_encoder(src)output = self.transformer_encoder(src, src_mask)output = output.view(4, embed_dim * 4)output = self.linear(output)return output

3.3 定义训练函数

代码如下：

# 训练循环
def train(dataloader, model, loss_fn, optimizer):size = len(dataloader.dataset)  # 训练集的大小num_batches = len(dataloader)   # 批次数目, (size/batch_size，向上取整)train_loss, train_acc = 0, 0  # 初始化训练损失和正确率for X, y in dataloader:  # 获取图片及其标签X, y = X.to(device), y.to(device)# 计算预测误差pred = model(X)          # 网络输出loss = loss_fn(pred, y)  # 计算网络输出和真实值之间的差距，targets为真实值，计算二者差值即为损失# 反向传播optimizer.zero_grad()  # grad属性归零loss.backward()        # 反向传播optimizer.step()       # 每一步自动更新# 记录acc与losstrain_acc  += (pred.argmax(1) == y.argmax(1)).type(torch.float).sum().item()train_loss += loss.item()train_acc  /= sizetrain_loss /= num_batchesreturn train_acc, train_loss

3.4 定义测试函数

代码如下：

def test(dataloader, model, loss_fn):size        = len(dataloader.dataset)  # 测试集的大小num_batches = len(dataloader)          # 批次数目, (size/batch_size，向上取整)test_loss, test_acc = 0, 0# 当不进行训练时，停止梯度更新，节省计算内存消耗with torch.no_grad():for imgs, target in dataloader:imgs, target = imgs.to(device), target.to(device)# 计算losstarget_pred = model(imgs)loss        = loss_fn(target_pred, target)test_loss += loss.item()test_acc  += (target_pred.argmax(1) == target.argmax(1)).type(torch.float).sum().item()test_acc  /= sizetest_loss /= num_batchesreturn test_acc, test_loss

四、训练模型

vocab_size = len(vocab) # 词汇表的大小
embed_dim  = 64         # 嵌入维度
num_class  = len(label_name)# 创建 Transformer 模型，并将其移动到设备上
model = TransformerModel(vocab_size, embed_dim, max_len,num_class).to(device)

4.1 模型训练

代码如下：

import torch.optim as optim# 超参数
EPOCHS     = 50 criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)for epoch in range(1, EPOCHS + 1):model.train()  # 切换为训练模式train_acc, train_loss = train(train_dataloader, model, criterion, optimizer)model.eval()  # 切换为测试模式test_acc, test_loss = test(valid_dataloader, model, criterion)# 获取当前的学习率lr = optimizer.state_dict()['param_groups'][0]['lr']print('| epoch {:1d} | train_acc {:4.4f} train_loss {:4.4f} | lr {:4.4f}'.format(epoch,train_acc,train_loss,lr))print('| epoch {:1d} | test_acc {:4.4f} test_loss {:4.4f} | lr {:4.4f}'.format(epoch,test_acc,test_loss,lr))print('-' * 69)

代码输出：

| epoch 1 | train_acc 0.1109 train_loss 2.4322 | lr 0.0100
| epoch 1 | test_acc 0.1129 test_loss 2.5413 | lr 0.0100
---------------------------------------------------------------------
| epoch 2 | train_acc 0.1305 train_loss 2.5153 | lr 0.0100
| epoch 2 | test_acc 0.1860 test_loss 2.3967 | lr 0.0100
---------------------------------------------------------------------
| epoch 3 | train_acc 0.1987 train_loss 2.2949 | lr 0.0100
| epoch 3 | test_acc 0.1938 test_loss 2.2864 | lr 0.0100
---------------------------------------------------------------------
| epoch 4 | train_acc 0.2620 train_loss 2.1522 | lr 0.0100
| epoch 4 | test_acc 0.3000 test_loss 2.0298 | lr 0.0100
---------------------------------------------------------------------
| epoch 5 | train_acc 0.3127 train_loss 2.0171 | lr 0.0100
| epoch 5 | test_acc 0.3149 test_loss 1.9701 | lr 0.0100
---------------------------------------------------------------------
| epoch 6 | train_acc 0.3718 train_loss 1.8808 | lr 0.0100
| epoch 6 | test_acc 0.3508 test_loss 1.9691 | lr 0.0100
---------------------------------------------------------------------
| epoch 7 | train_acc 0.4099 train_loss 1.7816 | lr 0.0100
| epoch 7 | test_acc 0.4322 test_loss 1.7314 | lr 0.0100
---------------------------------------------------------------------
| epoch 8 | train_acc 0.4507 train_loss 1.6637 | lr 0.0100
| epoch 8 | test_acc 0.4678 test_loss 1.6182 | lr 0.0100
---------------------------------------------------------------------
| epoch 9 | train_acc 0.4877 train_loss 1.5725 | lr 0.0100
| epoch 9 | test_acc 0.5041 test_loss 1.5307 | lr 0.0100
---------------------------------------------------------------------
| epoch 10 | train_acc 0.5080 train_loss 1.4986 | lr 0.0100
| epoch 10 | test_acc 0.4579 test_loss 1.6603 | lr 0.0100
---------------------------------------------------------------------
| epoch 11 | train_acc 0.5290 train_loss 1.4443 | lr 0.0100
| epoch 11 | test_acc 0.5182 test_loss 1.4684 | lr 0.0100
---------------------------------------------------------------------
| epoch 12 | train_acc 0.5518 train_loss 1.3683 | lr 0.0100
| epoch 12 | test_acc 0.5145 test_loss 1.4765 | lr 0.0100
---------------------------------------------------------------------
| epoch 13 | train_acc 0.5759 train_loss 1.3115 | lr 0.0100
| epoch 13 | test_acc 0.5512 test_loss 1.3966 | lr 0.0100
---------------------------------------------------------------------
| epoch 14 | train_acc 0.5948 train_loss 1.2573 | lr 0.0100
| epoch 14 | test_acc 0.5153 test_loss 1.4676 | lr 0.0100
---------------------------------------------------------------------
| epoch 15 | train_acc 0.6159 train_loss 1.1923 | lr 0.0100
| epoch 15 | test_acc 0.6103 test_loss 1.2591 | lr 0.0100
---------------------------------------------------------------------
| epoch 16 | train_acc 0.6320 train_loss 1.1504 | lr 0.0100
| epoch 16 | test_acc 0.5942 test_loss 1.2556 | lr 0.0100
---------------------------------------------------------------------
| epoch 17 | train_acc 0.6514 train_loss 1.1079 | lr 0.0100
| epoch 17 | test_acc 0.5876 test_loss 1.2930 | lr 0.0100
---------------------------------------------------------------------
| epoch 18 | train_acc 0.6612 train_loss 1.0774 | lr 0.0100
| epoch 18 | test_acc 0.6202 test_loss 1.1998 | lr 0.0100
---------------------------------------------------------------------
| epoch 19 | train_acc 0.6735 train_loss 1.0297 | lr 0.0100
| epoch 19 | test_acc 0.6492 test_loss 1.1037 | lr 0.0100
---------------------------------------------------------------------
| epoch 20 | train_acc 0.6890 train_loss 0.9942 | lr 0.0100
| epoch 20 | test_acc 0.6727 test_loss 1.0470 | lr 0.0100
---------------------------------------------------------------------
| epoch 21 | train_acc 0.6955 train_loss 0.9585 | lr 0.0100
| epoch 21 | test_acc 0.6620 test_loss 1.0686 | lr 0.0100
---------------------------------------------------------------------
| epoch 22 | train_acc 0.7079 train_loss 0.9375 | lr 0.0100
| epoch 22 | test_acc 0.6835 test_loss 1.0248 | lr 0.0100
---------------------------------------------------------------------
| epoch 23 | train_acc 0.7182 train_loss 0.8986 | lr 0.0100
| epoch 23 | test_acc 0.6645 test_loss 1.0549 | lr 0.0100
---------------------------------------------------------------------
| epoch 24 | train_acc 0.7319 train_loss 0.8636 | lr 0.0100
| epoch 24 | test_acc 0.7079 test_loss 0.9836 | lr 0.0100
---------------------------------------------------------------------
| epoch 25 | train_acc 0.7354 train_loss 0.8479 | lr 0.0100
| epoch 25 | test_acc 0.6818 test_loss 1.0112 | lr 0.0100
---------------------------------------------------------------------
| epoch 26 | train_acc 0.7474 train_loss 0.8160 | lr 0.0100
| epoch 26 | test_acc 0.7033 test_loss 0.9536 | lr 0.0100
---------------------------------------------------------------------
| epoch 27 | train_acc 0.7545 train_loss 0.7959 | lr 0.0100
| epoch 27 | test_acc 0.7256 test_loss 0.9125 | lr 0.0100
---------------------------------------------------------------------
| epoch 28 | train_acc 0.7575 train_loss 0.7768 | lr 0.0100
| epoch 28 | test_acc 0.7273 test_loss 0.9175 | lr 0.0100
---------------------------------------------------------------------
| epoch 29 | train_acc 0.7687 train_loss 0.7512 | lr 0.0100
| epoch 29 | test_acc 0.7260 test_loss 0.8718 | lr 0.0100
---------------------------------------------------------------------
| epoch 30 | train_acc 0.7731 train_loss 0.7341 | lr 0.0100
| epoch 30 | test_acc 0.7343 test_loss 0.8861 | lr 0.0100
---------------------------------------------------------------------
| epoch 31 | train_acc 0.7822 train_loss 0.7141 | lr 0.0100
| epoch 31 | test_acc 0.7492 test_loss 0.8450 | lr 0.0100
---------------------------------------------------------------------
| epoch 32 | train_acc 0.7870 train_loss 0.6887 | lr 0.0100
| epoch 32 | test_acc 0.7517 test_loss 0.8145 | lr 0.0100
---------------------------------------------------------------------
| epoch 33 | train_acc 0.7909 train_loss 0.6792 | lr 0.0100
| epoch 33 | test_acc 0.7450 test_loss 0.8440 | lr 0.0100
---------------------------------------------------------------------
| epoch 34 | train_acc 0.7959 train_loss 0.6637 | lr 0.0100
| epoch 34 | test_acc 0.7607 test_loss 0.8006 | lr 0.0100
---------------------------------------------------------------------
| epoch 35 | train_acc 0.8023 train_loss 0.6424 | lr 0.0100
| epoch 35 | test_acc 0.7632 test_loss 0.7862 | lr 0.0100
---------------------------------------------------------------------
| epoch 36 | train_acc 0.8089 train_loss 0.6262 | lr 0.0100
| epoch 36 | test_acc 0.7613 test_loss 0.8067 | lr 0.0100
---------------------------------------------------------------------
| epoch 37 | train_acc 0.8097 train_loss 0.6231 | lr 0.0100
| epoch 37 | test_acc 0.7658 test_loss 0.7916 | lr 0.0100
---------------------------------------------------------------------
| epoch 38 | train_acc 0.8192 train_loss 0.6088 | lr 0.0100
| epoch 38 | test_acc 0.7635 test_loss 0.7985 | lr 0.0100
---------------------------------------------------------------------
| epoch 39 | train_acc 0.8192 train_loss 0.5928 | lr 0.0100
| epoch 39 | test_acc 0.7795 test_loss 0.7923 | lr 0.0100
---------------------------------------------------------------------
| epoch 40 | train_acc 0.8225 train_loss 0.5637 | lr 0.0100
| epoch 40 | test_acc 0.7735 test_loss 0.7977 | lr 0.0100
---------------------------------------------------------------------
| epoch 41 | train_acc 0.8254 train_loss 0.5689 | lr 0.0100
| epoch 41 | test_acc 0.7835 test_loss 0.7457 | lr 0.0100
---------------------------------------------------------------------
| epoch 42 | train_acc 0.8299 train_loss 0.5581 | lr 0.0100
| epoch 42 | test_acc 0.7878 test_loss 0.7349 | lr 0.0100
---------------------------------------------------------------------
| epoch 43 | train_acc 0.8309 train_loss 0.5326 | lr 0.0100
| epoch 43 | test_acc 0.7827 test_loss 0.7297 | lr 0.0100
---------------------------------------------------------------------
| epoch 44 | train_acc 0.8353 train_loss 0.5365 | lr 0.0100
| epoch 44 | test_acc 0.7897 test_loss 0.7302 | lr 0.0100
---------------------------------------------------------------------
| epoch 45 | train_acc 0.8376 train_loss 0.5215 | lr 0.0100
| epoch 45 | test_acc 0.7864 test_loss 0.7363 | lr 0.0100
---------------------------------------------------------------------
| epoch 46 | train_acc 0.8421 train_loss 0.5165 | lr 0.0100
| epoch 46 | test_acc 0.7586 test_loss 0.7887 | lr 0.0100
---------------------------------------------------------------------
| epoch 47 | train_acc 0.8467 train_loss 0.5021 | lr 0.0100
| epoch 47 | test_acc 0.7897 test_loss 0.7119 | lr 0.0100
---------------------------------------------------------------------
| epoch 48 | train_acc 0.8443 train_loss 0.5001 | lr 0.0100
| epoch 48 | test_acc 0.7998 test_loss 0.7234 | lr 0.0100
---------------------------------------------------------------------
| epoch 49 | train_acc 0.8517 train_loss 0.5065 | lr 0.0100
| epoch 49 | test_acc 0.7760 test_loss 0.7241 | lr 0.0100
---------------------------------------------------------------------
| epoch 50 | train_acc 0.8413 train_loss 0.4406 | lr 0.0100
| epoch 50 | test_acc 0.8098 test_loss 0.7949 | lr 0.0100
---------------------------------------------------------------------

4.2 模型评估

代码如下：

model.eval()  # 切换为测试模式
test_acc, test_loss = test(valid_dataloader, model, criterion)
print('模型准确率为：{:5.4f}'.format(test_acc))

打印输出：

模型准确率为：0.7842

五、模型调优

5.1 尝试修改了优化器为Adam，结果变得更差了，相比之下，本文任务更适合使用SGD优化器

5.2 增加了epoch数，从原来的50增加到了80

模型准确率达到了80.2%

5.3 还可以增加动态学习率

总结

在上周的基础上进一步修改了transformer进行分类的代码，对transformer有了进一步的了解