torch.nn.functional.normalize
计算方式
v = v max ( ∥ v ∥ p , ϵ ) \mathbf{v} = \frac{\mathbf{v}}{\max\left(\|\mathbf{v}\|_p, \epsilon\right)} v=max(∥v∥p,ϵ)v
pytorch例子
a = torch.arange(12).reshape(3,4).float()
torch.nn.functional.normalize(a, dim=-1)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],[0.3563, 0.4454, 0.5345, 0.6236],[0.4182, 0.4704, 0.5227, 0.5750]])
'''
这样除一下就相等了
a = torch.arange(12).reshape(3,4).float()
torch.nn.functional.normalize(a, dim=-1)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],[0.3563, 0.4454, 0.5345, 0.6236],[0.4182, 0.4704, 0.5227, 0.5750]])
'''
a / a.norm(dim=-1,keepdim=True)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],[0.3563, 0.4454, 0.5345, 0.6236],[0.4182, 0.4704, 0.5227, 0.5750]])
'''
nn.norm
计算方式
可以通过p参数指定是pro(Frobenium norm，其实就是平方和开根号)还是nuc（叫nuclear norm，其实就是SVD分解的奇异值，也就是Singular Value）
pytorch例子
a = torch.arange(12).reshape(3,4).float()
a.norm(dim=-1, keepdim=True)
'''
tensor([[ 3.7417],[11.2250],[19.1311]])
'''
LayerNorm
计算方式
分子减均值，分母对方差开根号，更多细节参考 超细节的BatchNorm/BN/LayerNorm/LN/SyncLN/ShuffleBN/RMSNorm知识点
pytorch例子
a = torch.arange(12).reshape(3,4).float()
(a - a.mean(dim=-1, keepdims=True)) / torch.sqrt(a.var(dim=-1, keepdims=True, correction=0))
'''
tensor([[-1.3416, -0.4472, 0.4472, 1.3416],[-1.3416, -0.4472, 0.4472, 1.3416],[-1.3416, -0.4472, 0.4472, 1.3416]])
'''
