Every beginner in Machine Learning starts by studying what regression means and how the linear regression algorithm works. In fact, the ease of understanding, explainability and the vast effective real-world use cases of linear regression is what makes the algorithm so famous. However, there are some situations to which linear regression is not suited. In this article, we will see what these situations are, what the kernel regression algorithm is and how it fits into the scenario. Finally, we will code the kernel regression algorithm with a Gaussian kernel from scratch. Basic knowledge of Python and numpy is required to follow the article.

e。通过学习什么回归方式，以及如何进行线性回归算法的工作非常初学者在机器学习开始。 实际上，算法的易懂性，可解释性和广泛有效的线性回归实际使用案例就是使该算法如此出名的原因。 但是，在某些情况下线性回归不适合。 在本文中，我们将了解这些情况，内核回归算法是什么以及它如何适合该场景。 最后，我们将从头开始使用高斯内核对内核回归算法进行编码。 阅读本文需要具备Python和numpy的基础知识。

线性回归简要回顾 (Brief Recap on Linear Regression)

Given data in the form of N feature vectors x=[x₁, x₂, …, xₙ] consisting of n features and the corresponding label vector y, linear regression tries to fit a line that best describes the data. For this, it tries to find the optimal coefficients cᵢ, i∈{0, …, n} of the line equation y = c₀ + c₁x₁+c₂x₂+…+cₙxₙ usually by gradient descent with the model accuracy measured on the RMSE metric. The equation obtained is then used to predict the target yₜ for new unseen input vector xₜ.

在N个特征向量x = [X₁中，x₂，...，Xₙ]组成的n个特征和对应的标签向量y，线性回归尝试的形式给定的数据，以适应线路最能描述的数据。 对于这一点，它试图找到最佳系数cᵢ， 我 ∈{0，...，N}的直线方程Y = C₀+ C₁X₁+ C₂X₂+ ... + CₙX通常由梯度ₙ的下降，并以RMSE指标衡量的模型准确性。 然后将获得的方程式用于预测新的看不见的输入向量x的目标y 。

Linear regression is a simple algorithm that cannot model very complex relationships between the features. Mathematically, this is because well, it is linear with the degree of the equation being 1, which means that linear regression will always model a straight line. Indeed, this linearity is the weakness of the linear regression algorithm. Why?

线性回归是一种简单的算法，无法对要素之间的非常复杂的关系建模。 从数学上讲，这是因为它很好，它在方程的次数为1时是线性的，这意味着线性回归将始终对直线建模。 确实，这种线性是线性回归算法的弱点。 为什么？

Well, let’s consider a situation where our data doesn’t have the form of a straight line: let’s take data generated using the function f(x) = x³. If we use linear regression to fit a model to this data, we will never get anywhere close to the true cubic function because the equation for which we are finding the coefficients does not have a cubic term! So, for any data not generated using a linear function, linear regression is very likely to underfit. So, what do we do?

好吧，让我们考虑一下数据不呈直线形式的情况：让我们看一下使用函数f(x)=x³生成的数据。 如果我们使用线性回归将模型拟合到该数据，我们将永远无法接近真正的三次函数，因为我们要为其找到系数的方程式没有三次项！ 因此，对于未使用线性函数生成的任何数据，线性回归很可能不适合。 那么我们该怎么办？

We can use another type of regression called polynomial regression which tries to find optimal coefficients of a (as the name suggests) polynomial equation with the degree of the equation being n, n⪈1. However, with polynomial regression another problem arises: as a data analyst, you cannot know what the degree of the equation should be so that the resulting equation fits best to the data. This can only be determined by trial and error which is made more difficult by the fact that above degree 3, the model built using polynomial regression is difficult to visualize.

我们可以使用一个叫做多项式回归另一种类型的回归它试图找到一个最佳系数(顾名思义)多项式方程与方程为N，N-⪈1的程度。 但是，使用多项式回归会出现另一个问题：作为数据分析人员，您不知道方程式的度数以使所得方程式最适合数据。 这只能通过反复试验来确定，而由于三次以上的事实，使用多项式回归建立的模型难以可视化，因此更加困难。

This is where kernel regression can come to the rescue!

这是内核回归可以解决的地方！

什么是内核回归？ (What is Kernel Regression?)

Seeing the name, you may ask that if ‘linear’ in linear regression meant a linear function and ‘polynomial’ in polynomial regression meant a polynomial function, what does ‘kernel’ mean? Turns out, it means a kernel function! So, what is a kernel function? Simply, it is a similarity function that takes two inputs and spits out how similar they are. We will see shortly how a kernel function is used in kernel regression.

看到名称，您可能会问，如果线性回归中的“线性”表示线性函数，而多项式回归中的“多项式”意味着多项式函数，那么“内核”是什么意思？ 原来，这意味着内核功能！ 那么，什么是内核函数？ 简而言之，它是一个相似度函数，它接受两个输入并吐出它们的相似度。 我们很快将看到在内核回归中如何使用内核函数。

Now about kernel regression. Unlike linear and polynomial regression in which the optimal parameter vector c=[c₁, c₂, …, cₙ] needs to be learnt, kernel regression is non-parametric, meaning that it calculates the target yₜ by performing computations directly on the input xₜ.

现在介绍内核回归。 与线性和多项式回归，其中最佳参数矢量c = [C₁，C₂，...，Cₙ]需要学习，核回归是非参数，这意味着它计算目标ÿₜ通过直接在执行计算输入xₜ。

How?

怎么样？

Given data points (xᵢ, yᵢ) Kernel Regression goes about predicting by first constructing a kernel k for each data point xᵢ. Then for a given new input xₜ, it computes a similarity score with each xᵢ (given by xᵢ-xₜ) using the kernel ; the similarity score acts as a weight wᵢ that represents the importance of that kernel (and corresponding label yᵢ) in predicting the target yₜ. The prediction is then obtained by multiplying the weight vector w= [w₁, w₂, …, wₙ] with the label vector y= [y₁, y₂, …, yₙ].

给定数据点( x ， y )内核回归通过首先为每个数据点x构造一个内核k来进行预测。 然后，对于一个给定的新的输入xₜ，它计算一个相似性得分与每个xᵢ使用内核(由Xᵢ-Xₜ给出); 相似度得分作为权重wᵢ，表示内核的重要性(和相应的标记yᵢ)在预测对象物Yₜ。 然后通过将权重向量w = [ w₁ ， w 2，… ， wₙ]乘以标记向量y = [ y y ， y 2，… ， yₙ]来获得预测。

Now, there can be different kernel functions which give rise to different types of kernel regressions. One such type is the Gaussian Kernel Regression in which the shape of the constructed kernel is the Gaussian curve also known as the bell-shaped curve. In the context of Gaussian Kernel Regression, each constructed kernel can also be viewed as a normal distribution with mean value xᵢ and standard deviation b. Here, b is a hyperparameter that controls the shape (in particular, the width of the Gaussian curve in Gaussian kernels) of the curve. The equation for the Gaussian kernel k is given below. Notice the similarity between this equation and that of the Gaussian (also called normal) distribution.

现在，可以有不同的内核函数，从而导致不同类型的内核回归。 一种这样的类型是高斯核回归，其中构造的核的形状是高斯曲线，也称为钟形曲线。 在高斯核回归的情况下，各构造内核还可以被看作是与平均值xᵢ和标准偏差B A正态分布。 在此，b是控制曲线的形状(特别是高斯核中的高斯曲线的宽度)的超参数。 高斯核k的等式在下面给出。 请注意，该方程式与高斯分布(也称为正态分布)的相似性。

We will code this type of kernel regression next.

接下来，我们将编码这种类型的内核回归。

编码高斯核回归 (Coding Gaussian Kernel Regression)

We will first look at the case of a one-dimensional feature vector and then extend it to n dimensions.

我们将首先看一维特征向量的情况，然后将其扩展到n维。

from scipy.stats import norm
import numpy as np 
import pandas as pd
import matplotlib.pyplot as plt
import mathclass GKR:
    
    def __init__(self, x, y, b):
        self.x = x
        self.y = y
        self.b = b
    
    '''Implement the Gaussian Kernel'''
    def gaussian_kernel(self, z):
        return (1/math.sqrt(2*math.pi))*math.exp(-0.5*z**2)
    
    '''Calculate weights and return prediction'''
    def predict(self, X):
        kernels = [self.gaussian_kernel((xi-X)/self.b) for xi in self.x]
        weights = [len(self.x) * (kernel/np.sum(kernels)) for kernel in kernels]
        return np.dot(weights, self.y)/len(self.x)

We define a class for Gaussian Kernel Regression which takes in the feature vector x, the label vector y and the hyperparameter b during initialization. Inside the class, we define a function gaussian_kernel() that implements the Gaussian kernel. You can see that we just write out the mathematical equation as code. Next, we define the function predict() that takes in the feature vector xₜ (referred to in code as X) whose target value has to be predicted. Inside the function, we construct kernels for each xᵢ, calculate the weights and return the prediction, again by plugging in the mathematical equations into code as-is.

我们为高斯核回归定义了一个类，该类在初始化过程中接受特征向量x，标签向量y和超参数b 。 在类内部，我们定义了一个实现高斯内核的函数gaussian_kernel() 。 您可以看到我们只是将数学方程写为代码。 接下来，我们定义函数predict() ，该函数接受必须预测目标值的特征向量x X (在代码中称为X )。 在函数内部，我们为每个x construct构造内核，计算权重并返回预测，再次将数学方程式直接插入代码中。

Now, let’s pass in some dummy data and see the prediction that is output. We predict the value for xₜ = 50 (by ignoring for demonstration purposes that it is already present in training data)

现在，让我们传递一些虚拟数据，看看输出的预测。 我们预测值对于xₜ= 50(通过忽略用于演示目的，它已经存在于训练数据)

gkr = GKR([10,20,30,40,50,60,70,80,90,100,110,120], [2337,2750,2301,2500,1700,2100,1100,1750,1000,1642, 2000,1932], 10)
gkr.predict(50)

This gives us an output of 1995.285

这样我们得到的输出是1995.285

Now, let’s extend the code for the case of n dimensional feature vectors. The only modification we need to make is in the similarity score calculation. Instead of obtaining the difference between xᵢ and xₜ, we calculate the similarity score in the n dimensional case as the Euclidean distance ||xᵢ-xₜ|| between them. Note that for the purposes of handling n dimensional vectors, we use numpy wherever needed.

现在，让我们针对n维特征向量的情况扩展代码。 我们需要做的唯一修改是相似度分数计算。 代替获得Xᵢ并且xₜ，我们计算在n维的情况作为欧几里得距离的相似性得分||之间的差的 Xᵢ-Xₜ|| 它们之间。 请注意，出于处理n维向量的目的，我们在需要时使用numpy。

from scipy.stats import multivariate_normal
'''Class for Gaussian Kernel Regression'''
class GKR:
    
    def __init__(self, x, y, b):
        self.x = np.array(x)
        self.y = np.array(y)
        self.b = b
    
    '''Implement the Gaussian Kernel'''
    def gaussian_kernel(self, z):
        return (1/np.sqrt(2*np.pi))*np.exp(-0.5*z**2)
    
    '''Calculate weights and return prediction'''
    def predict(self, X):
        kernels = np.array([self.gaussian_kernel((np.linalg.norm(xi-X))/self.b) for xi in self.x])
        weights = np.array([len(self.x) * (kernel/np.sum(kernels)) for kernel in kernels])
        return np.dot(weights.T, self.y)/len(self.x)

Again, let’s pass in some 2D dummy data and predict for xₜ = [20, 40].

再次，让我们传递一些2D虚拟数据，并预测x = [20，40]。

gkr = GKR([[11,15],[22,30],[33,45],[44,60],[50,52],[67,92],[78,107],[89,123],[100,137]], [2337,2750,2301,2500,1700,1100,1000,1642, 1932], 10)
gkr.predict([20,40])

We get yₜ = 2563.086.

我们得到y = 2563.086。

The extended code (including for visualizations) for this article can be found on GitHub and Kaggle.

可以在GitHub和Kaggle上找到本文的扩展代码(包括可视化)。

结论 (Conclusion)

We saw where and why linear regression and polynomial regression cannot be used and with that background understood the intuition behind and the working of kernel regression and how it can be used as an alternative. We went into the details of the Gaussian kernel regression and coded it from scratch in Python by simply plugging in the mathematical equations to code.

我们看到了不能在何处以及为什么不能使用线性回归和多项式回归，并且在这种背景下了解了内核回归的直觉和工作原理以及如何将其用作替代方法。 我们研究了高斯核回归的细节，并通过简单地将数学方程式插入代码中，从头开始用Python对其进行了编码。