A/B test on ads is the art of choosing the best advertisement that optimizes your goal (number of clicks, likes, etc). For example, if you change a simple thing like a filter in your pictures you will drive more traffic to your links.

广告的A / B测试是一种选择最佳广告的艺术，该广告可以优化您的目标(点击次数，喜欢的次数等)。 例如，如果您更改图片中的过滤器之类的简单内容，则会为链接带来更多流量。

In this post we will see how to do evaluate by yourself the efficacy of your own A/B test using Python and Bayesian Statistics.

在这篇文章中，我们将看到如何使用Python和贝叶斯统计自己评估自己的A / B测试的效率。

Article TL;DR for the Busy Data Scientist

Article TL; DR for the Busy Data Scientist

1. I ran two twitter ads. The one with a cold filter outperformed the one with warm filter.
   我投放了两个Twitter广告。 带有冷滤网的滤网的性能优于带暖滤网的滤网的性能。

2. Skip to section “The Code (starting from the end)” if you just want to copy and past some Python code to try your own A/B Test.

   如果您只想复制并粘贴一些Python代码来尝试自己的A / B测试，请跳至“ 代码(从末尾开始) ”部分。

3. Skip to the section “Putting all together” if you want to see my final result.

   如果要查看我的最终结果，请跳至“ 全部放在一起 ”部分。

4. Read it through if you want to learn how to do and why it works A/B Tests using Bayesian statistics.
   如果您想学习使用贝叶斯统计信息的方法以及为什么要使用A / B测试，请通读它。

问题 (The Problem)

Given two ads (A and B), which one has the highest Click-Through-Rate (CTR)? From a statistical point of view, the problem is to learn the unobservable TRUE CTR parameter from the observed data of impressions (ad views) and clicks. Just to avoid confusion, remember that CTR is calculated as the number of clicks divided by the number of impressions:

给定两个广告(A和B)，哪个具有最高点击率(CTR)？ 从统计的角度来看，问题是要从所观察到的印象(广告视图)和点击数据中了解到不可观察的TRUE CTR参数。 为避免混淆，请记住，点击率的计算方法是点击次数除以展示次数：

From this simple formula you might be thinking:

从这个简单的公式中，您可能会想到：

“If the equation is really just a simple division the only thing I need to do is to get the data from the ad performance , do the simple division and the higher number is the best Ad. DONE!?”

“如果等式实际上只是一个简单的除法运算，那么我唯一需要做的就是从广告效果中获取数据，然后进行简单的除法运算，数值越大越好。 完成！？”

Well, not really.

好吧，不是真的。

Say you have only one impression and one click, that is a CTR of 100%, but you should not be assuming that your TRUE CTR is actually 100% from one single view/impression. It most likely is not 100%, by the way. To put it simply, the observed CTR alone cannot tell us the performance of an ad. We need more, we need more data and we still need some Bayesian statistics. But before let me setup our our ad campaign.

假设您只有一次展示和一次点击，即100％的点击率，但您不应假设您的TRUE CTR实际上是从单个视图/展示获得的100％。 顺便说一句，它很可能不是100％。 简而言之，仅观察到的点击率并不能告诉我们广告的效果。 我们需要更多，我们需要更多数据，我们仍然需要一些贝叶斯统计数据。 但是在让我设置广告系列之前。

设置 (The Setup)

Let us explorer the problem a little bit further. For that matter, I ran two real ads in Twitter with real money (I spent $20 USD in case you are curious). The difference between both ads was only the filter in the images. One image had a cold filter and the other had a warm filter (default filters in my smart phone). The ad was for an Affiliate Amazon link to the book ‘Designing Data Intensive Applications ’(https://amzn.to/3iycLi6).

让我们进一步探讨该问题。 为此，我在Twitter上用真钱投放了两个真实的广告(如果您好奇的话，我花了20美元)。 这两个广告之间的区别只是图片中的过滤器。 一个图像有一个冷滤镜，另一个图像有一个暖滤镜(我的智能手机中的默认滤镜)。 该广告是针对一个亚马逊附属机构的链接，该链接指向《 设计数据密集型应用程序 》( https://amzn.to/3iycLi6 )一书。

The question is ‘Which filter maximizes my the Click-Through-Rate?’, i.e., which image makes the Amazon link most likely to be clicked? Here are the ads side by side:

问题是“哪个过滤器会最大化我的点击率？”，即哪个图像使Amazon链接最有可能被点击？ 以下是这些广告的并排：

After running both ads for a few hours I got the following impressions and clicks:

将两个广告都投放了几个小时后，我得到了以下印象和点击：

Ad A (cold filter): 190 impressions, 13 clicks, CTR 0.068 (6.8%)

广告A(冷过滤器)：190次展示，13次点击，点击率0.068(6.8％)

Ad B(warm filter): 143 impressions, 9 clicks, CTR 0.062 (6.0%)

广告B(热过滤器)：143次展示，9次点击，点击率0.062(6.0％)

From the data, we can see that A’s observed CTR is higher than B’s observed CTR (6.8% > 6.0%). The remaining of this post we will answer the following two questions:

从数据中我们可以看到，A的观察到的点击率高于B的观察到的点击率 (6.8％> 6.0％)。 在本文的其余部分，我们将回答以下两个问题：

1. Can we conclude that A’s real CTR is higher than B’s real CTR?
   我们是否可以得出结论，A的实际点击率高于B的实际点击率？
2. If we were to accept it is (or it is not), what is the probability that we are right or wrong?
   如果我们接受它是(或不是)，那么我们是对是错的概率是多少？

One way to stretch this question to the extreme would be the following hypothetical situation: Assume that ad A had 1 view and 0 clicks while ad B had 1 view and 1 click. Their observed CTR are 0% and 100%, but no one would say that ad B performs better than ad A just from these data points. Let’s see how to estimate our CTRs while our ad campaign is running.

将这个问题扩展到极端的一种方法是，在以下假设情况下进行：假设广告A具有1个视图和0次点击，而广告B具有1个视图和1次点击。 他们观察到的点击率分别为0％和100％，但没有人会说从这些数据点来看，广告B的效果要好于广告A。 让我们看看如何在广告系列投放期间估算点击率。

代码(从头开始) (The Code (starting from the end))

It will be easier to understand where we are going if we start at the end. So, let’s have a look at some code. Do not worry if you do not understand every line yet, the main point now is the plot the code produces, also I will walk you through the code later.

如果我们从头开始，将会更容易理解我们的发展方向。 因此，让我们看一些代码。 如果您还不了解每一行，请不要担心，现在的重点是代码产生的图形，稍后我将引导您完成代码。

Running this code in your notebook, you should get the following plot:

在笔记本中运行此代码，应该得到以下图表：

This plot shows the most likely values for each CTR given the observed data. Since the regions with the most likely values (the Highest Density intervals above) we cannot say that one ad is better than the other only, at least not with only this data. For comparison, see the same code if it were ran with the following FAKE data:

此图显示了给定观察到的数据的每个点击率的最可能值。 由于具有最可能值的区域(上述最高密度区间)，我们不能说一个广告仅比另一个广告好，至少不能仅凭此数据。 为了进行比较，如果它与以下FAKE数据一起运行，请参见相同的代码：

We will perform our A/B test in these 3 simple steps. We will:

我们将通过以下三个简单步骤来执行A / B测试。 我们会：

1. Gather the number of impressions and clicks of both ads;
   收集两个广告的印象数和点击数；
2. Plot the distribution of the most likely values for A and B’s CTR;
   绘制A和B点阅率最可能值的分布；
3. Compare their highest density intervals and decide depending on the intervals’ overlap.
   比较它们的最高密度间隔，并根据间隔的重叠进行确定。

If you paid attention to the Python code above (no judgement if you did not), you saw that we used a few keywords, most importantly prior, posterior and beta. In the next section we will jump to the math and explain why the previous Python code works. So buckle up, enter the Bayesian statistics.

如果您注意上面的Python代码(如果没有，则无法判断)，您会看到我们使用了一些关键字，最重要的是before，posterior和beta。 在下一节中，我们将跳至数学并解释为什么以前的Python代码可以工作。 系好安全带，输入贝叶斯统计数据。

A / B测试背后的数学 (The Math behind A/B Test)

先验 (The Prior)

The goal here is to estimate the real CTR for an ad given the observed data. Since seeing the REAL CTR would imply serving an ad to every single user of Twitter, it is monetarily (and often practically) impossible to do that. So we will need to make our 20 bucks worth it. The estimation will come with an uncertainty that we will be able to quantify, that is also important for our decision making when it comes to stop an ad and increase the budget on others.

这里的目标是根据观察到的数据估算广告的实际点击率。 由于看到REAL CTR意味着向Twitter的每个用户投放广告，因此从金钱上(通常实际上)是不可能做到的。 因此，我们需要使我们的20美元值得。 估算将带有我们可以量化的不确定性，这对于我们停止广告并增加其他广告的预算时的决策也很重要。

Let’s call θ the parameter we want to estimate (in this case the CTR of an ad) and p(θ) the probability distribution for θ. In a previous article, I talked about p(θ) being discrete or sometimes uniform, but here we will see that using the beta distribution to describe p(θ) makes sense and it convenient for computation.

我们将θ称为我们要估算的参数(在这种情况下，是广告的点击率)，而将p(θ)称为θ的概率分布。 在上一篇文章中 ，我谈到了p(θ)是离散的或有时是均匀的，但是在这里我们将看到使用beta 分布来描述p(θ)是有意义的，并且便于计算。

One way to think about p(θ) is to think as if it is our belief on the possible values of θ. For example, say we are talking about a problem that we have no prior knowledge, and all possible values of θ are equally likely. In this case, note that beta(1,1) describes our knowledge (or lack of) for possible values of θ. By the same token, if we were to use θ to describe our prior belief on the probability of a coin flip to be be heads, the distribution beta(25,25) would be a good candidate, as it is centered around 50% , but we would still allow some room for small biases (HDI spans within 0.4~0.6). See figure below:

思考p(θ)的一种方法是，认为它似乎是我们对θ可能值的信念。 例如，假设我们正在谈论的是一个我们没有先验知识的问题，并且所有可能的θ值都是同等可能。 在这种情况下，请注意beta(1,1)描述了我们对θ可能值的认识(或缺乏)。 出于同样的原因，如果我们使用θ来描述我们对硬币翻转为正面概率的先前信念，则分布beta(25,25)将是一个很好的候选者，因为它的中心在50％左右，但是我们仍然会留出一些小的偏差空间(HDI范围在0.4〜0.6之间)。 参见下图：

The parameters a and b can seem a little artificial, but away to describe a particular prior of a problem is via the prior mean m and the number of events n observed in the past. With m and n one can find the parameters a and b as follows:

参数a和b看起来有些虚假，但是描述问题的特定先验条件是通过先验均值m和过去观察到的事件数n来描述。 使用m和n可以找到参数a和b ，如下所示：

It is important to note that a beta distribution might not be the best fit for the problem, for that we could use model selection via Bayes Factor, a topic way beyond this article. Historically, the beta distribution is used because it is very easy to compute, specially when we use it with the bayesian rule given some new data. Let’s have a look on that next.

需要注意的是AB 埃塔分布可能不是问题的最佳契合，对于我们可以通过贝叶斯因子，一个主题的方式这篇文章以后使用模式的选择是很重要的。 从历史上看，使用beta分布是因为它非常易于计算，尤其是当我们将贝叶斯规则与给定一些新数据一起使用时。 接下来让我们看一下。

后部 (The Posterior)

The way we will do our A/B test will be assuming the prior distribution for our parameter θ (the real click-through-rate of the ad) to be beta(1,1), i.e., we will assume total lack of prior knowledge and set any possible value of θ to be equally likely. Then, we will update our p(θ) given our observed data, for that we use bayes rule (For an introduction to the Bayes’ Rule see my previous Article here):

我们进行A / B测试的方式将假设参数θ(广告的实际点击率)的先验分布为beta(1,1) ，即，我们将假设完全没有先验分布知道并将任何可能的θ值设置为同等可能。 然后，我们将给予我们的观测数据来更新我们的P(θ)，对于我们使用贝叶斯法则(对于一个介绍贝叶斯法则看我以前的文章在这里 )：

Our goal will be to calculate the posterior distribution p(θ|D) for our advertisement given the number of impressions and clicks the ad received.

我们的目标是给定印象数和点击数，计算出广告的后验分布p(θ| D)。

First, note that we are assuming that given an ad, the probably of clicking it when it is seen is equal to θ and the probability of not being clicked is equal to 1-θ, that is what we call a Bernoulli distribution.

首先，请注意，我们假设给定一个广告，看到它时点击它的可能性等于θ，未被点击的可能性等于1-θ，这就是我们所说的伯努利分布。

Result: After N impressions and z clicks, if the probability of the data (N, z) is a Bernoulli distribution, and if the prior distribution of θ is a beta(a,b) distribution, then the following holds to be true:

结果：在N次展示和z次点击之后，如果数据( N，z )的概率是伯努利分布，并且如果θ的先前分布是beta(a，b)分布，则以下条件成立：

For a detailed proof of this result I suggest John Kruschke’s “Doing Bayesian Data Analysis” (https://amzn.to/345ouR2), or ask me in Twitter (@solvingthehuman).

有关此结果的详细证明，我建议约翰·克鲁施克(John Kruschke)的“做贝叶斯数据分析”( https://amzn.to/345ouR2 )，或在Twitter(@solvingthehuman)上问我。

放在一起 (Putting all together)

With the equation from last section we can say that after N impressions and z clicks in an ad, the updated distribution of the ad’s CTR (the θ parameter) is given by:

使用上一节中的公式，我们可以说，在广告中获得N次展示和z次点击后，广告CTR的更新分布(θ参数)由下式给出：

beta(1+z, N-z+1)

beta(1 + z，N-z + 1)

Let’s see how this look like with some real data along with the complete Python code so you can do your own A/B tests:

让我们看看一些真实数据以及完整的Python代码的样子，以便您可以进行自己的A / B测试：

结论 (Conclusion)

In this article we saw how to use the beta distribution and Python to quickly decide which ad is performing better in an A/B test campaign.

在本文中，我们了解了如何使用Beta发行版和Python快速确定哪个广告在A / B测试广告系列中的效果更好。

Also, we saw that if you are planning to sell data science book on Twitter you might be better off using the cold filter for your pictures. I am very curious to know if this result holds up to other people.

此外，我们看到，如果您打算在Twitter上出售数据科学书籍，那么最好使用冷滤器来处理图片。 我很想知道这个结果是否能胜任其他人。

Let me know of your own A/B tests! Did you have similar results as mine? I am in Twitter @solvingthehuman.

让我知道您自己的A / B测试！ 您有和我类似的结果吗？ 我在Twitter @solvingthehuman。

更多？ (More?)

If you want to read:

如果您想阅读：

• An introduction about Bayesian Rule — Go here

  关于贝叶斯规则的介绍-转到此处

• The advantages of Bayesian over other methods — Go here

  围棋^ h -贝叶斯优于其他方法的优点ERE