TLDR: Designing metrics that help you make better decisions is hard. In The Times and The Sunday Times newsrooms, we have spent a lot of time trying to tackle three particular problems.

TLDR ：设计度量标准以帮助您做出更好的决策非常困难。 在《泰晤士报》和《星期日泰晤士报》的新闻编辑室中，我们花费了大量时间来尝试解决三个特定问题。

• How do we put metrics into context, so the people who use them can quickly understand if a value is good, bad or neither?

  我们如何将指标放在上下文中 ，以便使用它们的人可以快速了解值是好是坏？

• How do we account for uncertainty in metrics, so people don’t waste time agonising over insignificant differences?

  我们如何处理指标的不确定性 ，这样人们就不会浪费时间为微不足道的差异而苦恼？

• How do we focus on the decisions that people should consider making when confronted with these metrics?

  当面对这些指标时， 我们如何专注于人们应该考虑做出的决策？

None of these are novel problems, but they are rarely accounted for in web analytics tools, which present data in ways that — for us at least — are often meaningless without applying a lot of time and effort to interpret them. We’ve talked about some of this before but revisit the key problems here and will delve deeper into the practical aspects of trying to solve these in future posts.

这些都不是新颖的问题，但是它们很少在Web分析工具中得到解释，该工具以至少对我们而言至少没有意义的方式显示数据，而无需花费大量时间和精力来解释它们。 我们之前已经讨论过其中的一些问题，但是在这里重新讨论了关键问题，并将在以后的文章中深入探讨尝试解决这些问题的实际方面。

试图用数据做决定 (Trying to make decisions with data)

“If a measurement matters at all, it is because it must have some conceivable effect on decisions and behaviour. If we can’t identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value”

“如果一项度量根本很重要，那是因为它必须对决策和行为产生可想像的影响。 如果我们无法确定可能会受到建议的测量结果影响的决策以及它如何改变这些决策，那么该测量结果就毫无价值。”

Douglas W. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business, 2007

道格拉斯·哈伯德(Douglas W. Hubbard)，《如何衡量一切：在企业中发现无形资产的价值》，2007年

Like many digital businesses we use web analytics tools that measure how visitors interact with our websites and apps. These tools provide dozens of simple metrics, but in our experience their value for informing a decision is close to zero without first applying a significant amount of time, effort and experience to interpret them.

像许多数字企业一样，我们使用网络分析工具来衡量访问者如何与我们的网站和应用程序互动。 这些工具提供了数十种简单的指标，但是根据我们的经验，在没有首先花费大量时间，精力和经验来解释它们的情况下，它们用于告知决策的价值几乎为零。

Ideally we would like to use web analytics data to make inferences about what stories our readers value and care about. We can then use this to inform a range of decisions: what stories to commission, how many articles to publish, how to spot clickbait, which headlines to change, which articles to reposition on the page, and so on.

理想情况下，我们希望使用网络分析数据来推断读者喜欢并关心的故事。 然后，我们可以使用它来告知一系列决策：要委托的故事，要发布的文章数，如何发现点击诱饵，要更改的标题，要在页面上重新放置的文章等等。

Finding what is newsworthy can not and should not be as mechanistic as analysing an e-commerce store, where the connection between the metrics and what you are interested in measuring (visitors and purchases) is more direct. We know that — at best — this type of data can only weakly approximate what readers really think, and too much reliance on data for making decisions will have predictable negative consequences. However, if there is something of value the data has to say, we would like to hear it.

查找具有新闻价值的内容不能也不应该像分析电子商务商店那样机械化，在电子商务商店中，指标与您想要衡量的内容(访问者和购买者)之间的联系更加直接。 我们知道，充其量来说，此类数据只能弱化读者的真实想法，过多地依赖数据进行决策将产生可预测的负面影响。 但是，如果数据需要说些有价值的话，我们希望听到它。

Unfortunately, simple web analytics metrics fail to account for key bits of context that are vital if we want to understand if their values are higher or lower than what we should expect (and therefore interesting).

不幸的是，简单的网络分析指标无法说明关键的上下文环境 ，如果我们想了解它们的值是高于还是低于我们期望的值(因此很有趣)，这是至关重要的。

Moreover, there is inherent uncertainty in the data we are using, and even if we can tell whether the value is higher or lower than expected, it is difficult to tell whether this is just down to chance.

而且，我们使用的数据存在固有的不确定性 ，即使我们可以判断该值是高于还是低于预期，也很难判断这是偶然的。

Good analysts, familiar with their domain often get good at doing the mental gymnastics required to account for context and uncertainty, so they can derive the insights that support good decisions. But doing this systematically when presented with a sea of metrics is rarely possible or the best use of an analyst’s valuable sense-making skills. Rather than all their time being spent trying to identify what is unusual, it would be better if their skills could be applied to learning why something is unusual or deciding how we might improve things. But if all of our attention is focused on the lower level what questions, we never get to the why or how questions — which is where we stand a chance of getting some value from the data.

熟悉其领域的优秀分析人员通常会擅长于进行考虑上下文和不确定性的心理体操，因此他们可以得出支持良好决策的见解。 但是，在提供大量指标的情况下，系统地执行此操作几乎是不可能的，或者很难最大限度地利用分析师的有价值的判断能力。 不是所有花费的时间试图找出什么是不寻常的，它会更好，如果他们的技能可以应用到学习为什么有些不寻常或决定我们如何改善的事情。 但是，如果我们所有的注意力都集中在较低的水平有什么问题，我们永远到不了为什么或如何的问题-这是我们的立场充分利用数据的一些价值的机会。

语境 (Context)

“The value of a fact shrinks enormously without context”

“事实的价值在没有上下文的情况下会大大缩小”

Howard Wainer, Visual Revelations: Graphical Tales of Fate and Deception from Napoleon Bonaparte to Ross Perot, 1997

霍华德·威纳(Howard Wainer)，《 视觉启示：从拿破仑·波拿巴到罗斯·佩罗的命运与欺骗的故事集》 ，1997年

Take two metrics that we would expect to be useful — how many people start reading an article (we call this readers), and how long they spend on it (we call this the average dwell time). If the metrics worked as intended, they could help us identify the stories our readers care about, but in their raw form, they tell us very little about this.

采取两个我们期望会有用的指标-多少人开始阅读一篇文章(我们称为读者 )，以及他们在该文章上花费多长时间(我们将其称为平均停留时间 )。 如果这些指标能够按预期工作，则可以帮助我们确定读者所关心的故事，但是以原始形式，他们对此所知甚少。

• Readers: If an article is in a more prominent position on the website or app, more people will see it and click on it.

  读者 ：如果文章在网站或应用程序中的位置更为重要，则会有更多的人看到并单击它。

• Dwell time: If an article is longer, on average, people will tend to spend more time reading it.

  停留时间：平均而言，如果一篇文章较长，人们会倾向于花更多的时间阅读它。

Counting the number of readers tells us more about where an article was placed, and dwell time more about the length of the article than anything meaningful.

计算读者数量可以告诉我们更多关于文章放置在何处的信息，而停留时间更多关于文章的长度而不是有意义的事情。

It’s not just length and position that matter. Other context such as the section, the day of the week, how long since it was published, and whether people are reading it on our website or apps all systematically influence these numbers. So much so, that we can do a reasonable job of predicting how many readers an article will get and how long they will spend on it by only looking at its context, and completely ignoring the content of the article.

重要的不只是长度和位置。 其他内容(例如，部分，一周中的一天，自发布之日起多长时间以及人们是否在我们的网站或应用上阅读它)都会系统地影响这些数字。 如此多，以至于我们仅通过查看文章的上下文并完全忽略文章的内容，就可以做出合理的工作来预测一篇文章将获得多少读者以及他们将花多少时间。

From this perspective, articles are a victim of circumstance, and the raw metrics we see in so many dashboards tell us more about their circumstances than anything more meaningful — it’s all noise and very little signal.

从这个角度来看，文章是环境的受害者，我们在许多仪表板中看到的原始指标向我们介绍了他们的处境，而不是更有意义的事，这全都是噪音，很少有信号。

Knowing this, what we really want to understand is how much better or worse an article did than we would expect, given that context. In our newsroom, we do this by turning each metric (readers, dwell time and some others) into an index that compares the actual metric for an article to it’s expected value. We score it on a scale from 1 to 5, where 3 is expected, 4 or 5 is better than expected and 1 or 2 is worse than expected.

知道这一点，我们真正想了解的是，在这种情况下，一篇文章比我们预期的好还是坏。 在我们的新闻编辑室中，我们通过将每个指标(读者，停留时间和其他指标)转换为一个索引来将文章的实际指标与其预期值进行比较来实现此目的。 我们以1到5的等级给它打分，其中3是预期的，4或5比预期的好，1或2比预期的差。

The figures above show how we present this information when looking at individual articles. Article A had 7,129 readers, more than four thousand more readers than article B, and people spent 2m 44s reading article A, almost a minute longer than article B. A simple web analytics display would pick article A as the winner on both counts by a large margin. And completely mislead us.

上图显示了在查看单个文章时我们如何显示此信息。 A条拥有7,129位读者，比B条多了四千名读者，人们花了200万44秒阅读A条 ，比B条长了近一分钟。 一个简单的网络分析显示将在很大的程度上选择文章A作为两项的赢家。 并完全误导我们。

Once we take into account the context, and calculate the indices, we find that both articles had about as many readers as we would expect, no more or less. Even though article B had four thousand fewer, it was in a less prominent position, and so we wouldn’t expect so many. However, people did spend longer reading article B than we would expect, given factors such as it’s length (it was shorter than article A).

一旦考虑了上下文并计算了索引，我们发现这两篇文章的读者人数与我们预期的一样多，或多或少。 即使B条少了4000条，它的地位也不太重要，所以我们不会期望那么多。 但是，考虑到诸如篇幅的长度(比文章A短)，人们阅读文章B的时间确实比我们预期的要长。

The indices are the output of a predictive model, which predicts a certain value (e.g. number of readers), based on the context (the features in the model). The difference between the actual value and the predicted value (the residuals in the model) then form the basis of the index, which we rescale into the 1–5 score. An additional benefit is that we also have a common scale for different measures, and a common language for discussing these metrics across the newsroom.

索引是预测模型的输出，该预测模型基于上下文(模型中的功能 )预测某个值(例如，读者数量)。 然后，实际值与预测值(模型中的残差 )之间的差构成该指数的基础，我们将其重新换算为1-5得分。 另一个好处是，我们对于不同的度量标准也具有通用的标准，并且在整个新闻编辑室中讨论这些度量标准时具有通用的语言。

Unless we account for context, we can only really use data for inspection: ‘Just tell me which article got me the most readers, I don’t care why’. If the article only had more readers because it was at the top of the edition we’re not learning anything useful from the data, and at worst it creates a self fulfilling feedback loop (more prominent articles get more readers — similar to the popularity bias that can occur in recommendation engines).

除非我们考虑上下文，否则我们只能真正使用数据进行检查 ： “只要告诉我哪一篇文章吸引了我最多的读者，我不在乎为什么”。 如果该文章仅是该版本的顶部，那么该文章只吸引了更多读者，那么我们就不会从数据中学到任何有用的东西，最糟糕的是，它会形成一个自我实现的反馈循环(更多的知名文章会吸引更多的读者，这与人气偏向相似)在推荐引擎中可能会发生)。

In his excellent book Upstream, Dan Heath talks about moving from data for inspection to data for learning. Data for learning is fundamental if we want to make better decisions. If we want to use data for learning in the newsroom, it’s incredibly useful to be able to identify which articles are performing better or worse than we would expect, but that is only ever the start. The real learning comes from what we do with that information, trying something different, and seeing if it has a positive effect on our readers’ experience.

Dan Heath在他的优秀著作“ 上游”中谈到了从检查数据到学习数据的转变 。 如果我们要做出更好的决策，则学习数据至关重要。 如果我们想在新闻编辑室中使用数据进行学习，则能够识别出哪些文章的表现好于或差于我们的预期，这是非常有用的，但这仅仅是开始。 真正的学习来自于我们对这些信息的处理方式，尝试了不同的尝试，并查看它是否对我们的读者体验产生积极影响。

“Using data for inspection is so common that leaders are sometimes oblivious to any other model.”

“使用数据进行检查是如此普遍，以至于领导者有时会忽略任何其他模型。”

Dan Heath, Upstream: The Quest to Solve Problems Before They happen, 2020

丹·希思(Dan Heath)，上游： 寻求在问题发生之前解决问题的探索，2020年

不确定 (Uncertainty)

“What is not surrounded by uncertainty cannot be truth”

“没有不确定性的事物不能是真理”

Richard Feynman (probably)

理查德·费曼( 大概 )

The metrics presented in web analytics tools are incredibly precise. 7,129 people read the article we looked at earlier. How do we compare that to an article with 7,130 readers? What about one with 8,000? When presented with numbers, we can’t help making comparisons, even if we have no idea whether the difference matters.

Web分析工具中提供的指标非常精确。 7,129人阅读了我们之前看过的文章。 我们如何将其与拥有7,130位读者的文章进行比较？ 那一个有8,000个的呢？ 当用数字表示时，即使不知道差异是否重要，我们也无法进行比较。

We developed our indices to avoid meaningless comparisons that didn’t take into account context, but earlier versions of our indices were displayed in a way that suggested more preciseness than they provided — we used a scale from 0 to 200 (with 100* as expected).

我们开发索引的目的是避免不考虑上下文的​​无意义的比较，但是索引的早期版本以比提供的精度更高的方式显示-我们使用0到200的标度(预期值为100 * )。

*Originally we had 0 as our expected value, but quickly learnt that nobody likes having a negative score for their article, but something below 100 is more palatable.

*最初，我们的预期值为0，但很快就知道没有人喜欢对其文章给予负面评分，但低于100的评分则更可口。

Predictably, people started worrying about small differences in the index values between articles. ‘This article scored 92 , but that one scored 103, that second article did better, let’s look at what we can learn from it’. Sadly the model we use to generate the index is not that accurate, and models, like data have uncertainty associated with them. Just as people agonise over small meaningless differences in raw numbers, the same was happening with the indices, and so we moved to a simple 5 point scale.

可以预见的是，人们开始担心文章之间的索引值之间的微小差异。 “这篇文章获得了92分，而一篇文章获得了103分，第二篇文章表现更好，让我们看看我们可以从中学到什么。” 遗憾的是，我们用来生成索引的模型并不那么准确，并且模型(例如数据)具有不确定性。 就像人们为原始数据的微小无意义的差异而苦恼时一样，指数也发生了同样的情况，因此我们采用了简单的5分制。

Most articles get a 3, which can be interpreted as ‘we don’t think there is anything to see here, the article is doing as well as we’d expect on this measure’. An index of 2 or 1 means it is doing a bit worse or a lot worse than expected, and a 4 or a 5 means it is doing a bit better or a lot better than expected.

大多数文章的评分为3，这可以解释为“我们认为这里没有什么可看的，文章的效果与我们预期的一样好”。 指数2或1表示它的表现比预期差或差很多，指数4或5表示它的表现比预期好或差很多。

In this format, the indices provide just enough information for us to know — at a glance — how an article is doing. We use this alongside other data visualisations of indices or raw metrics where more precision is helpful, but in all cases our aim is to help focus attention on what matters, and free up time to validate these insights and decide what to do with them.

索引以这种格式提供了足够的信息，供我们一目了然地了解文章的运行情况。 我们将其与索引或原始指标的其他数据可视化结合使用，在这些数据可视化中，精度更高会有所帮助，但在所有情况下，我们的目标都是帮助将注意力集中在重要的事情上，并腾出时间来验证这些见解并决定如何处理这些见解。

为什么背景和不确定性经常被忽略？ (Why are context and uncertainty so often ignored?)

These problems are not new and covered in many great books on data sense-making — some are decades old, but more recently Howard Wainer, Stephen Few and R J Andrews.

这些问题并不是新出现的，并且在许多有关数据传感的伟大著作中都有介绍-有些已经有几十年的历史了，但是最近出现的是Howard Wainer ， Stephen Few和RJ Andrews 。

Practical guidance on dealing with uncertainty is easier to come by, but in our experience, thinking about context is trickier. From some perspectives this is odd. Predictive models — the bread and butter of data scientists — inherently deal with context as well as uncertainty, as do many of the tools for analysing time series data and detecting anomalies (such as statistical process control). But we are also taught to be cautious when making comparisons where there are fundamental differences between the things we are measuring. Since there are so many differences between the articles we publish, from length, position, who wrote them, what they are about, to the section and day of week on which they appear, we are left wondering whether we can or should use data to compare any of them. Perhaps the guidance on piecing all of this together to build better measurement metrics is less common, because how you deal with context is so contextual.

应对不确定性的实用指南更容易获得，但是根据我们的经验，对上下文的思考比较棘手。 从某些角度来看，这很奇怪。 预测模型(数据科学家的生死攸关)固有地处理了上下文以及不确定性，分析时间序列数据和检测异常(例如统计过程控制 )的许多工具也是如此。 但是，我们也被教导在进行比较时要谨慎，当我们要衡量的事物之间存在根本差异时。 由于我们发表的文章之间存在如此多的差异，从篇幅，位置，撰写者，内容，到出现的部分以及星期几，我们都想知道是否可以或应该使用数据来比较其中任何一个。 也许将所有这些拼凑在一起以建立更好的度量标准的指南不太常见，因为您如何处理上下文取决于上下文。

Even if you set out on this path, there are many mundane reasons to fail. Often the valuable context is unavailable. It took us months to bring basic metadata about our articles— such as length and the position in which they appear— into the same system as the web analytics data. An even bigger obstacle is how much time it takes just to maintain a reliable metrics system (digital products are constantly changing, and this often breaks the web analytics data, including ours as I wrote this). Ideas for improving metrics often stay as ideas or proof of concepts that are not fully rolled out as you deal with these issues.

即使您踏上了这条路，失败的原因很多。 通常有价值的上下文是不可用的 。 我们花了几个月的时间才将与文章有关的基本元数据(例如文章的长度和出现的位置)带到与Web分析数据相同的系统中。 更大的障碍是维护一个可靠的指标系统需要花费多少时间(数字产品在不断变化，这经常会破坏Web分析数据，包括我撰写本文时所涉及的数据)。 改善度量标准的想法通常仍然是想法或概念证明，在您处理这些问题时并未完全推出。

If you do get started, there are myriad choices to make to account for context and uncertainty— from technical to ethical — all involving value judgements. If you stick with a simple metric you can avoid these choices. Bad choices can derail you, but even if you make good ones, if you can’t adequately explain what you have done, you can’t expect the people who use the metrics to trust them. By accounting for context and uncertainty you may replace a simple (but not very useful) metric with something that is in theory more useful, but the opaqueness causes more problems than it solves. Even worse, people place too much trust in the metric and use it without questioning it.

如果您确实开始使用，则有很多选择要考虑从技术到道德的背景和不确定性，这些都涉及价值判断。 如果您坚持使用简单的指标，则可以避免这些选择。 错误的选择可能会使您脱轨，但是即使您做出了不错的选择，如果您无法充分解释自己所做的事情，也就无法期望使用这些指标的人们信任它们。 通过考虑上下文和不确定性，您可以用理论上更有用的东西代替简单(但不是非常有用)的度量，但是不透明会导致更多的问题无法解决。 更糟糕的是，人们对度量标准过于信任，并在不质疑它的情况下使用它。

As for using data to make decisions. We will leave that for another post. But if the data is all noise and no signal, how do you present it in a clear way so the people using it understand what decisions it can help them make? The short answer is you can’t. But if the pressure is on to present some data, it is easier to passively display it in a big dashboard, filled with metrics and leave it to others to work out what to do, in the same way passive language can shield you if you have nothing interesting to say (or bullshit as Carl T. Bergstrom would call it). This is something else we have battled with, and we have tried to avoid replacing big dashboards filled with metrics with big dashboards filled with indices.

至于使用数据做决定。 我们将其留给其他职位。 但是，如果数据全是噪音，却没有信号，您如何以清晰的方式呈现数据，以便使用该数据的人员了解可以帮助他们做出哪些决定？ 简短的答案是你不能。 但是，如果要提供一些数据的压力很大，则可以很容易地将其被动地显示在一个充满指标的大型仪表板中，然后交给其他人来确定要做什么，就像被动语言可以屏蔽您一样没什么好说的(或胡说八道的卡尔·T·伯格斯特姆称之为胡话)。 这是我们一直在与之抗争的另一件事，我们试图避免将充满指标的大型仪表板替换为充满索引的大型仪表板。

Adding an R for reliable and an E for explainable, we end up with a checklist to help us avoid bad — or CRUDE — metrics (Context Reliability Uncertainty Decision orientated Explainability). Checklists are always useful, as it’s easy to forget what matters along the way.

添加了可靠的R 和可解释的E，我们结束了一个清单，以帮助我们避免坏的-或原油-指标(C ontext [R eliabilityüncertainty d ecision定向的êxplainability)。 清单总是有用的，因为很容易忘记一路上最重要的事情。

Anybody promising a quick and easy path to metrics that solve all your problems is probably trying to sell you something. In our experience, it takes time and a significant commitment by everybody involved to build something better. If you don’t have this, it’s tough to even get started.

任何承诺快速，轻松地解决所有问题的指标的人都可能试图向您出售产品。 根据我们的经验，需要时间和每个参与人员做出重大承诺才能打造出更好的产品。 如果您没有这个，甚至很难上手。

非人类指标 (Non-human metrics)

Part of the joy and pain of applying these principles to metrics used for analytics — that is, numbers that are put in front of people who then use them to help them make decisions — is that it provides a visceral feedback loop when you get it wrong. If the metrics cannot be easily understood, if they don’t convey enough information (or too much), if they are biased, or if they are unreliable or if they just look plain wrong vs. everything the person using them knows, you’re in trouble. Whatever the reason, you hear about it pretty quickly, and this is a good motivator for addressing problems head on if you want to maintain trust in the system you have built.

将这些原理应用于分析指标的乐趣和痛苦的一部分(即，数字出现在人们面前，然后他们使用这些指标来帮助他们做出决策)是当您弄错了它时会提供内在的反馈循环。 如果度量标准不容易理解，如果它们传达的信息不足(或过多)，有偏见，不可靠或看起来像是错误的，而使用该工具的人所知道的一切，麻烦了。 无论出于何种原因，您都会很快听到它的消息，如果您希望对已构建的系统保持信任，这将是解决当前问题的良好动力。

Many metrics are not designed to be consumed by humans. The metrics that live inside automated decision systems are subject to many of the same considerations, biases and value judgements. It is sobering to consider the number of changes and improvements we have made based on the positive feedback loop from people using our metrics in the newsroom on a daily basis. This is not the case with many automated decision systems.

许多指标并非旨在供人类使用。 自动决策系统中存在的度量标准受许多相同的考虑因素，偏见和价值判断的影响。 基于每天在新闻编辑室中使用我们的指标的人们所产生的积极反馈循环，考虑到我们所做的更改和改进的数量是令人发人深省的。 对于许多自动化决策系统而言，情况并非如此。

For more background on INCA— the internal system our newsroom uses to access our metrics and indices — see here or The Digital Times, and we will be sharing more in upcoming posts.

有关INCA(新闻编辑室用来访问指标和索引的内部系统)的更多背景信息，请参阅 此处 或 The Digital Times ， 我们将在以后的文章中分享更多内容。