By Stuart George, Executive Director of Design Technology at Method

Method设计技术执行总监Stuart George

When Andrew Mason, founder of Groupon, wanted to improve his email conversion metrics, he turned to data analysis. His team tested the impact of sending two emails per day instead of one, and found that, while more double-emailed customers tended to unsubscribe, the ones who stayed generated more revenue. Ignoring his intuition, he had his team switch to the two-a-day model.

Groupon的创始人安德鲁·梅森(Andrew Mason)想改善电子邮件转换指标时，便转向数据分析。 他的团队测试了每天发送两封电子邮件而不是每天发送一封电子邮件的影响，并发现，尽管有更多重复发送电子邮件的客户倾向于退订，但留下的客户却产生了更多的收入。 无视他的直觉，他让他的团队改用每天两次的模型。

This was not a good decision. Whereas there is no doubt that the Groupon data scientists achieved statistically significant results, they failed to consider the long-term effects of the change. Groupon became little more than a “marketplace of coupons,” Mason admits, eventually burning through the revenue potential of their dwindling market.

这不是一个好决定。 毫无疑问，Groupon数据科学家取得了统计上显着的结果，但他们没有考虑这一变化的长期影响。 梅森承认 ，Groupon仅仅是一个“优惠券市场”，最终烧毁了其日益萎缩的市场的潜在收入。

From this example, it would be easy to conclude that data-driven decision making is more trouble than it’s worth.

从这个例子中，很容易得出结论，数据驱动的决策比其价值更大。

Putting your faith in statistics and modeling alone can be quite risky, as data-driven analytics and insights are very prone to “crimes” of malpractice and misuse. But equally, an over-reliance on intuition can lead to suboptimal decision making, especially within teams.

仅仅依靠统计和建模可能会带来很大的风险，因为数据驱动的分析和见解很容易发生渎职和滥用的“罪行”。 但同样，过分依赖直觉会导致决策不理想，尤其是在团队内部。

When data is expensive or difficult to source, many companies use qualitative frameworks to synthesize ideas and opinions. While these can help simplify incredibly complex information, they often rely too heavily on intuition and fail to immunize teams from the perils of human bias.

当数据昂贵或难以获取时，许多公司使用定性框架来综合思想和观点。 尽管这些可以帮助简化极其复杂的信息，但它们通常过于依赖直觉，无法使团队免受人为偏见的危害。

And we humans have a lot of biases to be wary of. Confirmation bias, availability bias, or representativeness bias, to name a few, can help us simplify decision making based on past experience — but this can often mean that we judge incorrectly. All these biases compound with other social biases in group decision making, creating minefields of judgement errors for project teams.

我们人类有很多偏见需要警惕。 确认偏见，可用性偏见或代表性偏见等可以帮助我们根据过去的经验简化决策，但这通常意味着我们判断不正确。 所有这些偏见与集体决策中的其他社会偏见加在一起，为项目团队创造了判断错误的雷区。

So how might we balance intuition and data to make better group decisions? At Method we have a method. We call it Fact-based Hypothesis Testing, and it’s a way to help us make better decisions from qual and quant evidence and remove the bias that occurs in these decisions. When evidence for a particular hypothesis is mainly subjective or subjective and qualitative, Fact-Based Hypothesis Testing can make rigorous statistical analysis possible. This is achieved by asking team members questions about how artifacts, evidence and data acquired during the project affect the likelihood of each hypothesis being true. These answers are then analyzed and combined using Bayesian statistics. The system creates an audit trail of how the group considered evidence during the course of the project and how the group’s opinions change through the course of a project.

那么，我们如何平衡直觉和数据来做出更好的团队决策呢？ 在Method中，有一个方法。 我们称其为“基于事实的假设检验” ，这是一种帮助我们根据合格和定量证据做出更好的决策并消除这些决策中出现偏见的方法。 当特定假设的证据主要是主观的或主观的和定性的时，基于事实的假设检验可以使进行严格的统计分析成为可能。 这是通过向团队成员询问有关在项目期间获取的工件，证据和数据如何影响每个假设为真的可能性来实现的。 然后使用贝叶斯统计分析和组合这些答案。 该系统会创建审核跟踪，以了解小组在项目过程中如何考虑证据以及小组的意见在项目过程中如何变化。

To illustrate how the system works, consider the development of an energy usage app called EcoWatch. Your team is trying to determine if the product is desirable to 25- to 34-year-old first-time homeowners by evaluating a number of pieces of evidence. You could frame the project as a test of two hypotheses:

为了说明该系统如何工作，请考虑开发一个名为EcoWatch的能源使用应用程序。 您的团队正在通过评估许多证据来确定25至34岁的首次购房者是否需要该产品。 您可以将项目设计为两个假设的检验：

“Hypothesis A: EcoWatch is desirable to 25–34 year old first-time homeowners” “Hypothesis B: EcoWatch is not desirable to 25–34 year old first-time homeowners”

“假设A：25-34岁的首次购房者需要EcoWatch”“假设B：25-34岁的首次购房者不希望使用EcoWatch”

First, you would define your team’s “prior probability of a hypothesis.” Ask each team member to evaluate how likely they believe each hypothesis is to be true in qualitative terms (on a scale of impossible to extremely likely). The system then converts the designer’s evaluations into probabilities which are combined to produce a group likelihood of a hypothesis being true before evaluating evidence.

首先，您要定义团队的“假设的先验概率”。 要求每个团队成员以定性的方式评估他们认为每个假设真实的可能性(以不可能到极有可能的程度)。 然后，系统将设计者的评估转换为概率，在评估证据之前，将这些概率合并以产生假设为真的组似然。

Then you would evaluate the evidence from the project — in this case, evidence may look like the results of a survey or the synthesis of a user test. For each piece of evidence, the system asks two questions:

然后，您将评估项目中的证据-在这种情况下，证据可能看起来像调查结果或用户测试的综合结果。 对于每个证据，系统都会提出两个问题：

“If Hypothesis A were 100 percent true, how likely is it that you would see this evidence?”

“如果假设A为100％正确，那么您看到该证据的可能性有多大？”

“If Hypothesis B were 100 percent false, how likely is it that you would see this evidence?”

“如果假设B为100％错误，那么您看到该证据的可能性有多大？”

If I were a team member, I may be inclined to say that the answer to the first question is “likely” and the second is “unlikely.” Using the theory of Bayesian statistics, we can combine all the team member’s answers to produce a group answer that fairly represents the group’s collective beliefs. This process continues as new evidence emerges or is added to the system, creating an audit trail of the likelihood of each hypothesis over time. By the end of the project, not only do we have the group’s preferred conclusion but also a rigorous and systematic way of understanding how the team arrived at its decision.

如果我是团队成员，我可能会倾向于说，第一个问题的答案是“可能”，第二个问题的答案是“不太可能”。 使用贝叶斯统计理论，我们可以将所有团队成员的答案结合起来，以得出公平地代表团队集体信念的团队答案。 随着新证据的出现或将新证据添加到系统中，此过程将继续进行，从而创建每个假设随时间变化的可能性的审计线索。 到项目结束时，我们不仅获得了小组的首选结论，而且以一种严格而系统的方式来了解团队是如何做出决定的。

The Fact-Based Hypothesis Testing framework has four key features:

基于事实的假设检验框架具有四个关键特征：

1. Independent: each person evaluates the relevance of evidence independently from all other designers, helping to mitigate the effect of team groupthink.

   独立性：每个人都独立于其他所有设计师评估证据的相关性，从而有助于减轻团队集体思考的影响。

2. Anonymous: each person’s answers are kept secret from the rest of the group, meaning there can be no finger-pointing if a person’s opinion dissents from the group.

   匿名 ：每个人的答案都与小组其他成员保密，这意味着如果一个人的意见与小组不同，就不会有指责。

3. Rigorous: the team’s answers are combined using a statistical procedure that avoids some of the pitfalls of simple aggregation techniques such as pooling or averaging.

   严谨：使用统计程序组合团队的答案，避免了简单的汇总技术(如合并或平均)的一些陷阱。

4. Calibrated: if the team leader believes a systematic bias could be at play in the group’s decision making, they can create fake evidence that, if true, would strongly confirm one hypothesis over all others. If the team members don’t evaluate this evidence in an appropriate fashion, the team leader can highlight the discrepancy to their team and address the bias.

   已校准：如果团队负责人认为系统的偏见可能会影响团队的决策，则他们可以创建虚假证据，如果为真，将强有力地证实一个假设高于所有其他假设。 如果团队成员没有以适当的方式评估此证据，则团队负责人可以突出显示与团队的差异并解决偏见。

If Andrew Mason of Groupon had evaluated his email decision with Fact-based Hypothesis Testing, he may have found that keeping with one email made sense to decrease customer churn. He would have been able to balance the data against his intuition, without feeling the need to choose one over the other. And he could have brought that decision to his shareholders with an audit trail, giving them a window into what hypotheses were considered, what evidence was evaluated, and how his team’s opinion of the hypotheses changed over time.

如果Groupon的Andrew Mason使用基于事实的假设测试评估了他的电子邮件决定，他可能会发现保留一封电子邮件可以减少客户流失。 他本来可以根据自己的直觉来平衡数据，而无需感觉需要选择一个。 而且他本可以通过审计跟踪将该决定带给其股东，让他们有机会了解所考虑的假设，所评估的证据以及其团队对假设的看法随时间变化的方式。

* * *

* * *

This article was written by Stuart George and edited by Erin Peace. Illustration by Claire Lorman. To learn more about our process, or understand how your teams might use Fact-based Hypothesis Testing, please get in touch.

本文由Stuart George撰写，由Erin Peace编辑。 克莱尔·洛曼(Claire Lorman)的插图。 要了解有关我们流程的更多信息，或了解您的团队如何使用基于事实的假设检验，请 联系 。