In the UK, families, educators, and government officials are in an uproar about the effects of a new algorithm for scoring “A-levels,” the advanced level qualifications used to evaluate students’ knowledge of specific subjects in preparation for university study.

在英国，家庭，教育工作者和政府官员对一种新的“ A-levels ”评分算法的效果感到震惊，“ A-levels ”是用于评估学生对特定科目的知识以准备大学学习的高级证书。

A-level courses usually culminate with exams conducted in testing centers. Because of COVID-19, this year’s exams were canceled. In lieu of those exams and their decisive scores, Ofqual, the government agency responsible for scoring students’ work in A-level classes, opted to use a new algorithm to grade students’ work by applying statistical models of school performance from earlier years. Teachers had already graded the students’ work as part of their coursework, but the algorithm overrode those grades, dropping final scores a full letter for 36% of entries and two full letters for 3% of entries. For thousands of students, A’s became B’s, B’s became C’s, and in a few cases, B’s became D’s. Some students failed their classes, because the algorithm determined that some students must fail. Meanwhile, 5% of well-off students attending private schools saw their scores increase.

A级课程通常以在测试中心进行的考试达到高潮。 由于COVID-19，今年的考试被取消了。 代替对这些考试及其决定性成绩的评分，负责对学生在A级课程中的工作进行评分的政府机构Ofqual选择采用一种新算法，通过应用早些年的学校成绩统计模型来对学生的工作进行评分。 老师已经为学生的作业评分，这是他们课程学习的一部分，但是该算法取代了这些成绩， 最终分数降低了36％的满分和2％的3％ 。 对于成千上万的学生，A变成B，B变成C，在少数情况下，B变成D。 一些学生的课程不及格，因为该算法确定某些学生必须不及格。 同时，上私立学校的小康学生中有5％的分数有所提高。

The algorithm’s grading scheme affected disadvantaged students the most. As The Guardian notes:

该算法的评分方案对处境不利的学生影响最大。 正如《卫报》所述 ：

“Pupils from disadvantaged backgrounds have been worst hit by the controversial standardisation process used to award A-level grades in England this year, while pupils at private schools benefited the most. Private schools increased the proportion of students achieving top grades — A* and A — twice as much as pupils at comprehensives. . . . Pupils in lower socioeconomic backgrounds were most likely to have the grades proposed by their teachers overruled, while those in wealthier areas were less likely to be downgraded, according to the analysis.”

“来自贫困家庭的学生受到今年用于授予英国A级成绩的有争议的标准化程序的打击最大，而私立学校的学生受益最大。 私立学校增加了达到最高成绩A *和A的学生比例，是综合学生的两倍。 。 。 。 分析认为，社会经济背景较低的学生最有可能推翻老师建议的成绩，而较富裕地区的学生则不太可能被降级。

The lower grades led some universities and medical schools to revoke the acceptance letters. Affected students were crushed.

低年级导致一些大学和医学院撤销了录取通知书。 受影响的学生被压碎了。

Now many universities are reversing those decisions, and the government is performing a “U-turn,” accepting teachers’ grades as the final A-level scores. Still not impressed, a senior Tory MP is now calling for the abolition of Ofqual itself.

现在，许多大学正在扭转这些决定，而政府正在执行“掉头”，接受教师的成绩作为最终的A级成绩。 仍然没有留下深刻印象的是， 一位高级保守党议员现在呼吁取消Ofqual本身 。

数据科学家的经验教训 (Lessons for Data Scientists)

Here are four lessons this debacle offers data scientists and data engineers.

这是这场灾难给数据科学家和数据工程师的四课。

1.如果结果看起来很奇怪，请仔细检查算法。 (1. If the results seem odd, double-check your algorithm.)

If you’re developing an algorithm that lowers results for a significant number of entries — let alone 40% of entries — it’s time to re-evaluate your algorithm, especially if the results affect people in life-altering ways, such as denying them a mortgage or affecting which university they can attend.

如果您正在开发一种算法，该算法会降低大量条目的结果(更不用说40％的条目了)，那么该是重新评估算法的时候了，尤其是当结果以改变生活的方式影响人们时，例如拒绝他们抵押或影响他们可以参加的大学。

Again, from The Guardian:

再次，从卫报 ：

“Ofqual instead chose to focus on its own measure of accuracy — whether it was right ‘within a grade’. . . . But as any A-level student will tell you, accuracy ‘within a grade’ is meaningless. Ofqual may mark itself highly if it gives an A student a B, but for that student, the difference is life-changing.”

“ Ofqual而是选择专注于自己的准确性衡量标准-是否“在等级内”是正确的。 。 。 。 但是，正如任何A级学生都会告诉您的那样，“在年级内”的准确性是没有意义的。 如果给A学生一个B，Ofqual可能会给予很高的评价，但是对于那个学生来说，差异是改变人生的。”

Keep your eyes open for shifts in data patterns, and understand what constitutes a significant change for the data science use case you’re working on.

睁大眼睛注意数据模式的变化，并了解什么构成了您正在研究的数据科学用例的重大变化。

2.如果结果似乎有偏差，请对算法进行三遍检查。 (2. If the results seem biased, triple-check your algorithm.)

It’s one thing to produce unexpected results. It’s another thing to produce unexpected results that favor the wealthy and disadvantage everyone else. There’s a growing concern among data scientists and the public about the effects of bias in data science algorithms. If results are not only unexpected but clearly biased against an economic or racial cohort, the algorithm should be re-examined and corrected.

产生意想不到的结果是一回事。 产生意想不到的结果，有利于其他所有人的富人和弱势，这是另一回事。 数据科学家和公众对数据科学算法中偏差的影响越来越关注。 如果结果不仅出乎意料，而且明显不利于经济或种族，则应重新检查和纠正该算法。

3.只要有可能，请寻求专家的帮助。 (3. Whenever possible, get help from experts.)

In April 2020, the Royal Statistical Society (RSS), a charity that promotes statistics for the common good, offered Ofqual the assistance of two of its fellows: Guy Nason, professor of statistics at Imperial College London, and Paula Williamson, professor of medical statistics at the University of Liverpool. But Ofqual would accept their assistance only if they agreed to sign a five-year non-disclosure agreement. The professors understandably refused, so Ofqual ended up applying its scoring algorithm without their guidance.

2020年4月，为促进公共利益而促进统计的慈善机构皇家统计学会 (RSS)向Ofqual 提供了两个研究员的协助 ：伦敦帝国理工学院统计学教授Guy Nason和医学教授Paula Williamson利物浦大学的统计数据。 但是，只有当他们同意签署为期五年的保密协议时，Ofqual才会接受他们的帮助。 教授们拒绝了，这是可以理解的，因此Ofqual最终在他们的指导下应用了其评分算法。

Many projects can benefit from a fresh perspective and outside expertise. If you can get second or third opinions, do so.

许多项目可以从崭新的视角和外部专业知识中受益。 如果您可以获得第二或第三意见，请这样做。

4.要透明。 (4. Be transparent.)

It’s troubling that a government agency would try to keep its grading algorithm secret — especially when that algorithm determines which students will end up attending which universities. One can’t help but wonder if Ofqual realized that its algorithm was biased and wished to conceal the details.

令人不安的是，政府机构将试图保密其评分算法，尤其是当该算法确定哪些学生最终将进入哪所大学时。 人们不禁要问，Ofqual是否意识到其算法有偏见，并希望隐瞒细节。

If data scientists want the public to trust the results of their algorithm, then it’s best to be open about how that algorithm works.

如果数据科学家希望公众信任其算法的结果，那么最好对算法的工作方式持开放态度。

As the leadership of the RSS wrote in a letter to the Office of Statistics Regulation on August 14, 2020:

正如RSS的领导在2020年8月14日给统计局的信中写道：

“One issue underpinning trustworthiness of statistics is their quality and accuracy, which is why we have summarised some of our technical concerns. But another element in trustworthiness is the transparency with which the statistics have been set out and considered, and the extent to which they meet public need.”

“统计数据可信赖性的一个问题是统计数据的质量和准确性，这就是我们总结一些技术问题的原因。 但是可信性的另一个要素是，统计数据的制定和考虑的透明度以及满足公众需求的程度。”

Transparency matters. People need to be able understand how criteria are evaluated and decisions are made. Critically, transparent discussions of algorithms should take place before analytical results are shared with the public. Transparency should help guide decision-making, not excuse it.

透明度很重要。 人们需要能够理解如何评估标准和制定决策。 至关重要的是，在与公众分享分析结果之前，应该对算法进行透明的讨论。 透明应该帮助指导决策，而不是原谅。

透明，公平和道德的重要性 (The Importance of Transparency, Fairness, and Ethics)

Ultimately, data science involves more than statistics. It also requires ethics, an open mind, and a clear understanding of the results that algorithms can have on people’s lives.

最终，数据科学不仅涉及统计。 它还需要道德，开放的心态以及对算法可能对人们的生活产生的影响的清晰理解。

Let’s close with these words from the RSS:

让我们从RSS中的这些词结束：

“The use of statistics for public good is based only partly on technical statistical issues. Some statistics are technically bad, wrong or worse than others because of the way that data are gathered, or the statistical modelling that takes place. But in many cases, statistics or statistical models are inadequate for the weight being put on them in decision-making, or embed various other judgements that need to be clear. . . . So while we continue to have concerns about various technical decisions made by the qualification regulators, we also believe that having an more open discussion about this well before individual results were announced would have resulted in more trust in, and more trustworthy, statistical choices, in part because there would have been greater understanding of the underlying principles being applied and more detailed justifications of them.”

“为公共利益使用统计信息仅部分基于技术统计问题。 由于收集数据的方式或进行的统计建模，某些统计在技术上比其他统计差，错或差。 但是在许多情况下，统计数据或统计模型不足以在决策中施加重担，或者嵌入各种其他需要明确的判断。 。 。 。 因此，尽管我们继续对资格认证监管机构做出的各种技术决策表示担忧，但我们也相信，在宣布单个结果之前就此问题进行更加公开的讨论，将会使人们更加信任，更值得信赖的统计选择。部分是因为人们将对所应用的基本原理有更深入的了解，并对其有更详细的论据。”

They point out that fairness is more than a matter of statistics:

他们指出，公平不仅仅是统计问题：

“‘Fairness’ is not of course a statistical concept. Different and reasonable people will have different judgements about what is ‘fair’, both in general and about this particular issue. . . . But a statistical procedure should be capable of being judged as ‘fair’ or ‘reasonable’ in advance of its being used or knowing which individuals may be affected.”

“公平”当然不是一个统计概念。 总体而言，对于这个“特殊”问题，不同的，合理的人会有不同的判断。 。 。 。 但是统计程序应该能够在被使用或知道哪些人可能受到影响之前被判断为“公平”或“合理”。

The importance of attention to detail, of openness to expert opinion, of transparency, and of a keen sense of what’s fair and how data science results affect real people — these are the lessons that data scientists can take away from the UK’s A-levels debacle.

注重细节，保持专家意见的开放性，透明性以及对公平事物以及数据科学成果如何影响真实人的敏锐感知的重要性，这些都是数据科学家可以从英国A级考试崩溃中吸取的教训。

On this occasion, even the manner of testing itself proves to be educational.

在这种情况下，甚至测试方式本身也被证明具有教育意义。