We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13\% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

中文翻译：

---

信息缺失、作者反应迟钝、实验缺陷：无法评估 NLP 中先前人类评估的可重复性

我们报告了我们在确定一组先前的 NLP 人类评估方面所做的努力，这些评估适用于一项协调研究，检查是什么使 NLP 中的人类评估更/更难重现。我们展示了我们的结果和发现，其中包括只有 13% 的论文具有 (i) 足够低的复制障碍，以及 (ii) 足够的可获得信息，以考虑复制，并且我们选择的所有实验中只有一个因为复制被发现有缺陷，使得进行复制的意义受到质疑。因此，我们不得不将我们的协调研究设计从重现方法更改为标准化然后重现两次的方法。