These pages of GEITP have been following the debate about “reproducibility in published data” and “how best to perform statistical analysis” on data planned for publicatioin. The two attached brief articles summarize the latest. Although many have declared how much “poor statistical analysis” is to blame for poor reproducibility, the first (3-page) article summarizes Nature asking five influential statisticians to “recommend one change to improve everyone’s science.” The common theme appeared to be “The problem is not our mathematical analyses, but rather the investigators who have problems in selecting the best approach to statistically analyzing their data.”
The second (1-page) article describes an intriguing free open-source algorithm (STATCHECK) designed to flag statistical errors in psychology papers. Since it first appeared in 2o15, this software has received mixed reaction from the research community. Statcheck was developed in 2o15 by Michèle Nuijten, a statistician at Tilburg University (the Netherlands), and Sacha Epskamp, a psychometrician at the University of Amsterdam. Statcheck screens papers for data reported in the standard format recommended by the American Psychological Association (APA) and uses them to calculate the P-value –– a controversial but widely used measure of statistical significance. If the calculated P-value differs from the one reported by the researchers, the tool flags it as an “inconsistency.”
Last year, Nuijten’s Tilburg University analyzed almost 700,000 results reported in more than 50,000 psychology studies using Statcheck, and had the results automatically posted on the post-publication peer-review site PubPeer, with email notifications to the authors. Some researchers welcomed the feedback, but the
German Psychological Society (DGPs) declared that the postings were “causing needless reputational damage.” Susan Fiske, psychologist at Princeton University and former head of the Association for Psychological Science, called the effort a “form of harassment.” [!!!!]
Nature 30 Nov 2o17; 551: 557–559 and Science 1 Dec 2o17; 358: 1118