Transparency and Reproducibility

This (excellently written, very timely and relevant) article by Ken Haapala was posted on the S.E.P.P. web site this week.


Transparency and Reproducibility

By Ken Haapala, President, Science and Environmental Policy Project (SEPP)

Writing for the National Association of Scholars, David Randall and Christopher Welser produced a study titled “The Irreproducibility Crisis of Modern Science: Causes, Consequences, and the Road to Reform.”

Transparency and Reproducibility are critical in the process of transforming sophisticated speculation into knowledge. Work must be checked and re-checked. Studies need to be replicated, reproducible, to be accepted. If the data and the procedures used (methodology) are not transparent, then reproducibility is impossible, and the results should be considered as speculative.

Unfortunately, a great deal of what passes for science by government entities is not transparent and reproducible. It is accepted because it appeals to the ideological biases of those responsible for oversight. Thus, others with similar biases become involved and the entire effort becomes one of groupthink. If government entities are involved it is what The Week That Was (TWTW) terms “Bureaucratic Science.” This bureaucratic science drives government policy, which can have expensive and disastrous consequences and waste significant resources pursuing a solution that does not work, or a solution to a non-problem.

In an article in the Wall Street Journal, Peter Wood, president of the National Association of Scholars, and Mr. Randall discuss problems involving psychology and medicine. Other examples include the PM2.5 myth that outdoor exposure to particulate matter smaller than 2.5 μm (2.5 micrometers) causes heart and lung diseases. This bureaucratic science has been heavily promoted by the EPA and subsequently expanded by the World Health Organization, the UN, the World Bank, and the EU. Yet, the critical studies establishing these claims have never been made public by the EPA. It may be that only in very high doses, PM 2.5 may be a health danger. The American public has a right to know. A right ignored by the EPA and its defenders.

Another example is the Linear No-Threshold Model (LNT), which was developed for radiation exposure by Hermann Muller, for which he received a Nobel Prize. Although it has been strongly contested, the model has been applied — for more than six decades, at costs in the trillions of U.S. dollars — by the EPA and other bureaucratic entities such as the National Center for Toxicological Research and the National Research Council. As compared with the dose-response model of toxicology, the LNT produces absurd results. Drinking too much water can kill a person. Therefore, according to the LNT model, a thousand-people drinking a gallon of water per day will kill someone from too much water.

The authors of the report have 40 specific recommendations to help make science more transparent and reproducible. Too keep the report readable, the authors do not go into detail on the miss-use of statistics. But, the recommendations for Statistical Standards, Data Handling, and Government Regulation bear repeating:


1. “Researchers should avoid regarding the P-value as a dispositive measure of evidence for or against a particular research hypothesis.

2. “Researchers should adopt the best existing practice of the most rigorous sciences and define statistical significance as P < .01 rather than as P < .05. 3. “In reporting their results, researchers should consider replacing ‘either-or’ tests of statistical significance with confidence intervals that provide the range in which a variable’s true value most likely falls. “DATA HANDLING 4. “Researchers should make their data available for public inspection after publication of their results. 5. “Researchers should experiment with born-open data—data archived in an open-access repository at the moment of its creation, and automatically time-stamped.” “GOVERNMENT REGULATION 29. “Government agencies should insist that all new regulations requiring scientific justification rely solely on research that meets strict reproducibility standards. 30. “Government agencies should institute review commissions to determine which existing regulations are based on reproducible research, and to rescind those which are not.” Such standards would make government science and policy more accountable to the public, thus more fitting for a democratic republic. Although the focus is primarily health issues, and the authors stated in a briefing that the US National Institutes of Health (NIH) are making some progress towards Transparency and Reproducibility, the Afterward was written by physicist William Happer. He wrote: “Science has always had problems with quality control. Some particularly bizarre examples were given by Irving Langmuir in his classic lecture, “Pathological Science,” where he describes “N rays,” “Mitogenetic Rays,” etc. Langmuir gave a table that maps very well onto points made by Randall and Welser: “Symptoms of Pathological Science: 1. “The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause. 2. “The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results. 3. “Claims of great accuracy. 4. “Fantastic theories contrary to experience. 5. “Criticisms are met by ad hoc excuses thought up on the spur of the moment. 6. “Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion. “But Langmuir, a great scientist, was not immune to self-deception. As described in J. R. Fleming’s book, ‘Fixing the Sky,’ Langmuir was convinced toward the end of his career that he and his colleagues had succeeded in controlling the weather by seeding clouds with silver iodide. Near the end, William Happer discusses a too popular concept in bureaucratic science, the Noble Lie: “Many scientists think of themselves as philosopher kings, far superior to those in the ‘basket of deplorables.’ The deplorables have a hard time understanding why scientists are so special, and why they should vote as instructed by them. More than two thousand years ago, Plato, who promoted the ideal of philosopher kings, also promoted the concept of the ‘noble lie,’ a myth designed to persuade a skeptical population that they should be grateful to be ruled by philosopher kings. Our current scientific community has occasionally resorted to the noble lie, a problem that can’t be fixed by better training in statistics. Noble lies are also irreproducible, and they damage the credibility of science.”

This entry was posted in Center for Environmental Genetics. Bookmark the permalink.