The attached brief report addresses an additional (very valuable) point, about the “data reproducibility crisis” hysteria currently going on. To those of us “in the trenches”, we’ve always been aware of the “variability of the furry beast” or cells in culture. A Sprague-Dawley rat in 1975 is not necessarily the same as a Sprague-Dawley rat in 2005. A C57BL/6J mouse in 2001 is not necessarily the same as a C57BL/6J mouse in 2016. And, for sure, any established cell line being passaged every week or two––is not the same in 2016 as it was in 2006.
It should be noted that these are all examples in biology. I recall one anecdotal story in the field of chemistry: “An unstable photo-oxidative chemical reaction could not be reproduced in a laboratory located in a city with very little air pollution [Jack Feldman, pers. commun.], whereas it had originally been reported in a lab located in a city having high levels of smog and air pollution.” Undoubtedly, there are also examples of such difficulties in reproducibility even in the field of physics.
Spectacular examples of translational failures and poor reproducibility have been attributed to various aspects of “poor experimental design and conduct”, including: small sample size, risk of bias, selective reporting, and publication bias. There was a recent review––challenging the increasingly common view that irreproducibility reflects “poor research conduct by the investigator”. Many of these articles have missed perhaps the most important point: ignorance of phenotypic plasticity in experimental design and analysis.
When studying living organisms, we are faced with inherent biological variation––that is distinct from random noise or measurement error––and that is fundamental to the correct interpretation of experimental results. Biological variation occurs in response to environmental variation. The response of an organism to an experimental treatment (e.g. a drug or other xenobiotic stressor) often depends not only on properties of the treatment, but also on the state of the organism, which is as much the product of past and present environmental influences as of its genetic architecture.
Such phenotypic plasticity of organisms is the result of gene-by-environment (G´E) interactions. And its consequence is the reaction norm of an organism’s response. As an extreme example, a mouse kept at 24oC might respond to a drug treatment differently from a mouse kept at 20oC. Food and light conditions can matter, as well as the lab animal’s rearing history, handling, housing conditions, female and male animals housed together, and many other unmeasured and unmeasurable variables. Even an intra-tracheal injection by one person can easily differ from an intra-tracheal injection by a second person in the same lab.
For all these reasons, we should expect results to differ––whenever an in vivo experiment is replicated. The magnitude of the expected difference is given by the reaction norm, and by the domain of the reaction norm represented by the study populations of the replicate studies. These variations should not be mistaken for random variation, which is a different form of variation, on top of phenotypic plasticity.
Trends Pharmacol Sci July 2o16 37: 509–510
[contributed by a shy colleague who resides in the EU––which, I understand, remains intact, at least at the time of this email posting.]