Bias in Animal Studies and Nature versus PLOS Tit for Tat

Jonathan Eisen never forgets to point out the inconvenience caused by closed- access journals, as you can see from his exchange with a Nature editor.

However, sometimes that tit for tat can go to another level. For example, someone forwarded us a Nature news story this morning.

Animal studies produce many false positives

We found out that it reported on a PLOS Biology paper.

This bias could partly explain why a therapy that does well in preclinical studies so rarely predicts success in human patients, says John Ioannidis, a physician who studies research methodology at Stanford University in California, and who is a co-author on the study published today in PLoS Biology. The results are too good to be true, he says.

However, Nature’s website did not take us directly to the PLOS Biology article. After clicking ‘Hide Context’, ‘Show Context’ few times, we realized that there was no hyperlink. Finally, we had to use google to get to the real article. Is that an error of the reporter or part of design of Nature’s content management system (at $30K-$40K per paper)?

Evaluation of Excess Significance Bias in Animal Studies of Neurological Dise ases

Animal studies generate valuable hypotheses that lead to the conduct of preventive or therapeutic clinical trials. We assessed whether there is evidence for excess statistical significance in results of animal studies on neurological disorders, suggesting biases. We used data from meta-analyses of interventions deposited in Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES). The number of observed studies with statistically significant results (O) was compared with the expected number (E), based on the statistical power of each study under different assumptions for the plausible effect size. We assessed 4,445 datasets synthesized in 160 meta-analyses on Alzheimer disease (n = 2), experimental autoimmune encephalomyelitis (n = 34), focal ischemia (n = 16), intracerebral hemorrhage (n = 61), Parkinson disease (n = 45), and spinal cord injury (n = 2). 112 meta-analyses (70%) found nominally (p?0.05) statistically significant summary fixed effects. Assuming the effect size in the most precise study to be a plausible effect, 919 out of 4,445 nominally significant results were expected versus 1,719 observed (p<10?9). Excess significance was present across all neurological disorders, in all subgroups defined by methodological characteristics, and also according to alternative plausible effects. Asymmetry tests also showed evidence of small-study effects in 74 (46%) meta-analyses. Significantly effective interventions with more than 500 animals, and no hints of bias were seen in eight (5%) meta-analyses. Overall, there are too many animal studies with statistically significant results in the literature of neurological disorders. This observation suggests strong biases, with selective analysis and outcome reporting biases being plausible explanations, and provides novel evidence on how these biases might influence the whole research domain of neurological animal literature.