[Edit. Fixed error in title. Thanks Steven and Sébastien for pointing out.]
An interesting political battle is taking place between Nature journal and few well-known scientists. Last month Nature published an article by Dr. John Ioannidis titled “Research grants: Conform and be funded”, which showed that only mediocre and conformist research projects received continual NIH funding. The issue is so well-accepted among most scientists that there is very little to argue against it. NIH study panel includes well-connected, but otherwise incompetent, older professors, who review grants based on their level of ignorance. Once in a blue moon, few young researchers (young defined as 40 years old !!) get in, provided they are well-connected. When we wrote a grant on lincRNAs in 2006, none of the reviewers knew anything about them and the ‘resident expert’ talked nonsense on miRNAs.
Consider the following interview of Erin K. O’Shea for example.
Q: Do you think there is too much conformity in how grants are awarded?
A: I do. I actually do. … Consider the positive first:
It works badly if someone tries to do something that they haven’t done before. … When you try to step out of the field in which you work, it is very, very difficult to get money, and I speak with firsthand experience. For 20 years I had NIH grants. I do not now and it’s in part because I changed fields.
If you don’t continue along the kind of track and want to do something new that involves different methods or different approaches or a different problem or a different system, it’s very hard to get money from the NIH to to that. It’s not impossible, but it’s very difficult.
Q: Has the problem gotten better or worse?
A: My sense is this kind of conformity problem has gotten worse over the years as money gets more and more restricted and science gets bigger and bigger. … The review panels are comprised of NIH-funded people … very few of whom have made these high-impact discoveries. … People have an actual tendency to select for a type that is their own.
Dr. Ioannidis and his colleague backed the above notion by going through papers with 1,000 or more citations, and found that about 40% of them were supported by NIH. More than receiving funding, the authors also analyzed the citation track record of those sitting on NIH study panels, and found that they rarely included those scientists with highly cited papers. Please take a look at the following figure from the supplementary section of Nature paper.
Steven Salzberg of JHU wrote a rebuttal of the article based on his reanalysis of subset of original data on highly cited papers. Salzberg claimed that ~100% of ‘eligible projects’ in his subset were funded by NIH. His full letter is available from Simply Statistics blog.
What do our readers think? Most criticisms I read in Simply Statistics blog and Nature website were based on whether the cutoff should have been at 1000, or whether citation is a measure of research quality, etc. Also, some readers dismissed the conclusion because of inclusion of review articles, even though authors clearly factored that in.
The main analysis considered papers regardless of their type, and sub-field within the biosciences. 27% of them were reviews, and another 5% were methods papers. For such extremely-cited work, one can argue that the influence and value is not decisively diminished or enhanced by the type of the paper or methods/design of the research. We nevertheless performed comparisons separating articles from reviews in a case-control study (see below).