[Edit. Fixed error in title. Thanks Steven and Sébastien for pointing out.]
An interesting political battle is taking place between Nature journal and few well-known scientists. Last month Nature published an article by Dr. John Ioannidis titled “Research grants: Conform and be funded”, which showed that only mediocre and conformist research projects received continual NIH funding. The issue is so well-accepted among most scientists that there is very little to argue against it. NIH study panel includes well-connected, but otherwise incompetent, older professors, who review grants based on their level of ignorance. Once in a blue moon, few young researchers (young defined as 40 years old !!) get in, provided they are well-connected. When we wrote a grant on lincRNAs in 2006, none of the reviewers knew anything about them and the ‘resident expert’ talked nonsense on miRNAs.
Consider the following interview of Erin K. O’Shea for example.
Q: Do you think there is too much conformity in how grants are awarded?
A: I do. I actually do. … Consider the positive first:
[snip]
It works badly if someone tries to do something that they haven’t done before. … When you try to step out of the field in which you work, it is very, very difficult to get money, and I speak with firsthand experience. For 20 years I had NIH grants. I do not now and it’s in part because I changed fields.If you don’t continue along the kind of track and want to do something new that involves different methods or different approaches or a different problem or a different system, it’s very hard to get money from the NIH to to that. It’s not impossible, but it’s very difficult.
Q: Has the problem gotten better or worse?
A: My sense is this kind of conformity problem has gotten worse over the years as money gets more and more restricted and science gets bigger and bigger. … The review panels are comprised of NIH-funded people … very few of whom have made these high-impact discoveries. … People have an actual tendency to select for a type that is their own.
Dr. Ioannidis and his colleague backed the above notion by going through papers with 1,000 or more citations, and found that about 40% of them were supported by NIH. More than receiving funding, the authors also analyzed the citation track record of those sitting on NIH study panels, and found that they rarely included those scientists with highly cited papers. Please take a look at the following figure from the supplementary section of Nature paper.
Steven Salzberg of JHU wrote a rebuttal of the article based on his reanalysis of subset of original data on highly cited papers. Salzberg claimed that ~100% of ‘eligible projects’ in his subset were funded by NIH. His full letter is available from Simply Statistics blog.
What do our readers think? Most criticisms I read in Simply Statistics blog and Nature website were based on whether the cutoff should have been at 1000, or whether citation is a measure of research quality, etc. Also, some readers dismissed the conclusion because of inclusion of review articles, even though authors clearly factored that in.
The main analysis considered papers regardless of their type, and sub-field within the biosciences. 27% of them were reviews, and another 5% were methods papers. For such extremely-cited work, one can argue that the influence and value is not decisively diminished or enhanced by the type of the paper or methods/design of the research. We nevertheless performed comparisons separating articles from reviews in a case-control study (see below).
could you fix the spelling of my name in the title of your blog?
-Steven Salzberg
Fixed it. Sorry for the error !!
From Salzberg’s letter:
————————————————————————————————————
> I downloaded the full text for a random sample of 125 of the 700 highly cited papers [data available upon request].
Why not use all of them — 700 is not a big number anyway compared to number of articles published per year.
> A majority of these papers were either reviews (63), which do not report original findings, or not in
> the life sciences (17) despite being included in the authors’ database. For the remaining 45 papers,
> I looked at each paper to see if the work was supported by NIH.
45 is not a big number neither.
> In a few cases where the paper did not include this information, I used the NIH grants database to
> determine if the corresponding author has current NIH support. 34 out of 45 (75%) of these
> highly-cited papers were supported by NIH.
But that’s the point of the original letter to Nature — NIH funding does not lead to the most cited
articles although the authors of these cited articles may have NIH funding for their other research
activities.
> the choice to require that both the first and last author be PIs on an NIH grant
I agree with Steven Salzberg on that one. This criterion is not appropriate for numerous papers.
> Conform and be funded
This is not unique to NIH, this phenomenon is present in other granting agencies I suppose.
————————————————————————————————————
Obviously, this kind of data-mining project is more amenable with automated software. Steven Salzberg
was courageous to do it manually. But in the end, not doing a large number of articles lead to
invalid claims.
I personally don’t agree with Nature and I don’t agree with Steven Salzberg neither.
I mean, researchers need money to do their research, and some research has more impact.
Sure, some researchers are better at picking up stuff that impacts society, but this art never
reaches perfection.
I think there is a lot of legacy too regarding which researchers get NIH funding. Some say that all
the good seats are occupied, and that you either wait from an old fart to die or just move into another
less saturated industry.
>I personally don’t agree with Nature and I don’t agree with Steven Salzberg.
You see, Salzberg only pointed out his disagreement with data analysis of Nature paper, but carefully avoided the questions about (i) whether NIH is epitome of perfection or can be greatly improved, and (ii) what kind of data analysis can be used to figure that out. In our opinion, the debate is political and no amount of data digging can help. Here is why. The only way to tell whether NIH helped US society during the course of its existence is to create two identical USAs – one with NIH and one without NIH – and run parallel experiments. Given that social experiments can be run only once, it is impossible to ‘prove’ anything. Political factions in favor of or against NIH will point out shortcomings of backward data analysis to make a point.
Kind of questions to ask -
1. Would human society have sequenced human genome sequenced by year 2000 irrespective of NIH’s existence?
2. Who funded Newton, Darwin, Einstein, Dirac, Feynman, etc.? NIH and other government funding agencies are cold-war phenomena and nothing of that type existed in previous centuries.
[...] Is Steven Salzberg Defending the Indefensible? [...]
I think there is plenty wrong with the methodology of the paper – especially the requirement that first and last authors be NIH funded. The numbers that Prof. Salzberg comes up with feel pretty good, I am surprised at the 75%, I would have thought that it were a much smaller number.
But it is really the sentence at the end of the Nature paper that caught my eye
“Funding all scientists who are key authors of unrefuted papers that have 1,000 or more citations would be a negligible amount in the big picture of the NIH budget, simply because there are very few such people.”
Imagine that world for a second: the one where if you hit the magical 1000 citations all your research proposals would get automatically funded. Imagine that mad dash, immense pressure for getting citations, the misuse and abuse that this would promote. As they say the road to hell is paved with good intentions.
I for one sincerely hope that time will never come.
Good points Istvan. The system will be gamed badly, if such crazy rule is implemented.
In general, I think NIH is the most transparent (or least bad) of all funding agencies out there in USA. I do not know why scientists do not point out that there will be plenty of money for science, if warmongering and financial bailouts are replaced with funds for scientific research.