Are We Incentivizing Hype in Science? A Case Study
Cross-posted with permission from Bjrn Brembs blog:
There is a lively discussion going on right now in various forums on the incentives for scientists to publish their work in this venue or another. Some of these discussions cite our manuscript on the pernicious consequences of journal rank, others don’t. In our manuscript, we speculate that the scientific community may be facing a deluge of fraud and misconduct, because of the incentives to publish in high-ranking journals, a central point of contention in the discussions lnked to above. An example of just how subtle these incentives may skew the scientific debate, even in the absence of any obvious misconduct, happened to land on my desk this morning in the form of a paper quite close to my own field of research, published in the (for our field) very highly ranked journal “Current Biology”.
This paper caught my attention not only because it concerned fly behavior or featured a colleague I happen to know quite well, but because it stated in the abstract that: “blocking synaptic output from octopamine neurons inverts the valence assigned to CO2 and elicits an aversive response in flight”. We currently have a few projects in our lab that target these octopamine neurons, so this was a potentially very important finding. It was my postdoc, Julien Colomb, who spotted the problem with this statement first, in Fig. 3 (click for larger image):
The important data to look at is in Fig. 3D. It shows an attraction for CO2 vs. air in wildtype flies (WT), but an aversion in the genetically manipulated flies (Tdc2-GAL4/UAS-TNT). This is what is stated in the abstract: wildtype flies are attracted to CO2 and flies where octopamine release is blocked, avoid CO2. However, the important control experiments, are those that test for off-target effects of the genetic manipulation. In other words, do the transgenes inserted into the fly genome have an effect of their own, independent of their combined effect on octopamine? In this case, there are two transgenes, a GAL4 transgene (the driver) and a UAS transgene (the effector). Their CO2 scores are shown at the end (Tdc2-GAL4/+ and UAS-TNT/+, respectively). Interestingly, these lines both show a strongly reduced preference for CO2. Their preference is so strongly reduced that it is not even different from that for air. To put it differently: neither of both control lines show normal, wild type behavior. They may not be able to detect CO2 any more, or have secondary alterations that simply reduce the preference for CO2, or a myriad of other explanations. Importantly, nobody can know if these two effects, which either alone already reduce the preference for CO2 dramatically, together could lead to an avoidance of CO2 that is completely independent of the targeted octopamine neurons.
In this respect it is important to point out that the authors are not trying to hide this effect. In the text, in what appears to be a contradiction to the abstract, the authors write:
We note that the Tdc2-GAL4/+ driver line does not spend a significantly greater amount of time in the CO2 plume by comparison to air, but this line, as well as the UAS-TNT/+ parent line, spends significantly more time in the CO2 plume in comparison to their progeny. Therefore, this experimental result cannot be fully attributable to the genetic background.
The last sentence, of course, is incorrect: if both lines independently reduce the attractiveness of CO2, then it is very conceivable, one might even say straightforward, that both together might reduce it so much, that the resulting value of CO2 is negative, leading to an aversive response in the flies, irrespective of the involvement of octopamine.
Given what is known about the action of octopamine in these processes, the hypotheses that the authors claim to have corroborated is beautiful, makes sense and is biologically plausible. So the result they present in the abstract “blocking synaptic output from octopamine neurons inverts the valence assigned to CO2” makes this a very sexy paper for the field that unites several disparate findings and puts a whole set of results in a broader perspective (and may well be correct!). Of course, these considerations are crucial for marketing your paper to one of the top journals in the field. Had the authors discarded the octopamine results from their paper, it is highly unlikely it would have been published in Current Biology.
To be very clear: I can’t find any misconduct in this paper, only clever marketing of the sort that occurs in almost every ‘top-journal’ paper these days and is definitely common practice. On the contrary, this is exactly the behavior incentivized by the current system, it’s what the system demands, so this is what we get. It’s precisely this kind of marketing we refer to in our manuscript, that is selected for in the current evolution of the scientific community. If you don’t do it, you’ll end up unemployed.
In this respect it is worth speculating about the particular incentives the authors of this study might have experienced. The first author is a postdoc in Mark Frye’s lab, so she needs to publish in top journals to get a job. The second author was an undergraduate, so likely less involved in the drafting and revising of the paper and the last author is a junior investigator for HHMI, so likely under enormous pressure (or at least perceived pressure) to publish in top journals not only to justify his award, but also to do well in future evaluations.
Obviously, this is just a case study, N=1, an anecdote, but I think it exemplifies the incentives and how they can distort the scientific debate. For instance, see the Tweet I sent around after I read the abstract (but before I had a look at the actual data:
Octopamine reverses Carbon Dioxide preference in flies feedly.com/k/Upirw7 Bjrn Brembs (@brembs) January 25, 2013
Wasserman, S., Salomon, A., & Frye, M. (2013). Drosophila Tracks Carbon Dioxide in Flight Current Biology DOI: 10.1016/j.cub.2012.12.038
