Gene expression study shows those enjoying subtle humor are healthier than those who do not
Here is a riddle for our readers. Let us say you open the newspaper and read about the latest and greatest new study.
“Gene expression Study shows those enjoying subtle humor are healthier than those who do not”
“Gene expression Study shows Republican-type people are smarter than Democrat- type people”
“Gene expression study in mice shows babies receiving proper parental care are less prone to develop psychological problems later in life”
How do you figure out whether the ‘study’ is properly conducted?
Four act play
You go to the original paper being cited in that report and find that it followed a ‘four-act play’.
Act A. questionnaire or phenotypic observations
In the first act, a number of questions are given to about 50-60 people. The fifty people are classified into two groups based on the answers of those questions.
“Do you drink tea?” Yes means you enjoy subtle humor.
“Do you follow Kim Kardashian in twitter?” Yes means you cannot enjoy subtle humor.
“Do you have a gun?” - Yes means you are a ‘republican-type’ person.
The classification can also be done using phenotypic observations. By now it is well established that mice cannot fill up questionnaire. So, a grad student is sent to the cage five times a day to check whether the mother is taking proper care of its babies.
Act B. gene expression data collected and analyzed according to clusters
The second act is data-intensive. The researchers collect, let’s say, blood samples from those 50 people and perform gene expression measurements through one of the high-throughput technologies. Those data get heavily churned and massaged through procedures that remind one of making butter from milk, except that in case of gene expression data, the ultimate product is ‘genes with p<0.05’.
Act C. Focus on three or four over/under-expressed genes identified by gene expression study
Next, a handful of ‘relevant genes’ are pulled out from the list of genes with p<0.05 and further analyzed. This part depends on the imagination of the researcher. Maybe he checks the 5’ region for all those genes and finds a common motif. Voila !!
Act D. Publication in a highly visible peer-reviewed journal and press- release
It is not difficult to fool the editors and reviewers of highly visible journals given one follows proper procedure (as listed above). After all, who is really going into tons and tons of data and check all assumptions in the analysis?
Media loves to report breathlessly about any unusual human-related finding and that is how you happened to see the ‘new study’ in the first place.
Is it important to debunk bad studies?
One challenge in detecting bad studies is that the four steps are often similar to excellent legitimate studies. Therefore, some readers of bad papers tend to assume that anyone following the above four steps is reporting an outstanding discovery, but is that always so?
The bigger challenge is that the experts in various fields are busy with their own lives, and are not interested in reanalyzing data from someone else’s paper. The rewards tend to be very low. We asked a few researchers working on basic science, whether they would be interested in taking time to re-evaluate one of these studies. Often the response was that it was not important to do so, because ‘bad science would eventually get rejected’, when others could not replicate the observations. Yes, maybe in hundred years, but here is how things work out in real life until then.
a) Those publishing bad studies get more funding to publish even more bad studies, and those not interested in debunking do not get heard (or worse, leave science).
b) Once bad science get published in highly visible journals, the doctors, nutritionists, psychologists, etc. feel compelled to guide their patients based on such ‘latest research’. That is a very important social issue that needs to be considered.
c) Things can be even worse. There are cases, where the doctors are ‘forced to use’ recommendations from such research due to updated government regulations.
We will write more on this in a subsequent commentary posted later today.
