John Ionniadis is an epidemiologist who has decided to focus his critical eye on the published medical research findings. And he has come up with something a little worrisome: According to Ionniadis, false findings may be the majority of published scientific results:
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Take the discovery that the risk of disease may vary between men and women, depending on their genes. Studies have prominently reported such sex differences for hypertension, schizophrenia and multiple sclerosis, as well as lung cancer and heart attacks. In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes.
Upon closer scrutiny, almost none of them held up. Only one was replicated.
Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.
Medical sciences are not the only fields where researchers are expected to come up with new and astonishing findings. Most social science research works under the same pressures. Imagine how much success a researcher might have in getting a no-difference-found paper published in, say, the field of gender differences? And note that in most traditional fields the researchers are mining an increasingly empty mine, where most of the really valuable lodes have already been exploited. Hence, it is the peripheral and more speculative ideas which are now increasingly presented as important new findings.
All this matters in political research, too. An additional twist to trying to understand the meaning of new research findings in a field related to some important government policy is that much of the research is now done within politically motivated think tanks, and such research is not subjected to the peer review process, however faulty that might be. Despite that flaw, the studies are routinely referred to as important scientific results, worthy of affecting public policy decisions.
----
Cross-posted on TAPPED.