A researcher’s look at Rigor Mortis: Are motivators and incentives to find a cure hurting scientific research?

Science is under siege, and we are all feeling the heat. More paperwork, more justification, more validation, more rules and regulations.

The past five years have seen an explosion in the amount of time and effort spent demonstrating we are actually complying with rather obvious scientific norms. Where is all of this coming from?

The event that triggered much of this change was the publication of the book Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions by Richard Harris. The cover art depicts a toe tag hanging off the ‘I’ in Mortis, with the cause of death labeled “Biomedical Research.”

The book chronicles issues in life science research, including the reproducibility crisis, the often-irrational incentives pitting the career advancement of scientists against producing the best science, and whether science funding is focused on producing societal benefits.

Harris originally wanted to title his book “Science Friction,” a much more accurate description of its contents, and maybe the sensationalist title promises more than the book delivers. Despite this, Rigor Mortis is exceptional for highlighting the many challenges and difficulties that scientists must navigate every day. Practicing and aspiring scientists could use Rigor Mortis as a reminder of the public perception of our profession and the need for us to understand how our actions can erode confidence in scientific research.

Five challenges of experimental science After decades of work as a science reporter, Harris wrote Rigor Mortis after reflecting on the many science stories he has written since 1986 suggesting cures were just around the corner, but in fact never materialized. He spent two years talking to hundreds of scientific researchers about why science has faced a reproducibility crisis. It has disappointed public expectations at a time when biomedical research appears to be in a Golden Age of astonishing technological advances and significant financial support.

Rigor Mortis identifies and explores five challenges facing experimental science.

1. Models of disease Animal models do not accurately mimic human biology. Drug research today involves creating an animal (usually mouse) model of disease that mimics the human disease, and then finding a cure for the animal model. However, animals (especially mice) are often poor proxies for human biology for a variety of reasons, and “cures” found in mice are often ineffective in humans.

2. Questionable accuracy of reagents Biological reagents used in life science are often of questionable quality. Cell lines and antibodies, two staples of life-science research, especially suffer from this problem.  Many cell lines, even from national cell banks, are different cell types or come from different sources than advertised. Many antibodies in research are generated by the immune systems of animals, and therefore suffer large variance between batches. They often do not even recognize the protein of interest. Extensive care must be taken to ensure the accuracy and selectivity of these reagents.

3. Need for statistics training Most life-science education programs do not include the advanced training in statistics required in today’s science. Scientists must have increased statistical literacy as “big data” saturates research. As recently as 10 years ago, scientists could only make a few measurements at time. Recent developments in “omics” have allowed scientists to simultaneously measure hundreds to millions of signals in routine experiments. This increased experimental complexity requires increased statistical sophistication to avoid potential pitfalls like batch effects or p-hacking. As experiments become more complex and more specialists become involved, experimental consistency also becomes a challenge.

4. Data sharing and transparency The explosion of experimental complexity and big data also means life science needs better data sharing and experimental method transparency. As the methods and analysis become more complex, methods sections in papers can no longer contain all the details. The increased cost of projects means it is no longer possible for other labs to independently verify the experimental results. However, life science lacks a consistent, centralized system to share the methods and data. This often makes it impossible for other groups to reproduce the published experiments or independently validate the results from original data.

5. Competition Because the number of PhD degrees awarded each year is significantly greater than the demand for those degree holders, science operates in a “publish or perish” culture. This creates incentives against producing the best research. There is a rush to be the first to publish new research, even when the results are not conclusive. The high-pressure environment also increases the chance of confirmation bias in scientists, causing them to see significant results where they aren’t and dismiss data that weakens the hypothesis. There is also a low percentage of outright fraudulent research, which is often difficult to correct in literature.

Sometimes, experiments fail Harris blames outcomes that don’t meet expectations as science not being conducted properly or rigorously, but failures are an inescapable part of experimental science. The most egregious example provided by Harris is the narrative about fialuridine (FIAU).

FIAU worked in animal models with minimal toxicity, but turned out to be highly toxic in a 1993 human safety trial. Harris labels this as a “debacle” and bemoans how animal models misled scientists and wasted money. The story of FIAU should be a reminder to all of us of how hard science is even when everything is done correctly.

Complex challenges and integrity Rigor Mortis is not a book about solutions to the problems it identifies. Harris builds a great case for how deep-seated and complex these challenges are, but then offers a solution that doesn’t work upon deeper reflection.

He highlights how industrial research lacks the problematic incentives faced in academia. But companies also face as many, if not more, incentives to do poor science (such as Sarepta not performing the FDA-required experiments to prove efficacy) or outright cheating (such as Theranos lying to investors about having made large advances in blood-testing technology). Harris decries putting patients through drug trials based on unclear animal model results, but instead insinuates we should do more research directly on humans (i.e. skip animal models). Certainly, taking this path will produce more FIAU results, not fewer.

The scientific community must take steps to improve reproducibility and integrity, but there are no simple solutions to removing all human incentives for bad behavior.

Impact Many parts of the book have an alarmist tone that sounds critically hostile to academic science. This paints an unfortunate picture of the scientific enterprise to the general public. But Harris raises many good points that scientists are turning into policy.

Rigor Mortis has catalyzed new rules on grant applications and guidelines for the performance of research for the National Institutes of Health. In addition, scientific journals have and will continue to increase rigor in performing and interpreting experiments.

For scientists, it’s a reminder of the general perception of our discipline.