You can find the original article and more like it here at the BioLegend Blog.

The irreproducibility of published data has become a legitimate concern in recent times. In 2012, a landmark study conducted by Glenn Begley and his team concluded that the majority of cancer research studies were not reproducible, demonstrating that a startling 47 out of 53 published findings could not be replicated (Nature 483, 531-533; 2012). The implications for biomedical research are potentially staggering considering the billions spent on research every year, with some suggesting that as much as $28 billion is wasted on irreproducible pre-clinical research every year (Freedman LP, et. al. PLoS Biology 13: e1002165). Antibodies used in research have become a popular target for blame. Many instances of poor specificity, cross-reactivity, and lot-to-lot variability have cost researchers millions and possibly billions, and will likely continue to do so in the future. But, we need to have a keen eye for other factors that are likely to contribute to experimental irreproducibility beyond just the antibodies.

Bias

As humans, we all have preferences, desires, pressures, and feelings that might be reflected in the experimental outcomes, no matter how hard we try to separate them from data. One of the difficulties with a hypothesis-based approach is that most researchers want the data to support their original hypothesis. This inherently introduces bias from the start. Along the way, bias contributes to almost all aspects of experimentation. Below is a list of some areas, but not all, where bias has effects:

• Selective reporting, i.e. excluding outlier data
• Data selecting, i.e. counting phenotypes by microscopy
• Limited data sets
• Interpretation of data
• Experimental design
• Reagent selection
• Collaborators

There are a few straightforward steps that one might take to try to remove bias as much as possible. These might include:

• Blinding and randomization of treated versus control groups
• Including as many controls as possible, both positive and negative
• Use proper statistics to determine significance
• Use well published protocols and reagents
• Use quantifiable measurements, making sure all instruments and assays are properly calibrated

Variability

Commercial resources for critical research reagents like cell culture media, antibodies, and recombinant proteins may go through many rounds of validation testing prior to sale, but what about other day to day materials that are commonly used? Some things to consider:

• Do you routinely check the quality of your DI water?
• Are any of your chemicals expired?
• When was the last time your instruments and equipment were calibrated?
• What is the range of temperature fluctuation in your refrigerator?
• Do you always buy the same reagents from the same source? Even if you do, how does the supplier control lot-to-lot variability?
• How long do you keep stock solutions? Would you know if they are contaminated?
• Are conditions for the storage and handling of mice consistent from week to week, and month-to-month?
• Are your cell lines authenticated? Are they mycoplasma free?

The potential for variability and irreproducibility, even within the same lab, are likely too high if processes, reagents, equipment, and protocols are not consistent.


In addition to variability in research materials, there is also an inherent variability in human and research animal subjects that makes reproducibility a challenge.

The academic system relies on a large proportion of its data to be generated by people regarded as "trainees," either graduate students or post-docs, while experienced Principal Investigators spend their time seeking funding and writing grants. Their status as trainees reflects on their lack of expertise in a subject matter or experimentation, often times both. Yet, the expectation is that they perform the majority of the experiments to generate data for publications. Even in labs with the most well renowned PI's, there will always be new trainees entering, and creating new data. In addition to this, experimental training comes in vastly different forms and quality. Some students are trained by experienced mentors, while others may be trained by the student who started in the lab last quarter. The lack of universal standards for training should be major concern for reproducibility in science.

It's not all doom and gloom though. Scientific discoveries have led to multitudes of breakthroughs that have directly impacted the quality and quantity of life we get to have today. New discoveries will be built upon the groundwork of science that came before. Despite all the imperfections in the way we do science, the body of work that science builds upon provides us a means of constantly changing our understanding of the world and ourselves. We do have a long way to go in reducing irreproducibility, but just consider: it's not always the antibodies.

Learn how BioLegend addresses Validation and Reproducibility.

Written by Dzung Nguyen, PhD.