In May of this year, a fairly big paper emerged in one of our major research areas (trying to expand blood stem cells outside the body). We know the research group behind it and had a bit of a head start trying to transfer the protocol to our lab, but it turned out to not be as straightforward as we first hoped. The first few attempts showed some definite promise, but it didn’t consistently match the reported success of the original study. In the cases where it did work, it was the best result in blood stem cell expansion that I’d ever seen. This made it obvious that as a lab group we needed to find out where things were (sometimes) going wrong. Enter the lab challenge.
As some readers may know, I’m a big fan of putting my money where my mouth is, so I called the researchers in my lab together and offered a small financial carrot to see if any individual could consistently produce similar results to the published findings. It was a 28-day protocol and involved regular maintenance so we set the rules of the challenge: everyone got the protocol to read and was supplied with equal starting pots of blood stem cells and the same reagents to work with. Then they were left to their own devices to “interpret the protocol.”
This word “interpret” might come as a surprise to our non-science readers – it could easily be assumed that a scientific protocol is rigid, with little to no room for “interpretation.” However, the reality in very large and long protocols is there are a huge number of areas where scientists use non-exact language and this can make a huge amount of difference depending on the reader. I’ve listed some examples below that illustrate some of these less than specific items that biologists need to deal with on a daily basis. Terms like “gently flick”, “gradually remove”, “gently dribbling” and “aggressive mixing” all have room for interpretation. Combined with changing the reagents “every two to three days” in a procedure called a “complete” medium change with a further proviso to remove “ approximately 95 percent” of the medium leaves much to be interpreted by the individual reader. Whereas many protocols have sufficient wiggle room in them (i.e. they’ll work even if you miss the mark a little), it turns out that this particular one is quite finicky, leading to significantly different results (in our lab’s case this ranged from no stem cells retained to seeing incredible expansion from even single blood stem cells).
Some of the implications of this are quite far-reaching. For example, a number of articles/studies have pointed to a reproducibility crisis in biomedical science casting doubt on the quality of biomedical research in academic institutions. Depending on how you view the “experiment” we just performed, you could make this claim, although I would rather suggest that it is dependent on who is carrying out the protocol, or more accurately, how the protocol was executed.
So where do we go from here? Well the first obvious thing is to encourage more precise writing of protocols. Secondly, because research papers are almost always tight for space, we need to encourage the writing of more specific protocols chapters and lobby researchers to publish their protocols on open websites. Finally, (and it’s surprising this isn’t done more) researchers should be encouraged to create small videos for the particularly tricky aspects of a protocol – post them, share them, and amend practice based on feedback (there are also video journals now!). Sometimes it really feels like we are stuck in the dark ages when it comes to sharing best practice in science when the reality is that we should be doing everything possible to make sure that experiments across the world are done in as reproducible and rigorous a manner as possible.