In my last post, I highlighted a new tool for driving open debate on manuscripts prior to publication in peer-reviewed journals (PREreview). Subsequent to that post, we’ve now undertaken our first scientific discussion (typically called “journal clubs) of a “preprint” manuscript. The most critical thing in this process, however, seems to be speed.
Initially, we sent around four bioRxiv papers for the students and postdocs to choose from and the votes cast actually picked a different paper than the one we ended up reviewing because the selected one popped up with “Now published in Stem Cells” which sort of defeated the purpose of doing a “pre-review.” We therefore moved on to the next most popular article and assessed “Heterogeneous Responses of Hematopoietic Stem Cells to Inflammatory Stimuli are Altered with Age” from David Baltimore’s group at Caltech. Our review of the paper now appears in the public domain and comments can be submitted by readers (as of this post going live, there were none!). Readers of this blog are less likely to be interested in the exact specifics of the paper, but I have copied parts of the short introduction that we posted on the site below to give a sense of the reaction in our group to assessing a bioRxiv article as opposed to a published article as we would normally do.
…this was our first attempt at discussing a preprint paper from BioRxiv so we had a few opening questions and general discussion about the concept between the 15 or so members of the journal club. Our groups (Laurenti and Kent) are based at the WT / MRC Cambridge Stem Cell Institute in Cambridge, U.K. and the opinions and comments found below are a summary of the group discussion rather than the opinion of any single researcher partaking in the journal club.
First question: Was reading this any different than a normal paper?
Some people had worried that a pre-publication paper would suffer from major gaps in organisation or presentation and this was not the case here. Overall, there was not a substantial difference from published papers and people agreed that the structure of the paper was good and the content relatively easy to follow. That said, one cynical comment was raised, querying whether the peer review process prohibited authors from telling their version of the story (e.g., one interpretation), that might actually diverge from the interpretation of the broader field (i.e., peer review makes the article “less readable” but also “more accurate”).
Second question: Would you continue to explore pre-print servers for new content?
Resounding “yes” from majority of researchers (many of whom had not heard of preprint servers prior to the discussion)
Overall, people clearly enjoyed the process and still (several weeks later) speak about the discussion and the “pre-review” style. We’ve been asked by the PREreview group to help beta test the platform and the whole idea of pre-review in general and are doing so – the organizers are quite friendly and I’m sure they would like to hear from more groups willing to go ahead and discuss work in their own journal clubs. Notwithstanding all the positive vibes though, there were some troubling aspects of PREreview which I think warrant some discussion below:
Unknown status of manuscript
The paper we selected was originally uploaded on July 13, 2017 making it about four months old when we sat down to discuss. The obvious benefit is that our groups were seeing the data well before any publication would make possible, but it also meant that there was a pretty decent chance that the authors would have already submitted the paper. Some people might query whether everything on bioRxiv is already submitted. It’s currently impossible to determine this. That said, it is obvious that things are very close to submission even if the submit button hasn’t been clicked, based on the mature state of the articles. If an article is submitted, it makes it challenging for the mission of pre-review as a concept since any article being assessed is almost certainly being assessed elsewhere at the same time, meaning that the authors have simultaneously subjected themselves to multiple critics (the bioRxiv readership and the selected peer reviewers of a journal). Therefore, it would be useful if bioRxiv articles indicated whether or not a paper is truly “pre-review” or not (e.g., has it been submitted). This would allow authors to legitimately seek pre-submission feedback and would allow readers (and pre-reviewers) to selectively target articles seeking such feedback.
As alluded to above, if a paper is already submitted, any “pre-review” that is published might come at an awkward time in the traditional review/publishing process. In undertaking our review of the Mann et al., paper, we had not intended to flag new issues that would impede publication of the manuscript as it went through peer review at a journal. However, this is a formal possibility (if reviewers are combing the internet for such “pre-reviews”) and consequently, exposure to PREreview might result in less enthusiasm from authors to post on bioRxiv (which would be the exact opposite of the goals of bioRxiv, ASAPbio, etc). Surely we should be rewarding, not penalizing, authors for publishing on pre-print servers.
Secondly, the feedback that we give (valuable or not) doesn’t “count” in the same way that formal peer review does and the authors are certainly not obliged to listen to any of the comments. It is also quite unclear at this stage how much, if any, value our review will be to the community that eventually reads this paper in a journal. If PREreview becomes more prevalent, it will pose a major challenge to the new generation of scientists and journals to interact with and communicate an article’s meta-data to best assess how it has been received by the community.
So, where do we go from here? Our group will likely undertake a couple of other PREreviews over the coming months to determine whether it is something we will do as standard practice or not (biologists appear to try most things at least three times, after all…). Hopefully, sharing this experience will help other journal clubs across the world liberate their discussions for pre-prints or other things. In my opinion, it is one of the greatest flaws in our current system that we lock away such academic discussion about research by restricting it to a meeting room with tiny groups of people – I hope that in my lifetime as an academic this can change.