Skip navigation
The Black Hole

Peer Review and Publishing – the best of the worst?

Peer Review and Publishing – the best of the worst? A look at the peer review system and the idea that anonymity of reviewers is an idea of the past.

BY DAVID KENT | NOV 13 2009

QUICK HITS:

  1. This is RALLY WEEK for the STEM CELL CHARTER – follow their blog , twitter feed, or join their facebook group. David Eaves just made a heartfelt plea on his blog regarding the Charter – I encourage a read!
  2. TheMarkNews has been flying under my radar for some time now and I’ve just been forwarded onto it (Thanks Trevor!) – it’s a great place to see current opinions on relevant news items from a Canadian perspective. …from their site: At its core The Mark is a national movement to record Canadian ideas and propel the people behind them. It is a collection of thoughts and a tool for facilitating interdisciplinary dialogue and debate between outstanding Canadians.

Peer Review and Publishing – the best of the worst?

I need to have a Cell, Science, or Nature paper to get my academic job.

This, my friends, is what every young medical science trainee has at the front of their brain and will drive many of the topics that come up on our blog. More and more of the major journals are edited by journal editors and not academic scientists and the job relies more on finding what will sell subscriptions/advertising rather than the putting out the very best science (thankfully these groups have some overlap, but you’ve certainly heard people query – “…and how did this get into Journal X?”.

An extremely interesting component of the whole process (and the subject of this post) is peer review. It has been described as the best, but still quite lousy, system we have to assess the quality of a journal article. What was it that Winston Churchill said about democracy?? We’re still working on that aren’t we? – peer review has got to change!

At its heart, peer review, as practiced by most journals, is a system where an editor – after a first round of sometimes extensive culling – will send a paper out to 2-4 academics in a related field of research to query the importance, relevance, rigor, and merits of a particular submission. I recently attended a Peer Review workshop here at Cambridge where 50-60 grads, PDFs, and young PIs spent an afternoon debating peer review with a panel of journal editors, professors, and people from Sense About Science, the latter of whom have launched a public campaign about the merits of peer review and published some preliminary figures in Sept 2009.

One of the most heated discussions was on anonymity in the current peer review system. The vast majority of reviewers, and I can only speak from my own field, are anonymous and the paper authors are visible. The first question to be asked: “Should it be double blind” – resounding consensus was, “it wouldn’t matter, the reviewer would generally be able to tell who the person was anyhow based on their work” – it is also interesting to note that double blind studies have been done and women still come out worse off (a great series of articles are here and here)

Back to the anonymity issue… if we can’t be double blind how about no blinding at all? This stimulated massive discussion and I came out with the overwhelming feeling that young scientists (who dominated the room and I can’t generalize about professors because they didn’t reach statistical significance…) would be a-ok with open peer review. Some nay sayers brought up potential drawbacks:

  1. I’d be afraid to review someone harshly that’s got more power than me
  2. I don’t feel qualified to be an expert opinion and worry about losing face for accepting an article that turns out to be terrible.

For these arguments, I have some sympathy as I want a job too, but I can’t help responding with:

  1. If you make good points, aren’t personal, and generally do justice to the review, we have to believe that you’ll come out on top, otherwise the system is not worth saving and peer review needs to go… AND… for every enemy you make that has “power” there’s almost certainly a friend to be made that doesn’t like Professor Powerful .
  2. If you don’t qualify as an expert – turn it down…maybe this will put the pressure on journals to find good reviewers and on scientists to stop submitting above the level where something belongs (and yes, this is a massive problem that I’ll go into in a future blog)

The benefits of public reviewers are plentiful, and there are a few ways to achieve it that range in transparency. A) We could simply have names appear only on articles that are published, B) we could publish names and the actual reviews along with the publication, or C) we could publish everything (negative, positive, etc) for the author (or the public to read). Some of these would be viewed as rather extreme, so I’ll detail just the most realistic option’s benefits, those of first method:

  • Every article would have 2-4 names associated with it, giving the article extra credibility – Imagine thinking “ I think Professor X is great and so smart – he likes this article, I should definitely read it!” – this is akin to theFaculty of 1000 site for biology and medicine (though it lacks complete coverage)
  • It would do well to break up networks of back scratching and I’m sure that bioinformatics and statistics folks would love to do the meta-analysis… if you notice that every paper that Professor X writes is reviewed and approved by Professor Y or the former post doc from Professor Y’s lab, then things start to look fishy.
  • It would make reviewers think long and hard before putting their final decisions out there because their name is permanently affixed to the credibility of the article
  • It would discourage journal editors from overruling multiple reviewers – this of course presumes that we’d be bright enough to publish a YES or NO next to the review.
  • On the tail of the last point, you also get the benefit of knowing if a paper was unanimously accepted or if it was a tough sell (and you might even have the nerve to follow up with someone to see why they didn’t like it!)
  • Imagine… you could publicly shame reviewers with stats like over the past 20 years Professor X has never positively reviewed an article by a female PI or first author (which I’m sure would never happen – right? right?)

* If we get into the practice of publishing the actual reviews, it would be an excellent new metric for the assessment of young scientists without extensive publication records. If you can articulate ideas clearly and identify key holes in logic/experiments, this will reflect well on you.

So… these are just some thoughts, mostly meant to stimulate discussion and let people vent about the pros/cons of peer review, anonymous or otherwise…

Next topic: Say NO to the second PDF

ABOUT DAVID KENT
David Kent
Dr. David Kent is a principal investigator at the York Biomedical Research Institute at the University of York, York, UK. He trained at Western University and the University of British Columbia before spending 10 years at the University of Cambridge, UK where he ran his research group until 2019. His laboratory's research focuses on the fundamental biology of blood stem cells and how changes in their regulation lead to cancers. David has a long history of public engagement and outreach including the creation of The Black Hole in 2009.
COMMENTS
Post a comment
University Affairs moderates all comments according to the following guidelines. If approved, comments generally appear within one business day. We may republish particularly insightful remarks in our print edition or elsewhere.

Leave a Reply to Lisa McDonnell Cancel reply

Your email address will not be published.

  1. Lisa McDonnell / November 14, 2009 at 20:31

    I love these ideas! Imagine if everyone had to be transparent about their reviews. They would actually critically review, instead of pseudo-reviewing, being biased because of the projects they have going on in their own labs….or what they think of the author (OK, it would never eliminate bias completely….but it might reduce it).
    I suppose I’m a bit of a pessimist…does anyone actually think that editors and reviewers in general would agree to transparency and publishing reviews? It’s scary having to be credible….do the editors and reviewers have the guts to do it??

  2. SubC / November 19, 2009 at 18:26

    I suppose peer review is like capitalism or democracy….. it is NOT perfect but it IS the best we have got.
    As a postdoc, I get to review quite a few manuscripts (both with my supervisor and on my own)and recently I have come across manuscripts with the author page removed to maintain confidentiality. If done properly, this could be a boon to new researchers who might be able to compete with better known peers on a more level playing field.
    However, making the reviewers names available might scare off potential reviewers (journal editors are always cajoling ppl to do reviews to begin with). Another option might be to offer remuneratyion to reviewers who wish to be identified, might be a strong incentive for postdocs and sessionals to take on more reviewing duties.
    Just a few thoughts…. and pls discuss what you think.

  3. Dave K / November 21, 2009 at 10:36

    Hi again SubC,
    Thanks for being so active on the site – and feel free to follow up on email if you’d like to suggest an area of discussion as well.
    For the peer review, I think you’re right about the advantage of double blind for young researchers who wouldn’t be “decipherable” to most of the keenest eyes in the field. It is definitely better than the current system that just screams for biases.
    I guess this blog posting was (and is still) trying to advocate for the most open system possible, one where science becomes a little more pure and gets disputed on an experimental level not a political level. Hiding under the cover of anonymity allows politics to factor in much more easily. The open reviews of EMBO (see last week’s quick hits ) are anonymous, but at least they are open about “how” a paper gets in…
    For journals and their need to cajole reviewers – I think the only way the system of open peer review would work is if all journals starting moving in that direction, which is a pipe dream for certain. I think medical scientists need to commit themselves to the idea of openness (look at computer science or even fruit fly biologists for leaders in sharing data/ideas) – we’re obsessed with competition in our field and not with getting information that we have confidence in out there for people to build from. I think the latter will lead to faster, better solutions and a lot less wasted time.
    As for remuneration… I’m not a fan. It would encourage people to take on peer review as a means of income, not as a service to the field. I’d doubt (but I don’t know) that there are many professors who would support paid peer reviews (and it would probably get passed back to the submission costs, which would put an even greater strain on grants). It would probably be attractive to younger people, I’ll concede that – but I don’t think that would actually improve the level of peer review which is the task at hand. I have a feeling that some (dare I say unethical) scientists would take it on simply for the cash… this is a really really dangerous paradigm.