Peer Review and Publishing – the best of the worst?

QUICK HITS:

1. This is RALLY WEEK for the STEM CELL CHARTER – follow their blog , twitter feed, or join their facebook group. David Eaves just made a heartfelt plea on his blog regarding the Charter – I encourage a read!
2. TheMarkNews has been flying under my radar for some time now and I’ve just been forwarded onto it (Thanks Trevor!) – it’s a great place to see current opinions on relevant news items from a Canadian perspective. …from their site: At its core The Mark is a national movement to record Canadian ideas and propel the people behind them. It is a collection of thoughts and a tool for facilitating interdisciplinary dialogue and debate between outstanding Canadians.

Peer Review and Publishing – the best of the worst?

I need to have a Cell, Science, or Nature paper to get my academic job.

This, my friends, is what every young medical science trainee has at the front of their brain and will drive many of the topics that come up on our blog. More and more of the major journals are edited by journal editors and not academic scientists and the job relies more on finding what will sell subscriptions/advertising rather than the putting out the very best science (thankfully these groups have some overlap, but you’ve certainly heard people query – “…and how did this get into Journal X?”.

An extremely interesting component of the whole process (and the subject of this post) is peer review. It has been described as the best, but still quite lousy, system we have to assess the quality of a journal article. What was it that Winston Churchill said about democracy?? We’re still working on that aren’t we? – peer review has got to change!

At its heart, peer review, as practiced by most journals, is a system where an editor – after a first round of sometimes extensive culling – will send a paper out to 2-4 academics in a related field of research to query the importance, relevance, rigor, and merits of a particular submission. I recently attended a Peer Review workshop here at Cambridge where 50-60 grads, PDFs, and young PIs spent an afternoon debating peer review with a panel of journal editors, professors, and people from Sense About Science, the latter of whom have launched a public campaign about the merits of peer review and published some preliminary figures in Sept 2009.

One of the most heated discussions was on anonymity in the current peer review system. The vast majority of reviewers, and I can only speak from my own field, are anonymous and the paper authors are visible. The first question to be asked: “Should it be double blind” – resounding consensus was, “it wouldn’t matter, the reviewer would generally be able to tell who the person was anyhow based on their work” – it is also interesting to note that double blind studies have been done and women still come out worse off (a great series of articles are here and here)

Back to the anonymity issue… if we can’t be double blind how about no blinding at all? This stimulated massive discussion and I came out with the overwhelming feeling that young scientists (who dominated the room and I can’t generalize about professors because they didn’t reach statistical significance…) would be a-ok with open peer review. Some nay sayers brought up potential drawbacks:

1. I’d be afraid to review someone harshly that’s got more power than me
2. I don’t feel qualified to be an expert opinion and worry about losing face for accepting an article that turns out to be terrible.

For these arguments, I have some sympathy as I want a job too, but I can’t help responding with:

1. If you make good points, aren’t personal, and generally do justice to the review, we have to believe that you’ll come out on top, otherwise the system is not worth saving and peer review needs to go… AND… for every enemy you make that has “power” there’s almost certainly a friend to be made that doesn’t like Professor Powerful .
2. If you don’t qualify as an expert – turn it down…maybe this will put the pressure on journals to find good reviewers and on scientists to stop submitting above the level where something belongs (and yes, this is a massive problem that I’ll go into in a future blog)

The benefits of public reviewers are plentiful, and there are a few ways to achieve it that range in transparency. A) We could simply have names appear only on articles that are published, B) we could publish names and the actual reviews along with the publication, or C) we could publish everything (negative, positive, etc) for the author (or the public to read). Some of these would be viewed as rather extreme, so I’ll detail just the most realistic option’s benefits, those of first method:

• Every article would have 2-4 names associated with it, giving the article extra credibility – Imagine thinking “ I think Professor X is great and so smart – he likes this article, I should definitely read it!” – this is akin to theFaculty of 1000 site for biology and medicine (though it lacks complete coverage)
• It would do well to break up networks of back scratching and I’m sure that bioinformatics and statistics folks would love to do the meta-analysis… if you notice that every paper that Professor X writes is reviewed and approved by Professor Y or the former post doc from Professor Y’s lab, then things start to look fishy.
• It would make reviewers think long and hard before putting their final decisions out there because their name is permanently affixed to the credibility of the article
• It would discourage journal editors from overruling multiple reviewers – this of course presumes that we’d be bright enough to publish a YES or NO next to the review.
• On the tail of the last point, you also get the benefit of knowing if a paper was unanimously accepted or if it was a tough sell (and you might even have the nerve to follow up with someone to see why they didn’t like it!)
• Imagine… you could publicly shame reviewers with stats like over the past 20 years Professor X has never positively reviewed an article by a female PI or first author (which I’m sure would never happen – right? right?)

* If we get into the practice of publishing the actual reviews, it would be an excellent new metric for the assessment of young scientists without extensive publication records. If you can articulate ideas clearly and identify key holes in logic/experiments, this will reflect well on you.

So… these are just some thoughts, mostly meant to stimulate discussion and let people vent about the pros/cons of peer review, anonymous or otherwise…

Next topic: Say NO to the second PDF