Peer Review

If you've spent some time in graduate school you might have learned a thing or two about the open secret about peer review: it's not entirely done by peers and it's not always that thorough of a review.

Like it or not, grad students often find these reviews on their desk. I've heard of PIs leaving the job to grad students alone, though this does not occur in my lab. It's understandable, though, that this might happen. PIs have huge demands on their time. As invaluable as their opinions are, there is no way some PIs can field every manuscript. Apparently they have their own concerns about peer review (some of which apparently involves colorful dinos)

But what of the reviews performed by graduate students? There are two ways of looking at the duty of peer review.

Looked at one way, a peer review is a learning opportunity. Often the material details development on the edge of some field that contacts at least tangentially upon the student's research. It's a chance to see another lab's raw, early draft manuscript, and learn what merits publication in high level journals and what does not. It is a learning experience, and a chance to contribute to the body of science as a whole.

But let's just step out of the shiny world of gumdrops and candy canes for a moment. Peer review can be an inane chore. While students provide some value added to the journal and the author of the manuscript, it's harder to see where the review process benefits their progression in the ladder. Their role in the review is, effectively, anonymous, and comes with no honor, distinctions, gold stars, pats on the head, brownie points or first author publications.

Stated simply, there is no clear match between incentives and the quality of the peer review. Mistakes are often buried somewhere deep in the unending, jargon-filled paragraphs of the methods section. Whether a grad student takes the effort to check these methods, line by line, comes down to a question of how many hours (if any) of sleep they might prefer to have that evening.

Where's the incentive to dig in? Some grad students seem to possess a deep personal drive to throw other scientists under the bus, but it's probably a minority at most institutions. We can't rely on pure sadism to drive the scientific engine. There must be a way to reward careful and well considered reviews, particularly where they find obscure errors and tenuous methods.

I don't claim to know what the reward should be or how it could be structured. I've considered the possibility that confidential peer review is a mistake, and that publications should instead be edited by the journal seeking to publish. If Nature wants the value added of an expert opinion in the field, let them pay for it! Certainly they demand payment for their subscriptions, so why should their product be provided for free?

At risk of shifting to a seemingly radical alternative, perhaps open access and open comment system is the way to go. Take all comers that pass a basic editorial spot check, and allow insightful, observational comments to come from the community. Those comments can then be tied to the reputation of those who make them. Great insights can be noticed, and unnecessary bickering (I'm looking at you, reviewer #3), can be ignored. Online systems for grading and sorting comments based on reputation systems exist in many forms. Perhaps its time to turn them loose on science.

Concurrently, give labs the hard task of determining their own publication threshold. Perhaps more self review will go on in-house if authors know that they can publish whatever they please, and that they'd better get it right the first time. Perhaps some systems could allow papers to have a 'versioning' system, wherein they could be updated (to a point) to reflect public comment.

I don't claim that pay-to-review or open-comment systems solve the problems inherent in the current publication regime, but I think they deserve consideration. Let us at least recognize that good work deserves good incentives, and that the adding motivation to peer-review can only improve our scientific rigor.