Myths about Peer Review

Michael Nielsen has an interesting article discussing three myths about scientific peer review.

I certainly wasn't aware how little peer review was used until recently. For example, perhaps only one of Einstein's 300 papers was subject to what we modern scientists consider as peer review, and Einstein was far from pleased at the (correct) negative report he obtained.

I'm less than convinced by Nielsen's claim (Myth 2) that peer review is not reliable. Its not perfect, but just because its not perfect does not mean that it doesn't serve some useful function at all. All practicing scientists certainly know the system has a large random component. With respect to papers destined for Journals I'd say on peer review has improved all my papers (although often in very minor ways) and that my reviews of other's papers have improved them. Some of the stuff in a Referee's report turns out to be a waste of time, other parts may be good suggestions or advice, and other parts total misreadings or misunderstandings of the paper (in which case you know other readers will probably also misunderstand it too, so you'd better explain yourself better).

In the two cases where I have rejected a paper in the 14 years of my scientific career (including my time as a PhD student) both cases had demonstrable fatal flaws. Both of those were theory papers with mistakes that a non-expert almost certainly would not have identified themselves, and thus letting them enter the literature would have been detrimental. Bad observational papers are harder to reject as the data may be ultimately useful even if the Author's analysis is sub-par or their interpretation is merely crazy speculation.

Peer-review, as applied to grant or observing time, certainly has a larger random component that reviewing papers, but that is understandable given the broader range of science you'd encounter on a peer review committee (i.e. subjects further from your own specialized area of expertise), and the large number of proposals the committee has to read, rate and debate in the short amount of time allocated. A committee may like a proposal one year that another committee didn't like a previous year. There is a large random component in the absolute ranking of proposals, but generally people can come to agreements on what are the best and worst proposals. At that imperfect level peer review works.

Nielsen's Myth 3 is correct. Just because a paper passes peer review does not mean it is correct!

But I do think peer review is a necessary component of modern science, even if (as Nielsen argues) science functioned acceptably without widespread peer review in the 19th and early 20th centuries. Modern science is vastly larger and more complex in scope and depth that it was even 50 years ago. The degree of specialization is such that even the smartest and most widely-read scientists can not fully judge papers outside their own narrow range of expertise themselves. Peer review is an imperfect filter, but without it there would be even more wrong science out there. There is no way that increasing the amount of flawed, bad or fundamental wrong papers can be beneficial to science, can there?

Ultimately peer review requires that you have informed, qualified, peers (I suppose they're not technically peers if they aren't qualified). If science becomes so specialized that no-one else understands anyone else's work then peer review will fail. Similarly "peer" review by unqualified reviewers doesn't work (e.g. the slashdot comment rating system - as the number of readers increased and their average technical knowledge decreased the system devolved in to pure uninformed opinion. Eventually the signal-to-noise ratio decreased so far that I personally consider slashdot unreadable). Reviewers have to know something to make useful reviews!

Its also worth remembering that astronomy/astrophysics journals have relatively low rejection rates compared to other sciences. If I were in a field where my papers suffered a 50% chance of being rejected I might have a less positive view of peer review!