Re: Cracks in the Global Warming Case?
On the subject of peer review and 'experts in the field' and 'conspiracies':
http://michaelnielsen.org/blog/three...c-peer-review/
From the comments:
On the subject of peer review and 'experts in the field' and 'conspiracies':
http://michaelnielsen.org/blog/three...c-peer-review/
In the absence of compelling systematic studies, is there anything we can say about the reliability of peer review?
The question of reliability should, in my opinion, really be broken up into three questions.
First, does peer review help verify the validity of scientific studies; second, does peer review help us filter scientific studies, making the higher quality ones easier to find, because they get into the “best” journals, i.e., the ones with the most stringent peer review; third, to what extent does peer review suppress innovation?
As regards validity and quality, you don’t have to look far to find striking examples suggesting that peer review is at best partially reliable as a check of validity and a filter of quality.
Consider the story of the German physicist Jan Hendrik Schoen. In 2000 and 2001 Schoen made an amazing series of breakthroughs in organic superconductivity, publishing his 2001 work at a rate of one paper every 8 days, many in prestigious journals such as Nature, Science, and the Physical Review. Eventually, it all seemed a bit too good to be true, and other researchers in his community began to ask questions. His work was investigated, and much of it found to be fraudulent. Nature retracted seven papers by Schoen; Science retracted eight papers; and the Physical Review retracted six. What’s truly breathtaking about this case is the scale of it: it’s not that a few referees failed to pick up on the fraud, but rather that the refereeing system at several of the top journals systematically failed to detect the fraud. Furthermore, what ultimately brought Schoen down was not the anonymous peer review system used by journals, but rather investigation by his broader community of peers.
You might object to using this as an example on the grounds that the Schoen case involved deliberate scientific fraud, and the refereeing system isn’t intended to catch fraud so much as it is to catch mistakes. I think that’s a pretty weak objection – it can be a thin line between honest mistakes and deliberate fraud – but it’s not entirely without merit. As a second example, consider an experiment conducted by the editors of the British Medical Journal (ref). They inserted eight deliberate errors into a paper already accepted for publication, and sent the paper to 420 potential reviewers. 221 responded, catching on average only two of the errors. None of the reviewers caught more than five of the errors, and 16 percent no errors at all.
None of these examples is conclusive. But they do suggest that the refereeing system is far from perfect as a means of checking validity or filtering the quality of scientific papers.
What about the suppression of innovation? Every scientist knows of major discoveries that ran into trouble with peer review. David Horrobin has a remarkable paper (ref) where he documents some of the discoveries almost suppressed by peer review; as he points out, he can’t list the discoveries that were in fact suppressed by peer review, because we don’t know what those were. His list makes horrifying reading. Here’s just a few instances that I find striking, drawn in part from his list. Note that I’m restricting myself to suppression of papers by peer review; I believe peer review of grants and job applications probably has a much greater effect in suppressing innovation.
At this point I expect most readers will have concluded that I don’t much like the current peer review system. Actually, that’s not true, a point that will become evident in my post about the future of peer review. There’s a great deal that’s good about the current peer review system, and that’s worth preserving. However, I do believe that many people, both scientists and non-scientists, have a falsely exalted view of how well the current peer review system functions. What I’m trying to do in this post is to establish a more realistic view, and that means understanding some of the faults of the current system.
The question of reliability should, in my opinion, really be broken up into three questions.
First, does peer review help verify the validity of scientific studies; second, does peer review help us filter scientific studies, making the higher quality ones easier to find, because they get into the “best” journals, i.e., the ones with the most stringent peer review; third, to what extent does peer review suppress innovation?
As regards validity and quality, you don’t have to look far to find striking examples suggesting that peer review is at best partially reliable as a check of validity and a filter of quality.
Consider the story of the German physicist Jan Hendrik Schoen. In 2000 and 2001 Schoen made an amazing series of breakthroughs in organic superconductivity, publishing his 2001 work at a rate of one paper every 8 days, many in prestigious journals such as Nature, Science, and the Physical Review. Eventually, it all seemed a bit too good to be true, and other researchers in his community began to ask questions. His work was investigated, and much of it found to be fraudulent. Nature retracted seven papers by Schoen; Science retracted eight papers; and the Physical Review retracted six. What’s truly breathtaking about this case is the scale of it: it’s not that a few referees failed to pick up on the fraud, but rather that the refereeing system at several of the top journals systematically failed to detect the fraud. Furthermore, what ultimately brought Schoen down was not the anonymous peer review system used by journals, but rather investigation by his broader community of peers.
You might object to using this as an example on the grounds that the Schoen case involved deliberate scientific fraud, and the refereeing system isn’t intended to catch fraud so much as it is to catch mistakes. I think that’s a pretty weak objection – it can be a thin line between honest mistakes and deliberate fraud – but it’s not entirely without merit. As a second example, consider an experiment conducted by the editors of the British Medical Journal (ref). They inserted eight deliberate errors into a paper already accepted for publication, and sent the paper to 420 potential reviewers. 221 responded, catching on average only two of the errors. None of the reviewers caught more than five of the errors, and 16 percent no errors at all.
None of these examples is conclusive. But they do suggest that the refereeing system is far from perfect as a means of checking validity or filtering the quality of scientific papers.
What about the suppression of innovation? Every scientist knows of major discoveries that ran into trouble with peer review. David Horrobin has a remarkable paper (ref) where he documents some of the discoveries almost suppressed by peer review; as he points out, he can’t list the discoveries that were in fact suppressed by peer review, because we don’t know what those were. His list makes horrifying reading. Here’s just a few instances that I find striking, drawn in part from his list. Note that I’m restricting myself to suppression of papers by peer review; I believe peer review of grants and job applications probably has a much greater effect in suppressing innovation.
- George Zweig’s paper announcing the discovery of quarks, one of the fundamental building blocks of matter, was rejected by Physical Review Letters. It was eventually issued as a CERN report.
- Berson and Yalow’s work on radioimmunoassay, which led to a Nobel Prize, was rejected by both Science and the Journal of Clinical Investigation. It was eventually published in the Journal of Clinical Investigation.
- Krebs’ work on the citric acid cycle, which led to a Nobel Prize, was rejected by Nature. It was published in Experientia.
- Wiesner’s paper introducing quantum cryptography was initially rejected, finally appearing well over a decade after it was written.
At this point I expect most readers will have concluded that I don’t much like the current peer review system. Actually, that’s not true, a point that will become evident in my post about the future of peer review. There’s a great deal that’s good about the current peer review system, and that’s worth preserving. However, I do believe that many people, both scientists and non-scientists, have a falsely exalted view of how well the current peer review system functions. What I’m trying to do in this post is to establish a more realistic view, and that means understanding some of the faults of the current system.
MikeM said,
January 8, 2009 @ 10:48 pm
Economist George Akerlof wrote his most famous paper, “The Market for Lemons: Quality Uncertainty and the Market Mechanism” in 1967. The paper used the market for used cars as an example of asymmetric information – where a seller knows more about the goods he is selling than a buyer does. The first journal Akerlof tried rejected the paper on the grounds that it was trivial. The second also rejected it. The third rejection came with a referee’s comment that the paper was wrong in its reasoning – that if it were right, economics would be different.
The paper was eventually published, Akerlof shared a Nobel Prize for economics in 2001 primarily on the basis of “The Market for Lemons”, and economics is now different.
January 8, 2009 @ 10:48 pm
Economist George Akerlof wrote his most famous paper, “The Market for Lemons: Quality Uncertainty and the Market Mechanism” in 1967. The paper used the market for used cars as an example of asymmetric information – where a seller knows more about the goods he is selling than a buyer does. The first journal Akerlof tried rejected the paper on the grounds that it was trivial. The second also rejected it. The third rejection came with a referee’s comment that the paper was wrong in its reasoning – that if it were right, economics would be different.
The paper was eventually published, Akerlof shared a Nobel Prize for economics in 2001 primarily on the basis of “The Market for Lemons”, and economics is now different.
MikeM, that was a terrific link to George Akerlof. In particular, Akerlof’s describes an example peer-review failure because “The economists of the time felt that it would violate their methodology to consider a problem, such as the role of asymmetric information, that was out of its traditional focus.”
“Out of our traditional focus” is a very common reason for rejection, not only in academic publishing, but also in business, and politics (and if you think about it, even romance).
I think it was Marvin Minsky, in Society of Mind, who pointed out how very necessary it is, that human cognition has strong censorship mechanisms, operating largely on the preconscious level, that reject ideas that don’t match preconceptions.
This is no bad thing. But the paradoxical result is that it is (sometimes) more difficult for a good idea to find an audience than a mediocre one.
“Out of our traditional focus” is a very common reason for rejection, not only in academic publishing, but also in business, and politics (and if you think about it, even romance).
I think it was Marvin Minsky, in Society of Mind, who pointed out how very necessary it is, that human cognition has strong censorship mechanisms, operating largely on the preconscious level, that reject ideas that don’t match preconceptions.
This is no bad thing. But the paradoxical result is that it is (sometimes) more difficult for a good idea to find an audience than a mediocre one.
John Sidles said,
January 9, 2009 @ 3:12 pm
Sérgio Nunes’ post indirectly points to another unintended consequence of peer review, which comes about as follows (humor alert!).
(1) The most inexpensive kind of faculty to hire are theorem-provers. (2) The most rigorous form of peer review is theorem-checking. (3) Hence, in any academic ecosystem that regulated by peer-review, the theorem-provers eventually become the dominant species of professor.
Arguably, this trend has been wonderful news for mathematics, OK news for the physical sciences, not-so-good news for engineering … and it has been an utter disaster for economics.
January 9, 2009 @ 3:12 pm
Sérgio Nunes’ post indirectly points to another unintended consequence of peer review, which comes about as follows (humor alert!).
(1) The most inexpensive kind of faculty to hire are theorem-provers. (2) The most rigorous form of peer review is theorem-checking. (3) Hence, in any academic ecosystem that regulated by peer-review, the theorem-provers eventually become the dominant species of professor.
Arguably, this trend has been wonderful news for mathematics, OK news for the physical sciences, not-so-good news for engineering … and it has been an utter disaster for economics.
Beginning in the seventeenth century (or perhaps a little earlier – I’m not positive) what would now be equivalent to publishing in a premier journal (Physical Review, Nature, JAMA, etc.), consisted of having one’s paper presented to one of the learned societies of the time. The historical ‘remnants’ of this process are still evident in the names of several of the Royal Society’s journals, notably “Proceedings of…” and “Notes and Records of…”
This process always involved some sort of peer review and often suffered from the same problems peer review still suffers from. Two of the most notable examples of this were the nearly simultaneous attempts by Abel and Galois to have their work (the origins of group theory) communicated to the French Academy of Sciences. Both were unsuccessful (at least in their short lifetimes) and, in fact, Galois received what amounted to a referee’s report dated July 4, 1831 that rejected one of the foremost papers in the history of mathematics. The referees were Poisson and Lacroix. Abel was simply ignored (again, until he was dead).
Ironically, though neither knew the other, they were working on the same problem nearly simultaneously and had trouble not just with the same society, but the same people. At one point or another both ran into the immense ego of Cauchy in their attempts to be recognized by the Academy.
Mario Livio’s book, despite its “popular” title (The Equation That Couldn’t Be Solved: How Mathematical Genius Discovered the Language of Symmetry), represents very serious scholarly research into the history of group theory and, particularly, Abel and Galois themselves. There is also a book out on the history of learned societies that I have not yet read, but, thanks to this post, will make my next read. I don’t recall the title and it is on the bookshelf in my office.
This process always involved some sort of peer review and often suffered from the same problems peer review still suffers from. Two of the most notable examples of this were the nearly simultaneous attempts by Abel and Galois to have their work (the origins of group theory) communicated to the French Academy of Sciences. Both were unsuccessful (at least in their short lifetimes) and, in fact, Galois received what amounted to a referee’s report dated July 4, 1831 that rejected one of the foremost papers in the history of mathematics. The referees were Poisson and Lacroix. Abel was simply ignored (again, until he was dead).
Ironically, though neither knew the other, they were working on the same problem nearly simultaneously and had trouble not just with the same society, but the same people. At one point or another both ran into the immense ego of Cauchy in their attempts to be recognized by the Academy.
Mario Livio’s book, despite its “popular” title (The Equation That Couldn’t Be Solved: How Mathematical Genius Discovered the Language of Symmetry), represents very serious scholarly research into the history of group theory and, particularly, Abel and Galois themselves. There is also a book out on the history of learned societies that I have not yet read, but, thanks to this post, will make my next read. I don’t recall the title and it is on the bookshelf in my office.
Comment